JP2020204755A - Speech processing device and speech processing method - Google Patents

Abstract

To provide techniques allowing production of a speech waveform with a desired pitch and high quality.SOLUTION: A speech processing device comprises: an acquisition unit to acquire an acoustic feature quantity for generating a speech waveform; and a generation unit to generate the speech waveform by inputting, into a neural network, an acoustic feature quantity and a periodic waveform signal that corresponds to a fundamental frequency of the speech waveform, and performing transform processing using information output by the neural network. The neural network outputs first information for generating an aperiodic component, and second information indicating a periodic component. The transform processing is processing to add second information to information obtained by arithmetic processing using the first information and an aperiodic waveform signal.SELECTED DRAWING: Figure 1

Description

The present invention relates to a voice processing device and a voice processing method.

As a recent voice processing device, a device that generates a voice waveform using a neural network is known. In the techniques described in Non-Patent Document 1 and Non-Patent Document 2, voice waveforms are generated by a neural network using convolution. Further, in the technique described in Non-Patent Document 3, the audio waveform is generated from the acoustic feature amount by applying the technique described in Non-Patent Document 1 and Non-Patent Document 2.

A. van den Oord et al., "Wavenet: A Generative Model for Raw Audio", arXiv preprint arXiv: 1609.03499, 2016 A. van den Oord et al., "Parallel WaveNet: Fast High-Fidelity Speech Synthesis", arXiv preprint arXiv: 1711.10433, 2017 Akira Tamamori et al., "Speaker-dependent Wavenet vocoder", In: INTERSPEECH, pp. 1118-1122, Aug. 2017 Takuhiro Kaneko et al., "CycleGAN-VC: Non-parallell Voice Conversion Usage Cycle-Consistent Adversarial Networks", 5th EURASIP Conference. 2114-2118

In the technique described in Non-Patent Document 3, a voice waveform is generated by using spectrum information, fundamental frequency information, etc., which are acoustic features, as auxiliary information. In the field of voice processing technology using such a neural network, a technology capable of generating a high-quality voice waveform and a technology capable of generating a voice waveform having a desired pitch are desired.

The present invention has been made to solve the above-mentioned problems, and can be realized as the following forms.

(1) According to one embodiment of the present invention, a voice processing device is provided. This voice processing device inputs the periodic waveform signal corresponding to the fundamental frequency of the voice waveform to the neural network and the acquisition unit that acquires the acoustic feature amount for generating the voice waveform, and inputs the acoustic feature amount. Therefore, it is provided with a generation unit that generates the voice waveform by performing a conversion process using the information output by the neural network. The neural network outputs first information for generating aperiodic components and second information indicating periodic components, and the conversion process is calculated using the first information and the aperiodic waveform signal. This is a process of adding the processed information and the second information. According to the audio processing device of this form, the information obtained by performing arithmetic processing using the first information for generating the aperiodic component and the aperiodic waveform signal, and the second information indicating the periodic component are added together. Since the voice waveform is generated, it is possible to generate a voice waveform having a desired pitch with high quality.
(2) In the audio processing of the above form, the first information is information indicating the strength of the aperiodic component for each predetermined frequency band, and the conversion processing is the aperiodic waveform signal for each frequency band. , And the information obtained by multiplying the corresponding first information and the second information may be added. According to the audio processing device of this form, the information obtained by multiplying the aperiodic waveform signal for each predetermined frequency band by the first information indicating the strength of the aperiodic component for each corresponding frequency band and the periodic component. Since the voice waveform is generated by adding the second information indicating the above, it is possible to generate a voice waveform having a desired pitch with high quality.
(3) In the voice processing apparatus of the above-described embodiment, the generation unit may further input a signal indicating the degree of presence / absence of a period according to the voice waveform to be generated to the neural network. According to the voice processing device of this form, it is possible to generate a high-quality voice waveform according to information about an excitation source such as a silent part of the voice waveform to be generated or a voiceless consonant part.
(4) In the voice processing apparatus of the above-described embodiment, the generation unit may input a plurality of periodic waveform signals having different phases to the neural network. According to this form of voice processing device, it is possible to more effectively generate a voice waveform having a desired fundamental frequency.
(5) In the voice processing device of the above embodiment, a learning unit that further learns the relationship between the acoustic feature amount, the periodic waveform signal, the first information, and the second information by machine learning and reflects them in the neural network. May be provided. According to the voice processing device of this form, the relationship between the acoustic feature amount and the voice waveform can be learned, and the learning result can be reflected in the generation unit.

The present invention can be realized in various aspects. For example, a voice processing system using this type of voice processing device, a method executed in the information processing device to realize the functions of the voice processing device and the voice processing system, a computer program, and a server for distributing the computer program. It can be realized in the form of a device, a non-temporary storage medium that stores the computer program, or the like.

It is explanatory drawing which shows the outline of the voice processing apparatus. It is a figure which shows an example of various parameters in an acoustic feature quantity. It is explanatory drawing for demonstrating the neural network in 1st Embodiment. It is explanatory drawing for demonstrating the conversion process in the generation of a voice waveform. It is a flowchart which shows the voice waveform generation processing. It is explanatory drawing which shows the other aspect of the neural network. It is a figure which shows an example of the generated voice waveform. It is a figure which showed the experimental result of the subjective evaluation experiment. It is a figure which shows an example of a periodic auxiliary signal. It is a figure of an example of a plurality of periodic waveform signals having different phases.

A. First Embodiment:
FIG. 1 is an explanatory diagram showing an outline of a voice processing device 100 according to an embodiment of the present invention. The voice processing device 100 includes an acquisition unit 10, a generation unit 20, and a learning unit 30. The acquisition unit 10, the generation unit 20, and the learning unit 30 are realized by software when one or more CPUs or GPUs execute a program stored in the memory. Note that some or all of these may be realized in hardware by a circuit.

The acquisition unit 10 acquires an acoustic feature amount for generating a voice waveform. The details of the acoustic features will be described later. For example, the acquisition unit 10 may extract an acoustic feature amount from a voice waveform of a pre-recorded voice by using a well-known voice analysis technique, and the acoustic feature generated in advance according to a text or a musical score to be spoken. You may get the quantity.

The generation unit 20 includes a noise generation source 21 and a bandpass filter unit 22. The noise source 21 generates an aperiodic waveform signal. The aperiodic waveform signal is a signal representing noise, for example, Gaussian noise. The bandpass filter unit 22 filters the aperiodic waveform signal generated by the noise source 21 for each predetermined frequency band, and generates a plurality of aperiodic waveform signals having different frequency bands.

The generation unit 20 generates a voice waveform by performing a conversion process using information output by a neural network having a plurality of output channels (Neural Network). The generation unit 20 inputs a periodic waveform signal corresponding to the fundamental frequency of the generated voice waveform to the input layer of the neural network, and inputs the acoustic feature amount acquired by the acquisition unit 10 to the neural network as auxiliary information. The first information and the second information are output.

The periodic waveform signal is a periodic waveform signal corresponding to the fundamental frequency of the voice waveform to be generated. The periodic waveform signal may include a speech style, a singing style, and the like. For example, when generating a voice waveform to which vibrato is added, a periodic waveform signal corresponding to the fundamental frequency in the state where vibrato is added may be used. The periodic waveform signal is, for example, a sine waveform signal having the same frequency as the fundamental frequency of the generated voice waveform, or a cosine waveform signal having a frequency one octave higher than the fundamental frequency of the generated voice waveform. Further, the periodic waveform signal may be a non-sinusoidal triangular wave, a sawtooth wave, a short diameter wave or a pulse wave signal. The fundamental frequency of the voice waveform to be generated may be obtained from the voice waveform of the pre-recorded voice by using a well-known voice analysis technique, and is generated in advance according to the text or the score to be spoken. You may use the fundamental frequency.

The generation unit 20 performs conversion processing using the first information and the second information output by the neural network and the aperiodic waveform signal generated by the noise generation source 21, and generates a voice waveform. In the present embodiment, the generation unit 20 performs conversion processing using the first information, the second information, and the aperiodic waveform signal generated by the bandpass filter unit 22. The first information is information for generating an aperiodic component, and in the present embodiment, is information indicating the strength of the aperiodic component for each predetermined frequency band. The second information is information indicating a periodic component, and more specifically, is amplitude information obtained by sampling the periodic component for each sampling period. The details of the conversion process will be described later.

The learning unit 30 learns the relationship between the acoustic feature amount, the periodic waveform signal, the first information, and the second information by supervised machine learning or unsupervised machine learning (see, for example, Non-Patent Document 4). And optimize the various parameters used in the neural network. In supervised machine learning, for example, natural voice of a voice waveform to be generated is used as teacher data, and learning is performed by comparing with the result of conversion processing using the first information and the second information. The learning unit 30 reflects the learning result in the neural network used by the generation unit 20. By doing so, the generation unit 20 can generate the voice waveform by reflecting the learning result of the learning unit 30. The voice processing device 100 does not have to include the learning unit 30. In this case, the generation unit 20 may generate a voice waveform from the first information and the second information by a conversion process described later, reflecting the learning result obtained by an external learning device or the like that performs machine learning. it can.

FIG. 2 is a diagram showing an example of various parameters in the acoustic features. In the present embodiment, the acoustic feature amount is a voice feature amount. Spectral parameters include mer cepstrum and line spectrum pair (Line Spectrum Pair (LSP)). These are sometimes referred to as spectral information. The sound source information includes the fundamental frequency. The fundamental frequency is generally treated as a logarithmic fundamental frequency, and as related parameters, a distinction between voiced / unvoiced and an aperiodic index can be considered. Since the unvoiced part does not have a logarithmic fundamental frequency value, the voiced / unvoiced part may be distinguished by a method such as inserting a predetermined constant in the unvoiced part instead of including the voiced / unvoiced distinction in the sound source information. .. Since the fundamental frequency in the sound source information is the information included in the above-mentioned periodic waveform signal, it may be omitted. Further, the information regarding the distinction between voiced and unvoiced is also information included in the periodic auxiliary signal described later, and may be omitted. Further, the spectrum information and the sound source information may include a speech style, a singing style, and the like. For example, as the spectrum information, it is possible to use the spectrum information in a state where the vibrato of the loudness is added.

FIG. 3 is an explanatory diagram for explaining the neural network used by the generation unit 20. The neural network 200 includes a plurality of dilation layers L1 to L4. The number of dilation layers can be arbitrarily determined. The "dilation layer" is also referred to as an "extension layer" or an "intermediate layer".

The dilation layer L1 is a layer into which information is input. Hereinafter, it is also referred to as an “input layer”. The dilation layer L1 performs initial arithmetic processing and information convolution based on the input signal, and the dilation layers L2 to L4 perform information convolution based on the information transmitted from the lower layer. Each layer contains multiple nodes.

The generation of the first information and the second information by the neural network 200 will be described. In FIG. 3, the first information is shown as “a1”, “a2” ..., And the second information is shown as “b”. Hereinafter, this information is also referred to as "data". In this embodiment, the neural network 200 generates 24 first pieces of information. The periodic waveform signal samples S1 to S8 are input to the nodes N1 to N8 in chronological order after the initial arithmetic processing is performed on the dilation layer L1. Each node N1 to N8 of the dilation layer L1 transmits the information obtained by convolving the information to the dilation layer L2 which is the upper layer. For convenience of illustration, eight periodic waveform signal samples S1 to S8 are input to the dilation layer L1 shown in FIG. 3, but the number of input samples can be arbitrarily determined, for example, 3000. is there.

In the dilation layers L2 to L4, various operations are performed stepwise in each layer with respect to the information transmitted from the input layer L1. Acoustic features corresponding to each sample are input as auxiliary information AI to each node N1 to N8 of the input layer L1 and each node of the dilation layers L2 to L4. In addition to the information transmitted from the lower layer, a sample of the periodic waveform signal may be input to the dilation layers L2 to L4. In the dilation layer L4, the data finally calculated and the data of the rightmost node of each layer, that is, the data of the node to which the earliest data in the time series is input are added and the calculation process is performed. Data DA is output. In the present embodiment, the data DA is input with the first information a1 to a24 indicating the strength of the aperiodic component for each frequency band divided into 24 at the time of the input sample S8 in the time series. It is the second information b which is the amplitude information predicted as the periodic component of the audio sample at the time of sample S8. The neural network 200 in the present embodiment has a structure in which the closer the sample is in the time series, the stronger the influence on the output data DA is likely to be. Specifically, sample S8 is more likely to affect the prediction of data DA than sample S1.

FIG. 4 is an explanatory diagram for explaining the conversion process in the generation of the voice waveform. The generation unit 20 adds the information obtained by multiplying the aperiodic waveform signals nz1 to nz24 for each frequency band generated by the bandpass filter unit 22 by the corresponding first information a1 to a24, and the second information b. By doing so, a voice waveform is generated. The information obtained by multiplying the first information a1 to a24 by the aperiodic waveform signals nz1 to nz24 and the second information b need to be summed together, and the first information a1 to a24 are the aperiodic waveform signals nz1 to nz1 to each. The second information b may be added after adding the information multiplied by nz24, or the information obtained by multiplying the first information a1 to a24 by the aperiodic waveform signals nz1 to nz24 and the second information b, respectively. You may add them at the same time. The frequency bands in the first information a1 to a24 and the aperiodic waveform signals nz1 to nz24 are, for example, bands separated by 1000 Hz. The aperiodic waveform signals nz1 to nz24 are, for example, Gaussian noises generated by the bandpass filter unit 22 having different frequency bands. In the present embodiment, the frequency band is divided into 24, but the number of divisions is not limited to this.

FIG. 5 is a flowchart showing a voice waveform generation process using the voice processing device 100 in the present embodiment. First, the acquisition unit 10 acquires the acoustic feature amount in step S100. Next, in step S110, the generation unit 20 inputs the acoustic feature amount acquired in step S100 and the periodic waveform signal for a predetermined period into the neural network, and aperiodic components for each predetermined frequency band. The first information indicating the strength of is output and the second information indicating the periodic component is output. Finally, in step S120, the generation unit 20 performs conversion processing using the information output by the neural network in step S110 to generate a voice waveform.

According to the voice processing device 100 of the present embodiment described above, the generation unit 20 includes information that has been subjected to arithmetic processing using the first information for generating the aperiodic component and the aperiodic waveform signal, and the period. A voice waveform is generated by adding the second information indicating the components. More specifically, information obtained by multiplying a predetermined aperiodic waveform signal for each frequency band generated by the bandpass filter unit 22 by first information indicating the strength of the aperiodic component for each corresponding frequency band. And the second information indicating the periodic component are added to generate a sound waveform, so that a sound waveform having a desired sound pitch with high quality can be generated. In addition, it is possible to generate a voice waveform at a higher speed than a neural network having an autoregressive structure that predicts the next data by inputting the data output by the neural network itself into the neural network. Further, the learning unit 30 can learn the relationship between the acoustic feature amount, the periodic waveform signal, the first information, and the second information, and the learning result can be reflected in the generation unit 20. Further, even if the sound waveform has a fundamental frequency that greatly deviates from the learning range of the learning unit 30, the generation unit 20 outputs a periodic waveform signal corresponding to the fundamental frequency of the sound waveform to be generated to the input layer of the neural network. Since the sound waveform is generated by inputting to, a sound waveform having a desired pitch can be generated.

FIG. 6 is an explanatory diagram showing another aspect of the neural network. The neural network shown in FIG. 6 is configured by providing the structure of the neural network shown in FIG. 3 symmetrically. A sample of the periodic waveform signal is input to the input layer L1 as in the first embodiment. A sample of a past periodic waveform signal and a sample of a future periodic waveform signal in the time series of the output data DA are input to the input layer L1 of the neural network of the present embodiment. More specifically, the information obtained by performing the initial arithmetic processing is input to the samples S1 to S7 of the past periodic waveform signals to the nodes N1 to N7, and the initial arithmetic processing is input to the samples S8 of the current periodic waveform signal to the nodes N8. The information on which the arithmetic processing has been performed is input, and the information on which the initial arithmetic processing has been performed is input to the samples S9 to S15 of the future periodic waveform signals in the nodes N9 to N15. Further, at each node, the acoustic feature amount is input as auxiliary information as in the first embodiment. The neural network 200 shown in FIG. 6 has a structure in which the closer the sample is in the time series, the stronger the influence on the output data DA is likely to be. Specifically, the sample S8 is more likely to have a stronger influence on the prediction of the data DA than the sample S1 and the sample S15. By using such a neural network, not only a sample of the past periodic waveform signal in the time series of the generated data but also a sample of the future periodic waveform signal is input, so that a higher quality voice waveform can be generated.

FIG. 7 is a diagram showing an example of the voice waveform generated in the embodiment. The waveform shown in the upper row is a target voice waveform, and is a waveform to be generated by voice processing. The waveform shown in the middle row is the voice waveform generated in the example. The waveform shown in the lower row is a periodic waveform signal input to the neural network, and is a sine waveform having the same fundamental frequency as the target voice waveform. As shown in FIG. 7, the voice waveform generated in the embodiment fluctuates in the same period T, and has the same fundamental frequency as the target voice waveform.

Experimental result:
FIG. 8 is a diagram showing the mean opinion score (MOS), which is the experimental result of the subjective evaluation experiment for the generated voice waveform. In this experiment, the quality of the synthetic speech of the four methods was evaluated by a five-stage subjective evaluation experiment of "1: very bad, 2: bad, 3: normal, 4: good, 5: very good". FIG. 8 shows the scores of two of the four methods. There were 16 subjects, and each subject evaluated 10 phrases for each method from 10 songs that were test data. The voice waveform of the synthetic voice to be evaluated was generated by using the same acoustic features in both methods.

In the embodiment, a voice waveform was generated using the voice processing device 100 of the first embodiment described above and the neural network shown in FIG. In the comparative example, a voice waveform was generated by a vocoder technique using a neural network of wavenet (described in Non-Patent Document 1). The same acoustic features as in the examples were input to the wavenet neural network. As shown in FIG. 8, the score of the example was higher than the score of the comparative example. That is, when the generation unit 20 generates the voice waveform according to the above embodiment, the voice waveform can be generated with higher quality. The remaining two methods not shown in FIG. 8 are (1) the original voice produced by human singing as it is, and (2) the same method as in the embodiment, and the neural network in the embodiment is voiced. This is a method of generating a voice waveform using a trained neural network optimized by unsupervised machine learning (see, for example, Non-Patent Document 4) by the learning unit 30 of the processing device 100.

B. Second embodiment:
The generation unit 20 in the second embodiment further inputs a signal (hereinafter, referred to as “periodic auxiliary signal”) indicating the degree of presence / absence of a period according to the voice waveform to be generated to the input layer of the neural network. It differs from the first embodiment in that a voice waveform is generated. Since the configuration of the voice processing device 100 of the second embodiment is the same as the configuration of the voice processing device 100 of the first embodiment, the description of the configuration will be omitted.

In the present embodiment, the generation unit 20 inputs a periodic waveform signal and a periodic auxiliary signal to the input layer L1 of the neural network. That is, in the present embodiment, the node of the input layer of the neural network used by the generation unit 20 has two input channels. For example, a sample of the periodic waveform signal is input to the first channel, and a sample of the periodic auxiliary signal is input to the second channel. The order of the channels can be arbitrarily determined.

The periodic auxiliary signal can be determined according to the boundary position where the periodic waveform starts and the boundary position where the periodic waveform ends, and can be expressed by a value of 0 to 1, with the aperiodic waveform portion being 0 and the periodic waveform portion being 1. For example, the periodic auxiliary signal at the boundary position where the periodic waveform starts is a signal that linearly interpolates from 0.0 to 1.0 from the position before 240 samples to the position after 240 samples of the boundary that switches from unvoiced to voice. Yes, the periodic auxiliary signal at the boundary position where the periodic waveform ends is a signal that linearly interpolates from 1.0 to 0.0 from the position before 240 samples to the position after 240 samples of the boundary that switches from voiced to unvoiced. is there. FIG. 9 is a diagram showing an example of a periodic auxiliary signal. Further, the periodic auxiliary signal may be data in which phonemes or values for each frame are linearly interpolated.

According to the voice processing device 100 of the present embodiment described above, the generation unit 20 directly inputs the periodic waveform signal to the input layer of the neural network to generate the voice waveform, so that a desired fundamental frequency is set. It is possible to generate a voice waveform to have. Further, since the generation unit 20 further inputs a periodic auxiliary signal to the input layer of the neural network, the generation unit 20 responds to information on an excitation source such as a silent portion or an unvoiced consonant portion of the voice waveform to be generated. High-quality audio waveform can be generated.

C. Third Embodiment:
The generation unit 20 in the third embodiment is different from the first embodiment in that a plurality of periodic waveform signals having different phases are input to the input layer of the neural network to generate a voice waveform. Since the configuration of the voice processing device 100 of the third embodiment is the same as the configuration of the voice processing device 100 of the first embodiment, the description of the configuration will be omitted.

FIG. 10 is a diagram of an example of a plurality of periodic waveform signals having different phases. In the present embodiment, the generation unit 20 inputs the periodic waveform signal Ws and the periodic waveform signal Wc to the input layer L1 of the neural network. That is, in the present embodiment, the node of the input layer of the neural network used by the generation unit 20 has two input channels. A sample of the periodic waveform signal Ws is input to the first channel, and a sample of the periodic waveform signal Wc is input to the second channel. The order of the channels can be arbitrarily determined.

The periodic waveform signal Ws is a sine waveform having the same fundamental frequency as the generated voice waveform, and the periodic waveform signal Wc is a cosine waveform having the same fundamental frequency as the generated voice waveform. As shown in FIG. 10, in the periodic waveform signal Ws, the amplitude value in the case of timing t1 when rising and the value in the case of timing t2 when falling are both amplitude A1, but timing t1. The periodic waveform signal Wc in is the amplitude A2, and the periodic waveform signal Wc at the timing t2 is the amplitude A3 having a value different from the amplitude A2. Therefore, when the periodic waveform signal Ws is the amplitude A1, the neural network used by the generation unit 20 is rising when the periodic waveform signal Wc is amplitude A2, and falling when the periodic waveform signal Wc is amplitude A3. It can be uniquely determined.

According to the voice processing device 100 of the present embodiment described above, the generation unit 20 inputs a plurality of periodic waveform signals having different phases to the input layer of the neural network, so that the neural network used by the generation unit 20 has a period. It is possible to uniquely determine whether the value of the waveform signal is the value at the time of rising or the value at the time of falling. Therefore, the generation unit 20 can more effectively generate a voice waveform having a desired fundamental frequency, and can generate a higher quality voice waveform.

D. Fourth Embodiment In the fourth embodiment, the structure of the neural network used by the generation unit 20 is different from that of the first embodiment. Since the configuration of the voice processing device 100 of the fourth embodiment is the same as the configuration of the voice processing device 100 of the first embodiment, the description of the configuration will be omitted.

In the present embodiment, the generation unit 20 outputs the first information and the second information by using a neural network having a structure in which a plurality of neural networks shown in FIGS. 3 and 6 are vertically stacked. For example, when two neural networks are overlapped, the generation unit 20 inputs the information output by the lower neural network to the input layer L1 of the upper neural network, and outputs the first information and the second information. .. That is, the output of the lower neural network is prepared for the number of nodes of the input layer L1 of the upper neural network.

According to the voice processing device 100 of the present embodiment described above, the generation unit 20 directly inputs the periodic waveform signal to the input layer of the neural network to generate the voice waveform, so that a desired fundamental frequency is set. It is possible to generate a voice waveform to have. Further, since the generation unit 20 obtains the first information and the second information by using the neural network having a structure in which a plurality of neural networks are stacked, and generates the voice waveform, compared with the neural network having only one stage structure, When the same number of samples are input to generate a voice waveform, the neural network of each stage can be made smaller. Therefore, since it is possible to input a large number of samples and generate a voice waveform without increasing the parameters as a whole, it is possible to generate a higher quality voice waveform.

E. Other embodiments:
(E1) In the above embodiment, the acoustic feature amount acquired by the acquisition unit 10 is the feature amount of the singing voice. Instead, the acquisition unit 10 may acquire the feature amount of the spoken language as the acoustic feature amount. According to this form, it is possible to generate a voice waveform that is a text-synthesized voice that is not a singing voice. In addition, it is possible to generate a voice waveform that more accurately reproduces voice tones, accents, intonation, four tones in Chinese, and the like. Further, the acquisition unit 10 may acquire a feature amount representing voice quality as an acoustic feature amount. The feature amount representing the voice quality is an acoustic feature amount extracted from the voice of another person. According to this form, it is possible to generate a voice waveform that has undergone voice quality conversion that converts the acoustic features of one speaker into the acoustic features of another speaker. When performing voice quality conversion, the acoustic feature amount may be the acoustic feature amount of the voice to be converted or the acoustic feature amount of the voice to be converted. Further, the difference between these acoustic features may be used as the acoustic features, or both may be used. As a periodic waveform signal, a periodic signal having a fundamental frequency of the voice to be converted or a voice to be converted, a periodic signal which is a residual signal of the voice to be converted, and a periodic signal having a fundamental frequency of the voice to be converted are input to the neural network. You may. Further, the acquisition unit 10 may acquire the feature amount of the musical instrument sound as the acoustic feature amount and input it to the neural network as auxiliary information. According to this form, it is possible to generate a voice waveform that is not a singing voice but a musical instrument sound. When generating a percussion instrument sound, the acquisition unit 10 acquires a feature amount of the percussion instrument sound, and uses a pulse signal that rises at a timing when the percussion instrument is desired to be sounded as a periodic waveform signal. More specifically, when it is desired to generate an eight-beat hi-hat voice waveform, a pulse signal is used, which is 1 for every eighth note and 0 for the others.

(E2) In the above embodiment, the acquisition unit 10 acquires the acoustic feature amount by converting the musical score feature amount and the language feature amount, which are the sources of the voice waveform to be generated, into the voice feature amount by using a well-known conversion technique. You may. Further, the acquisition unit 10 may use the information obtained by converting the musical score feature amount and the language feature amount by using an arbitrary neural network as the acoustic feature amount. Further, the learning unit 30 simultaneously learns the neural network used for converting the musical score feature amount and the language feature amount and the neural network that outputs the first information and the second information in the above embodiment, and optimizes various parameters. You may.

(E3) In the above embodiment, the acoustic feature amount acquired by the acquisition unit 10 may include expression information in addition to the sound source information and the spectrum information. The expression information includes, for example, the pitch and amplitude of the pitch vibrato and its presence or absence in the case of singing, the vibrato cycle and amplitude of the loudness and its presence or absence, and the accent and intonation in the case of spoken language. In the case of sound, the degree of choking of the guitar and the presence or absence of it are included. Instead of including the distinction between the presence and absence of pitch vibrato in the singing expression information, the presence or absence of pitch vibrato may be distinguished by a method such as inserting a predetermined constant in the portion without pitch vibrato. Similarly, instead of including the distinction between the presence and absence of loudness vibrato in the singing expression information, the distinction between the presence and absence of loudness vibrato is made by inserting a predetermined constant in the part without the loudness vibrato. May be done.

(E4) In the above embodiment, the node of the input layer L1 of the neural network may have two or more input channels. For example, two input channels are provided in the input layer L1, a sample of a periodic waveform signal is input to the first channel, and one of the samples input to the first channel in time series is input to the second channel. A sample of the periodic waveform signal at the previous time point may be input. Further, the neural network may input a plurality of types of periodic waveform signal samples at the same time point in a time series to a plurality of input channels and output first information and second information to each channel. As a result, it is possible to generate a voice waveform representing a multiplex voice or a chord in which a plurality of voices are overlapped.

(E5) In the above embodiment, the generation unit 20 inputs a periodic waveform signal corresponding to the fundamental frequency of the voice waveform generated in the input layer of the neural network, and inputs the acoustic feature amount to the neural network as auxiliary information. There is. The generation unit 20 may further input the aperiodic waveform signal to the input layer of the neural network.

(E6) In the above embodiment, the noise generation source 21 generates Gaussian noise as an aperiodic waveform signal, but the present invention is not limited to this, and a signal representing other noise may be generated. The noise source 21 generates, for example, white noise.

(E7) In the above embodiment, the generation unit 20 outputs the first information and the second information by using one neural network. Instead, the generation unit 20 may output the first information and the second information, respectively, by using two neural networks. Further, in this embodiment, the generation unit 20 outputs the second information to the input layer of one neural network that outputs the first information as a periodic waveform signal corresponding to the fundamental frequency of the generated voice waveform by the other neural network. May be entered.

(E8) In the above embodiment, the generation unit 20 uses the neural network to output, as the first information, information for generating aperiodic components, such as acoustic features such as mer cepstrum and LSP (line spectrum pair). A conversion process for generating a voice waveform may be performed by outputting a quantity and performing arithmetic processing using the first feature quantity, the second feature quantity, and the aperiodic waveform signal. For example, the generation unit 20 uses a neural network to output first information which is a 24-dimensional aperiodic component mer cepstrum and second information which is a one-dimensional periodic component. Then, the generation unit 20 generates an aperiodic component by convolving the first information into the aperiodic waveform signal generated by the noise generation source 21, and generates a voice waveform by adding it to the second information.

(E9) In the second embodiment, the generation unit 20 may further input periodic waveform signals having different phases into the input layer of the neural network to generate a voice waveform. That is, the second embodiment and the third embodiment may be combined. More specifically, the generation unit 20 is, for example, a periodic waveform signal Ws composed of a sine waveform having the same fundamental frequency as the voice waveform to be generated and a periodic waveform signal composed of a cosine waveform having the same fundamental frequency as the voice waveform to be generated. Wc and the periodic auxiliary signal can be input to the input layer of the neural network.

(E10) In the second embodiment, the periodic auxiliary signal may be determined according to, for example, the linguistic information of the voice waveform to be generated. "Language information" is, for example, information on vowels and consonants. Linguistic information may be included in the acoustic features. More specifically, in the periodic auxiliary signal, the silent part and the unvoiced consonant part are 0.0, the vowel part is 0.9 and 1.0, and / b /, / d /, / g / etc. Data can be used in which the consonant portion in which the period and the non-period are mixed is a value of 0.3 to 0.7.

The present invention is not limited to the above-described embodiment, and can be realized with various configurations without departing from the spirit of the present invention. For example, the technical features in the embodiments corresponding to the technical features in each form described in the column of the outline of the invention are for solving the above-mentioned problems or for achieving a part or all of the above-mentioned effects. In addition, it is possible to replace or combine them as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be appropriately deleted.

10 ... Acquisition unit, 20 ... Generation unit, 21 ... Noise source, 22 ... Bandpass filter unit, 30 ... Learning unit, 100 ... Speech processing device, 200 ... Neural network, AI ... Auxiliary information, DA ... Data, L1- L4 ... dilation layer, N1 to N15 ... node, S1 to S15 ... sample, a1 to a24 ... first information, b ... second information, nz1 to nz24 ... aperiodic waveform signal

Claims (6)

It is a voice processing device
An acquisition unit that acquires acoustic features for generating audio waveforms,
The voice waveform is generated by inputting a periodic waveform signal corresponding to the fundamental frequency of the voice waveform into the neural network, inputting the acoustic feature amount, and performing conversion processing using the information output by the neural network. Equipped with a generator
The neural network outputs first information for generating aperiodic components and second information indicating periodic components.
The conversion process is a voice processing device that is a process of adding the information obtained by performing arithmetic processing using the first information and the aperiodic waveform signal and the second information. The voice processing device according to claim 1.
The first information is information indicating the strength of the aperiodic component for each predetermined frequency band.
The conversion process is a process of adding the information obtained by multiplying the aperiodic waveform signal for each frequency band by the corresponding first information and the second information. The voice processing device according to claim 1 or 2.
The generation unit is a voice processing device that further inputs a signal indicating the degree of presence / absence of a period according to a voice waveform to be generated to the neural network. The voice processing device according to any one of claims 1 to 3.
The generation unit is a voice processing device that inputs a plurality of periodic waveform signals having different phases to the neural network. The voice processing device according to any one of claims 1 to 4, and further.
A voice processing device including a learning unit that learns the relationship between the acoustic feature amount, the periodic waveform signal, the first information, and the second information by machine learning and reflects it in the neural network. It ’s a voice processing method.
The acquisition process to acquire the acoustic features for generating the audio waveform, and
The voice waveform is generated by inputting a periodic waveform signal corresponding to the fundamental frequency of the voice waveform into the neural network, inputting the acoustic feature amount, and performing conversion processing using the information output by the neural network. Equipped with a generation process
The neural network outputs first information for generating aperiodic components and second information indicating periodic components.
The conversion process is a voice processing method, which is a process of adding the information obtained by performing arithmetic processing using the first information and the aperiodic waveform signal and the second information.

Images