JP6578544B1 - Audio processing apparatus and audio processing method - Google Patents

Abstract

The present invention provides a technology capable of generating a speech waveform having a desired pitch with high quality. A speech processing device is a speech processing device, and acquires an acoustic feature amount for generating a speech waveform, and inputs a periodic waveform signal corresponding to the fundamental frequency of the speech waveform to a neural network. In addition, a generation unit that generates an audio waveform by inputting an acoustic feature amount and performing conversion processing using information output from the neural network is provided. The neural network outputs first information for generating an aperiodic component and second information indicating the periodic component, and the conversion processing is performed using the first information and the aperiodic waveform signal. This process is to add the second information and the second information. [Selection] Figure 1

Description

  The present invention relates to an audio processing device and an audio processing method.

  As a speech processing apparatus in recent years, one that generates a speech waveform using a neural network is known. In the techniques described in Non-Patent Document 1 and Non-Patent Document 2, a speech waveform is generated by a neural network using convolution. In the technique described in Non-Patent Document 3, the technique described in Non-Patent Document 1 and Non-Patent Document 2 is applied to generate a speech waveform from the acoustic feature amount.

A. van den Oord et al., “Wavenet: A General Model for Raw Audio”, arXiv preprint arXiv: 1609.003499, 2016 A. van den Oord et al., “Parallel WaveNet: Fast High-Fidelity Speech Synthesis”, arXiv preprint arXiv: 1711. 10433, 2017 Akira Tamori et al., “Speaker-dependent Wavenet vocoder”, In: INTERSPEECH, pp. 1118-1122, Aug. 2017 Takahiro Kaneko et al., “CycleGAN-VC: Non-parallel Voice Conversation Use-Cycle Persistent Networks,” 5th EURASIP Conferencep. 2114-2118

  In the technique described in Non-Patent Document 3, a speech waveform is generated using spectrum information, which is an acoustic feature amount, fundamental frequency information, or the like as auxiliary information. In the field of speech processing technology using such a neural network, a technology capable of generating a high-quality speech waveform and a technology capable of generating a speech waveform having a desired pitch are desired.

  The present invention has been made to solve the above-described problems, and can be realized as the following forms.

(1) According to an aspect of the present invention, an audio processing device is provided. This speech processing apparatus inputs an acoustic feature quantity for generating a speech waveform, and inputs a periodic waveform signal corresponding to the fundamental frequency of the speech waveform to a neural network and inputs the acoustic feature quantity. And a generation unit that generates the speech waveform by performing a conversion process using information output from the neural network. The neural network outputs first information for generating an aperiodic component and second information indicating the periodic component, and the conversion processing is performed using the first information and the aperiodic waveform signal. This is a process of adding the processed information and the second information. According to the speech processing apparatus of this aspect, the information obtained by performing the arithmetic processing using the first information for generating the non-periodic component and the non-periodic waveform signal and the second information indicating the periodic component are added together. Since the speech waveform is generated, a speech waveform having a desired pitch can be generated with high quality.
(2) In the audio processing according to the above aspect, the first information is information indicating the strength of a non-periodic component for each predetermined frequency band, and the conversion process is a non-periodic waveform signal for each frequency band. In addition, a process of adding information obtained by multiplying the corresponding first information and the second information may be performed. According to the speech processing apparatus of this aspect, information obtained by multiplying the first information indicating the strength of the aperiodic component for each corresponding frequency band by the aperiodic waveform signal for each predetermined frequency band, and the periodic component Therefore, a speech waveform with a desired pitch can be generated with high quality.
(3) In the speech processing apparatus of the above aspect, the generation unit may further input a signal indicating the degree of presence or absence of a period according to the speech waveform to be generated to the neural network. According to the speech processing apparatus of this aspect, for example, a high-quality speech waveform can be generated according to information related to an excitation source such as a silent portion of a speech waveform to be generated and a portion of a voiceless consonant.
(4) In the speech processing device of the above aspect, the generation unit may input a plurality of the periodic waveform signals having different phases to the neural network. According to the speech processing apparatus of this aspect, it is possible to generate a speech waveform having a desired fundamental frequency more effectively.
(5) In the speech processing device according to the above aspect, the learning unit further learns the relationship between the acoustic feature quantity, the periodic waveform signal, the first information, and the second information by machine learning and reflects the relationship in the neural network. May be provided. According to the speech processing device of this aspect, the relationship between the acoustic feature quantity and the speech waveform can be learned, and the learning result can be reflected in the generation unit.

  Note that the present invention can be realized in various modes. For example, a voice processing system using the voice processing apparatus of this embodiment, a method executed in the information processing apparatus for realizing the functions of the voice processing apparatus and the voice processing system, a computer program, and a server for distributing the computer program The present invention can be realized in the form of a device, a non-temporary storage medium storing the computer program, and the like.

It is explanatory drawing which shows the outline | summary of a speech processing unit. It is a figure which shows an example of the various parameters in an acoustic feature-value. It is explanatory drawing for demonstrating the neural network in 1st Embodiment. It is explanatory drawing for demonstrating the conversion process in the production | generation of an audio | voice waveform. It is a flowchart showing an audio | voice waveform generation process. It is explanatory drawing which shows the other aspect of a neural network. It is a figure which shows an example of the produced | generated audio | voice waveform. It is the figure which showed the experimental result of the subjective evaluation experiment. It is a figure which shows an example of a period auxiliary signal. It is a figure of an example of several periodic waveform signals from which a phase differs.

A. First embodiment:
FIG. 1 is an explanatory diagram showing an overview of a speech processing apparatus 100 according to an embodiment of the present invention. The speech processing apparatus 100 includes an acquisition unit 10, a generation unit 20, and a learning unit 30. The acquisition unit 10, the generation unit 20, and the learning unit 30 are realized in software by executing a program stored in the memory by one or more CPUs or GPUs. Note that some or all of these may be implemented in hardware by a circuit.

  The acquisition unit 10 acquires an acoustic feature amount for generating a speech waveform. Details of the acoustic feature amount will be described later. For example, the acquisition unit 10 may extract an acoustic feature amount from a voice waveform of a pre-recorded voice by using a well-known voice analysis technique, and an acoustic feature generated in advance according to a speech target text or a score. An amount may be obtained.

  The generation unit 20 includes a noise generation source 21 and a band pass filter unit 22. The noise generation source 21 generates an aperiodic waveform signal. An aperiodic waveform signal is a signal representing noise, for example, Gaussian noise. The band pass filter unit 22 performs a filtering process on the aperiodic waveform signal generated by the noise generation source 21 for each predetermined frequency band, and generates a plurality of aperiodic waveform signals having different frequency bands.

  The generation unit 20 generates a speech waveform by performing conversion processing using information output from a neural network having a plurality of output channels. The generation unit 20 inputs a periodic waveform signal corresponding to the fundamental frequency of the generated speech waveform to the input layer of the neural network, and inputs the acoustic feature amount acquired by the acquisition unit 10 to the neural network as auxiliary information, First information and second information are output.

  The periodic waveform signal is a periodic waveform signal corresponding to the fundamental frequency of the voice waveform to be generated. The periodic waveform signal may include an utterance style, a singing style, and the like. For example, when generating an audio waveform with vibrato added, a periodic waveform signal corresponding to the fundamental frequency in a state with vibrato added may be used. The periodic waveform signal is, for example, a sine waveform signal having the same frequency as the fundamental frequency of the voice waveform to be generated or a cosine waveform signal having a frequency one octave higher than the fundamental frequency of the voice waveform to be generated. Further, the periodic waveform signal may be a non-sinusoidal triangular wave, sawtooth wave, short diameter wave or pulse wave signal. The fundamental frequency of the voice waveform to be generated may be obtained from a voice waveform of a pre-recorded voice by using a well-known voice analysis technique, and is generated in advance according to the text or musical score to be spoken. Other fundamental frequencies may be used.

  The generation unit 20 performs a conversion process using the first information and the second information output from the neural network and the aperiodic waveform signal generated by the noise generation source 21 to generate a speech waveform. In the present embodiment, the generation unit 20 performs conversion processing using the first information, the second information, and the aperiodic waveform signal generated by the bandpass filter unit 22. The first information is information for generating an aperiodic component, and is information indicating the strength of the aperiodic component for each predetermined frequency band in the present embodiment. The second information is information indicating a periodic component, and more specifically is amplitude information obtained by sampling the periodic component for each sampling period. Details of the conversion process will be described later.

  The learning unit 30 learns the relationship between the acoustic feature quantity, the periodic waveform signal, the first information, and the second information by supervised machine learning or unsupervised machine learning (for example, see Non-Patent Document 4). Then, various parameters used in the neural network are optimized. In supervised machine learning, for example, natural speech of a speech waveform to be generated is used as teacher data, and learning is performed by comparison with the result of conversion processing using first information and second information. The learning unit 30 reflects the learning result on the neural network used by the generation unit 20. By doing so, the generation unit 20 can generate a speech waveform reflecting the learning result of the learning unit 30. The speech processing apparatus 100 may not include the learning unit 30. In this case, the generation unit 20 may generate a speech waveform from the first information and the second information by a conversion process described later, reflecting a learning result obtained by an external learning device or the like that performs machine learning. it can.

  FIG. 2 is a diagram illustrating an example of various parameters in the acoustic feature amount. In the present embodiment, the acoustic feature quantity is a voice feature quantity. Examples of the spectrum parameter include a mel cepstrum and a line spectrum pair (Line Spectrum Pair (LSP)). These are sometimes referred to as spectral information. There is a fundamental frequency as sound source information. The fundamental frequency is generally handled as a logarithmic fundamental frequency, and as related parameters, distinction between voiced / unvoiced and a non-periodic index can be considered. Since the unvoiced portion does not have a logarithmic fundamental frequency value, instead of including the voiced / unvoiced distinction in the sound source information, the voiced / unvoiced distinction may be performed by a method such as inserting a predetermined constant in the unvoiced portion. . Note that the fundamental frequency in the sound source information may be omitted because it is information included in the above-described periodic waveform signal. Also, the information regarding the distinction between voiced / unvoiced may be omitted because it is information included in the periodic auxiliary signal described later. Further, the spectrum information and the sound source information may include an utterance style, a singing style, and the like. For example, spectrum information in a state where a vibrato having a loudness is given can be used as the spectrum information.

  FIG. 3 is an explanatory diagram for describing a neural network used by the generation unit 20. The neural network 200 includes a plurality of dilation layers L1 to L4. The number of dilation layers can be arbitrarily determined. The “dilation layer” is also referred to as “expansion layer” or “intermediate layer”.

  The dilation layer L1 is a layer into which information is input. Hereinafter, it is also referred to as an “input layer”. The dilation layer L1 performs initial calculation processing and information convolution based on the input signal, and the dilation layers L2 to L4 perform information convolution based on information transmitted from the lower layer. Each layer includes a plurality of nodes.

  Generation of the first information and the second information by the neural network 200 will be described. In FIG. 3, the first information is indicated as “a1”, “a2”, and the second information is indicated as “b”. Hereinafter, this information is also referred to as “data”. In the present embodiment, 24 pieces of first information are generated by the neural network 200. The samples S1 to S8 of the periodic waveform signal are input to the nodes N1 to N8 in time series order after initial calculation processing is performed in the dilation layer L1. Each of the nodes N1 to N8 in the dilation layer L1 transmits information obtained by convolution of the information to the upper diation layer L2. For the sake of illustration, eight samples S1 to S8 of the periodic waveform signal are input to the dilation layer L1 shown in FIG. 3, but the number of input samples can be arbitrarily determined, for example, 3000 is there.

  In the dilation layers L <b> 2 to L <b> 4, various operations are performed in stages on the information transmitted from the input layer L <b> 1. Acoustic features corresponding to each sample are input as auxiliary information AI to the nodes N1 to N8 of the input layer L1 and the nodes of the dilation layers L2 to L4. In addition to the information transmitted from the lower layers, samples of periodic waveform signals may also be input to the dilation layers L2 to L4. In the dilation layer L4, by adding the finally calculated data and the data of the rightmost node of each layer, that is, the data of the node to which the earliest data is input in time series, the calculation process is performed. Data DA is output. In the present embodiment, the data DA is input in the time series, the first information a1 to a24 indicating the strength of the non-periodic component for each frequency band divided into 24 points at the time of the input sample S8. The second information b is amplitude information predicted as a periodic component of the audio sample at the time of the sample S8. The neural network 200 according to the present embodiment has a structure in which the closer the sample in the time series, the more easily the output data DA is affected. Specifically, the sample S8 is more likely to affect the prediction of the data DA than the sample S1.

  FIG. 4 is an explanatory diagram for explaining a conversion process in generating a speech waveform. The generation unit 20 adds the information obtained by multiplying the corresponding first information a1 to a24 to the aperiodic waveform signals nz1 to nz24 for each frequency band generated by the bandpass filter unit 22 and the second information b. A voice waveform is generated. The information obtained by multiplying the first information a1 to a24 by the aperiodic waveform signals nz1 to nz24 and the second information b may be added together, and the first information a1 to a24 may be added to the aperiodic waveform signals nz1 to nz1 respectively. The information multiplied by nz24 may be added and then the second information b may be added, or the information obtained by multiplying the first information a1 to a24 by the aperiodic waveform signals nz1 to nz24 and the second information b, respectively. You may add together. The frequency bands in the first information a1 to a24 and the non-periodic waveform signals nz1 to nz24 are bands divided every 1000 Hz, for example. The aperiodic waveform signals nz1 to nz24 are, for example, Gaussian noises with different frequency bands generated by the bandpass filter unit 22. In this embodiment, the frequency band is divided into 24, but the number of divisions is not limited to this.

  FIG. 5 is a flowchart showing a speech waveform generation process using the speech processing apparatus 100 in the present embodiment. First, the acquisition unit 10 acquires an acoustic feature amount in step S100. Next, in step S110, the generation unit 20 inputs the acoustic feature amount acquired in step S100 and a periodic waveform signal for a predetermined period to the neural network, and a non-periodic component for each predetermined frequency band. First information indicating the intensity of the second and second information indicating the periodic component are output. Finally, in step S120, the generation unit 20 performs a conversion process using the information output from the neural network in step S110 to generate a speech waveform.

  According to the audio processing device 100 of the present embodiment described above, the generation unit 20 performs processing using the first information for generating the non-periodic component and the non-periodic waveform signal, and the period. A speech waveform is generated by adding the second information indicating the component. More specifically, information obtained by multiplying the aperiodic waveform signal for each predetermined frequency band generated by the bandpass filter unit 22 with the first information indicating the strength of the aperiodic component for each corresponding frequency band. Since the speech waveform is generated by adding the second information indicating the periodic component, a speech waveform having a desired pitch can be generated with high quality. In addition, a speech waveform can be generated at a higher speed than a neural network having an autoregressive structure in which data output from the neural network itself is input to the neural network to predict the next data. Further, the learning unit 30 can learn the relationship between the acoustic feature quantity, the periodic waveform signal, the first information, and the second information, and the learning result can be reflected in the generation unit 20. Even if the speech waveform has a fundamental frequency greatly deviated from the learning range of the learning unit 30, the generation unit 20 generates a periodic waveform signal corresponding to the fundamental frequency of the speech waveform to be generated as an input layer of the neural network. Therefore, a speech waveform having a desired pitch can be generated.

  FIG. 6 is an explanatory diagram showing another aspect of the neural network. The neural network shown in FIG. 6 is configured by providing the neural network structure shown in FIG. 3 symmetrically. Similar to the first embodiment, a sample of a periodic waveform signal is input to the input layer L1. A sample of a past periodic waveform signal and a sample of a future periodic waveform signal in the time series of output data DA are input to the input layer L1 of the neural network of this embodiment. More specifically, information obtained by performing the initial calculation process on the samples S1 to S7 of the past periodic waveform signal is input to the nodes N1 to N7, and the node N8 is initialized to the sample S8 of the current periodic waveform signal. Information obtained by performing the arithmetic processing is input, and information obtained by performing the initial arithmetic processing on the samples S9 to S15 of the future periodic waveform signal is input to the nodes N9 to N15. In each node, as in the first embodiment, an acoustic feature amount is input as auxiliary information. The neural network 200 shown in FIG. 6 has a structure in which the closer the sample in the time series, the more easily the output data DA is affected. Specifically, the sample S8 tends to have a stronger influence on the prediction of the data DA than the sample S1 and the sample S15. By using such a neural network, not only past periodic waveform signal samples in the time series of data to be generated but also future periodic waveform signal samples are input, so that a higher quality speech waveform can be generated.

  FIG. 7 is a diagram illustrating an example of a speech waveform generated in the embodiment. The waveform shown in the upper part is a target speech waveform and is a waveform to be generated by speech processing. The waveform shown in the middle row is a speech waveform generated in the example. The waveform shown in the lower part is a periodic waveform signal input to the neural network, and is a sine waveform having the same basic frequency as the target speech waveform. As shown in FIG. 7, the speech waveform generated in the example fluctuates with the same period T and has the same fundamental frequency as the target speech waveform.

Experimental result:
FIG. 8 is a diagram showing an average opinion score (MOS), which is an experimental result of a subjective evaluation experiment on the generated speech waveform. In this experiment, the quality of the synthesized speech of the four methods was evaluated by a five-stage subjective evaluation experiment of “1: very bad, 2: bad, 3: normal, 4: good, 5: very good”. FIG. 8 shows scores of two methods out of the four methods. There were 16 subjects, and each subject evaluated 10 phrases for each technique from 10 songs as test data. The speech waveform of the synthesized speech to be evaluated was generated using the same acoustic feature amount in both methods.

  In the example, a speech waveform was generated using the speech processing apparatus 100 of the first embodiment described above and the neural network shown in FIG. In the comparative example, a speech waveform was generated by a vocoder technique using a neural network of wavenet (described in Non-Patent Document 1). In the wavenet neural network, the same acoustic features as in the example were input. As shown in FIG. 8, the score of the Example was higher than the score of the comparative example. That is, when the generation unit 20 generates a speech waveform according to the above embodiment, a speech waveform can be generated with higher quality. Note that the remaining two methods not shown in FIG. 8 are (1) the original voice of human singing is output as it is, and (2) the same technique as in the embodiment, in which the neural network in the embodiment is voiced. This is a speech waveform generation method using a learned neural network optimized by unsupervised machine learning (see, for example, Non-Patent Document 4) by the learning unit 30 of the processing device 100.

B. Second embodiment:
The generation unit 20 in the second embodiment further inputs a signal (hereinafter referred to as “period auxiliary signal”) indicating the degree of presence or absence of a period according to the speech waveform to be generated to the input layer of the neural network. The point which produces | generates an audio | voice waveform differs from 1st Embodiment. Since the configuration of the speech processing apparatus 100 of the second embodiment is the same as the configuration of the speech processing apparatus 100 of the first embodiment, description of the configuration is omitted.

  In the present embodiment, the generation unit 20 inputs the periodic waveform signal and the periodic auxiliary signal to the input layer L1 of the neural network. That is, in this embodiment, the node of the input layer of the neural network used by the generation unit 20 has two input channels. For example, a sample of the periodic waveform signal is input to the first channel, and a sample of the periodic auxiliary signal is input to the second channel. The order of channels can be determined arbitrarily.

  The periodic auxiliary signal can be determined in accordance with the boundary position where the periodic waveform starts and the boundary position where it ends, and can be expressed by a value of 0 to 1, where 0 is the non-periodic waveform portion and 1 is the periodic waveform portion. For example, the periodic auxiliary signal at the boundary position where the periodic waveform starts is a signal obtained by linear interpolation in 0.0 to 1.0 from the position 240 samples before the position where the boundary changes from unvoiced to voiced to the position 240 samples later. Yes, the periodic auxiliary signal at the boundary position where the periodic waveform ends is a signal that is linearly interpolated in units of samples from 1.0 to 0.0 from the position 240 samples before the position where the boundary changes from voiced to unvoiced to the position after 240 samples. is there. FIG. 9 is a diagram illustrating an example of the periodic auxiliary signal. Further, the periodic auxiliary signal may be data obtained by linearly interpolating a phoneme or a value for each frame.

  According to the speech processing apparatus 100 of the present embodiment described above, the generation unit 20 generates a speech waveform by inputting a periodic waveform signal directly to the input layer of the neural network, and thus a desired fundamental frequency is set. A speech waveform having the same can be generated. In addition, since the generation unit 20 further inputs a periodic auxiliary signal to the input layer of the neural network, for example, according to information on the excitation source such as a silent part of a speech waveform to be generated or a part of a voiceless consonant, A high-quality speech waveform can be generated.

C. Third embodiment:
The generation unit 20 in the third embodiment is different from the first embodiment in that a speech waveform is generated by inputting a plurality of periodic waveform signals having different phases to the input layer of the neural network. Since the configuration of the speech processing apparatus 100 of the third embodiment is the same as the configuration of the speech processing apparatus 100 of the first embodiment, description of the configuration is omitted.

  FIG. 10 is an example of a plurality of periodic waveform signals having different phases. In the present embodiment, the generation unit 20 inputs the periodic waveform signal Ws and the periodic waveform signal Wc to the input layer L1 of the neural network. That is, in this embodiment, the node of the input layer of the neural network used by the generation unit 20 has two input channels. A sample of the periodic waveform signal Ws is input to the first channel, and a sample of the periodic waveform signal Wc is input to the second channel. The order of channels can be determined arbitrarily.

  The periodic waveform signal Ws is a sine waveform having the same basic frequency as the voice waveform to be generated, and the periodic waveform signal Wc is a cosine waveform having the same basic frequency as the voice waveform to be generated. As shown in FIG. 10, in the periodic waveform signal Ws, the amplitude value at the timing t1 when rising and the value at the timing t2 when falling are both the amplitude A1, but at the timing t1. The periodic waveform signal Wc at is the amplitude A2, and the periodic waveform signal Wc at the timing t2 is the amplitude A3 having a value different from the amplitude A2. Therefore, the neural network used by the generation unit 20 is when the periodic waveform signal Ws has the amplitude A1, when the periodic waveform signal Wc is the amplitude A2, and when the periodic waveform signal Wc is the amplitude A3. Can be determined uniquely.

  According to the speech processing apparatus 100 of the present embodiment described above, the generation unit 20 inputs a plurality of periodic waveform signals having different phases to the input layer of the neural network. It is possible to uniquely determine whether the value of the waveform signal is a value when rising or a value when falling. Therefore, the generation unit 20 can more effectively generate a speech waveform having a desired fundamental frequency, and can generate a higher quality speech waveform.

D. Fourth Embodiment In the fourth embodiment, the structure of the neural network used by the generation unit 20 is different from that of the first embodiment. Since the configuration of the speech processing apparatus 100 according to the fourth embodiment is the same as the configuration of the speech processing apparatus 100 according to the first embodiment, description of the configuration is omitted.

  In the present embodiment, the generation unit 20 outputs the first information and the second information using a neural network having a structure in which a plurality of the neural networks shown in FIGS. 3 and 6 are vertically stacked. For example, when two neural networks are stacked, the generation unit 20 inputs the information output from the lower neural network to the input layer L1 of the upper neural network, and outputs the first information and the second information. . That is, the output of the lower neural network is prepared for the number of nodes in the input layer L1 of the upper neural network.

  According to the speech processing apparatus 100 of the present embodiment described above, the generation unit 20 generates a speech waveform by inputting a periodic waveform signal directly to the input layer of the neural network, and thus a desired fundamental frequency is set. A speech waveform having the same can be generated. In addition, since the generation unit 20 generates a speech waveform by obtaining the first information and the second information using a neural network having a structure in which a plurality of neural networks are stacked, compared to a neural network having a structure of only one stage, When generating the speech waveform by inputting the same number of samples, the neural network at each stage can be reduced. Therefore, since a speech waveform can be generated by inputting many samples without increasing the parameters as a whole, a higher-quality speech waveform can be generated.

E. Other embodiments:
(E1) In the above embodiment, the acoustic feature amount acquired by the acquisition unit 10 is a feature amount of singing voice. Instead of this, the acquisition unit 10 may acquire the feature amount of the spoken word as the acoustic feature amount. According to this embodiment, it is possible to generate a speech waveform that is a text synthesized speech that is not a singing voice. In addition, it is possible to generate a voice waveform that more accurately reproduces the tone, accent, intonation, and four voices in Chinese. Moreover, the acquisition part 10 may acquire the feature-value showing voice quality as an acoustic feature-value. The feature quantity representing the voice quality is an acoustic feature quantity extracted from another person's voice. According to this aspect, it is possible to generate a speech waveform that has undergone voice quality conversion for converting from an acoustic feature quantity of a certain speaker to an acoustic feature quantity of another speaker. When performing voice quality conversion, the acoustic feature quantity may be the acoustic feature quantity of the voice to be converted or the acoustic feature quantity of the voice to be converted. Further, the difference between these acoustic feature amounts may be used as the acoustic feature amount, or both may be used. As a periodic waveform signal, the neural network receives a periodic signal having the fundamental frequency of the speech to be converted and the speech to be converted, a periodic signal that is a residual signal of the speech to be converted, and a periodic signal having the fundamental frequency of the speech to be converted. May be. Further, the acquisition unit 10 may acquire a feature amount of a musical instrument sound as an acoustic feature amount and input it as auxiliary information to the neural network. According to this embodiment, it is possible to generate a speech waveform that is an instrument sound, not a singing voice. When generating the percussion instrument sound, the acquisition unit 10 acquires the characteristic amount of the percussion instrument sound, and uses a pulse signal that rises at a timing when the percussion instrument is desired to be generated as the periodic waveform signal. More specifically, when it is desired to generate a sound waveform of an eight beat hi-hat, a pulse signal that is 1 for every eighth note and 0 for others is used.

(E2) In the above embodiment, the acquisition unit 10 acquires the acoustic feature amount by converting the score feature amount and the language feature amount that are the basis of the speech waveform to be generated into a speech feature amount using a known conversion technique. May be. Further, the acquisition unit 10 may use, as the acoustic feature amount, information obtained by converting a score feature amount or a language feature amount using an arbitrary neural network. Further, the learning unit 30 simultaneously learns the neural network used for converting the score feature quantity and the language feature quantity and the neural network that outputs the first information and the second information in the above embodiment, and optimizes various parameters. May be.

(E3) In the above embodiment, the acoustic feature amount acquired by the acquisition unit 10 may include expression information in addition to the sound source information and the spectrum information. The expression information includes, for example, the period and amplitude of pitch vibrato and its presence in the case of singing, the period and amplitude and presence or absence of vibrato in the loudness, and the accent and intonation in the case of spoken language. In the case of sound, the degree of choking of the guitar and the presence or absence thereof are included. Instead of including the distinction of the presence / absence of the pitch vibrato in the singing expression information, the presence / absence of the vibrato of the pitch may be discriminated by a method such as inserting a predetermined constant in a portion where there is no pitch vibrato. Similarly, instead of including the distinction of the loudness of the vibrato in the singing expression information, distinguishing the presence of the loudness of the vibrato by a method such as inserting a predetermined constant in the part where the loudness is not vibrato May be performed.

  (E4) In the above embodiment, the node of the input layer L1 of the neural network may have two or more input channels. For example, two input channels are provided in the input layer L1, a sample of a periodic waveform signal is input to the first channel, and one of the samples input to the first channel in time series is input to the second channel. A sample of the periodic waveform signal at the previous time point may be input. The neural network may input a plurality of types of periodic waveform signal samples at the same time point in a plurality of input channels and output first information and second information to each channel. As a result, a speech waveform representing multiple speech or chords in which a plurality of voices are superimposed can be generated.

(E5) In the above embodiment, the generation unit 20 inputs a periodic waveform signal corresponding to the fundamental frequency of the voice waveform to be generated to the input layer of the neural network, and inputs an acoustic feature amount as auxiliary information to the neural network. Yes. The generation unit 20 may further input the aperiodic waveform signal to the input layer of the neural network.

(E6) In the above embodiment, the noise generation source 21 generates Gaussian noise as an aperiodic waveform signal. However, the present invention is not limited to this, and a signal representing other noise may be generated. The noise generation source 21 generates white noise, for example.

(E7) In the above embodiment, the generation unit 20 outputs the first information and the second information using one neural network. Instead, the generation unit 20 may output the first information and the second information using two neural networks, respectively. In this embodiment, the generation unit 20 outputs the second information output from the other neural network as a periodic waveform signal corresponding to the fundamental frequency of the generated speech waveform to the input layer of the one neural network that outputs the first information. May be entered.

(E8) In the above embodiment, the generation unit 20 is acoustic information such as a mel cepstrum or LSP (line spectrum pair), which is information for generating a non-periodic component, as the first information output using the neural network. A conversion process for generating a speech waveform may be performed by outputting a quantity and performing a calculation process using the first feature quantity, the second feature quantity, and the non-periodic waveform signal. For example, the generation unit 20 outputs, using a neural network, first information that is a 24-dimensional non-periodic component mel cepstrum and second information that is a one-dimensional periodic component. And the production | generation part 20 produces | generates an audio | voice waveform by producing | generating an aperiodic component by convolving 1st information with the aperiodic waveform signal produced | generated by the noise generation source 21, and adding it with 2nd information.

(E9) In the second embodiment, the generation unit 20 may further generate a speech waveform by inputting periodic waveform signals having different phases to the input layer of the neural network. That is, you may combine 2nd Embodiment and 3rd Embodiment. More specifically, the generation unit 20, for example, a periodic waveform signal Ws composed of a sine waveform having the same fundamental frequency as the speech waveform desired to be generated, and a periodic waveform signal composed of a cosine waveform having the same fundamental frequency as the speech waveform desired to be generated. Wc and the periodic auxiliary signal can be input to the input layer of the neural network.

(E10) In the second embodiment, the periodic auxiliary signal may be determined according to, for example, language information of a speech waveform to be generated. “Language information” is information on vowels and consonants, for example. The language information may be included in the acoustic feature amount. More specifically, in the periodic auxiliary signal, the silent part or the unvoiced consonant part is 0.0, the vowel part is 0.9 or 1.0, / b /, / d /, / g /, etc. Data in which the consonant part in which the period and the non-period are mixed has a value of 0.3 to 0.7 can be used.

  The present invention is not limited to the above-described embodiment, and can be realized with various configurations without departing from the spirit of the present invention. For example, the technical features in the embodiments corresponding to the technical features in each embodiment described in the summary section of the invention are for solving the above-described problems or achieving some or all of the above-described effects. In addition, replacement and combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 10 ... Acquisition part, 20 ... Generation part, 21 ... Noise generating source, 22 ... Band pass filter part, 30 ... Learning part, 100 ... Speech processing apparatus, 200 ... Neural network, AI ... Auxiliary information, DA ... Data, L1 L4 ... dilation layer, N1-N15 ... node, S1-S15 ... sample, a1-a24 ... first information, b ... second information, nz1-nz24 ... non-periodic waveform signal

Claims (6)

A voice processing device,
An acquisition unit for acquiring an acoustic feature for generating a speech waveform;
A periodic waveform signal corresponding to the fundamental frequency of the speech waveform is input to the neural network, the acoustic feature is input, and the speech waveform is generated by performing conversion processing using information output from the neural network. A generating unit
The neural network outputs first information for generating an aperiodic component and second information indicating the periodic component;
The speech processing apparatus, wherein the conversion process is a process of adding information obtained by performing arithmetic processing using the first information and the non-periodic waveform signal and the second information. The speech processing apparatus according to claim 1,
The first information is information indicating the strength of an aperiodic component for each predetermined frequency band,
The audio processing device, wherein the conversion process is a process of adding information obtained by multiplying the corresponding first information to the non-periodic waveform signal for each frequency band and the second information. The speech processing apparatus according to claim 1 or 2,
The speech processing apparatus, wherein the generation unit further inputs a signal indicating a degree of presence / absence of a period corresponding to a speech waveform to be generated to the neural network. The speech processing apparatus according to any one of claims 1 to 3, wherein
The speech processing apparatus, wherein the generation unit inputs a plurality of the periodic waveform signals having different phases to the neural network. The speech processing apparatus according to any one of claims 1 to 4, further comprising:
A speech processing apparatus, comprising: a learning unit that learns a relationship between the acoustic feature value, the periodic waveform signal, the first information, and the second information by machine learning and reflects the relationship in the neural network. An audio processing method,
An acquisition step of acquiring an acoustic feature for generating a speech waveform;
A periodic waveform signal corresponding to the fundamental frequency of the speech waveform is input to the neural network, the acoustic feature is input, and the speech waveform is generated by performing conversion processing using information output from the neural network. A production process to
The neural network outputs first information for generating an aperiodic component and second information indicating the periodic component;
The audio processing method, wherein the conversion process is a process of adding information obtained by performing arithmetic processing using the first information and an aperiodic waveform signal and the second information.

Images