JP5549166B2 - Audio processing device, program - Google Patents

Description

  The present invention relates to a speech processing apparatus and program for estimating a note period representing a period that can be regarded as one note in an input speech.

2. Description of the Related Art Conventionally, a voice processing device that specifies a note period representing a period that can be regarded as one note from an input voice is known (see, for example, Patent Document 1).
In the speech processing device described in Patent Document 1, if the average sound pressure in two analysis sections that are continuous along time progress increases by a predetermined value or more, the two analysis sections that are continuous are increased. Among them, the previous analysis interval is specified as the note start timing in time progress. Based on these specified note start timings, the interval between two note start timings that are continuous along the time progress of the input speech is estimated as a note period (hereinafter, such a note period estimation technique is conventionally known). Called estimation technology).

Japanese Patent No. 4128848

  By the way, when vibrato is applied to the input voice, the transition along the time axis of the sound pressure of the input voice (hereinafter referred to as the sound pressure transition) continuously repeats a convexity upward and a convexity downward. fluctuate. In regions where the sound pressure transition is convex upward, the average sound pressure in two consecutive analysis sections may increase by more than a specified value in each region.

  In this case, in the conventional estimation technique, each region in which the sound pressure transition is convex upward is specified as the note start timing in the utterance period in vibrato (hereinafter referred to as the vibrato period). Then, in the conventional estimation technique, the vibrato period is divided into a plurality of note periods that should not be estimated, and each of the divided periods is estimated as a note period.

That is, the conventional estimation technique has a problem that the note period cannot be accurately estimated.
Therefore, an object of the present invention is to improve estimation accuracy in a technique for estimating a note period.

The present invention, which has been made to achieve the above object, estimates a note period representing each period that can be regarded as one note from input speech that is continuous over time , specifies a pitch in the note period, and records music. Is a voice processing device.
In the speech processing apparatus of the present invention, the sound pressure transition specifying means specifies a sound pressure transition representing a transition along the time progression of the sound pressure in the input sound from the input sound, and the start timing detecting means is the specifying In the section where the sound pressure transition is monotonically increasing, the rate of increase of the sound pressure during the first specified period specified in the sound pressure transition first becomes equal to or greater than a predetermined value as time progresses. Each time is detected as a note start timing. However, the note start timing here is the start timing of the note period.

  And in the speech processing device of the present invention, the vibrato period specifying means specifies the vibrato period based on the pitch transition representing the transition along the time progression of the pitch in the input voice, and the in-period timing removing means is Of the note start timings detected by the start timing detection means, note start timings corresponding to the vibrato period (that is, timing within the period) are removed from the detection results of the start timing detection means. However, the vibrato period here is a period uttered by vibrato in the input voice.

Furthermore, in the speech processing apparatus of the present invention, the note period estimation means specifies the note end timing paired with each note start timing after the in-period timing is removed, and the paired note start timing and note note. Each period between the end timings is estimated as a note period.
Further, in the vibrato period specifying means in the present invention, if the increase / decrease detecting means has a fluctuation range of the pitch in the period pitch transition equal to or less than a specified width, an increasing section in which the pitch increases in the period pitch transition, and If a decrease section in which the pitch decreases is detected, and the number of detected increase sections and decrease sections is equal to or greater than the specified number, the period specifying means sets the second specified period corresponding to the period pitch transition as the vibrato period. Is configured to identify. Note that the period pitch transition here is a pitch transition in the second specified period. However, the second specified period is each of a plurality of periods defined so as to be continuous with each other over the entire pitch transition and along the time progress.

  That is, in the speech processing apparatus of the present invention, only the note start timing remaining after the in-period timing is removed from all the note start timings is used as the note period start timing.

  Therefore, according to the speech processing apparatus of the present invention, it is possible to prevent the use of the note start timing included in the vibrato period when estimating the note period. As a result, according to the speech processing apparatus of the present invention, at least the note period starting from the vibrato period is not estimated, and one vibrato period is divided into two or more note periods and estimated. Can be prevented.

In other words, according to the speech processing apparatus of the present invention, it is possible to improve the estimation accuracy of the note period in the input speech as compared with the conventional estimation technique.
The first specified period referred to here may be a period shorter than the entire section in which the sound pressure transition is monotonically increasing, or may be the entire section having a monotonically increasing period.

  According to the vibrato period specifying means configured as described above, the second specified period that satisfies the condition from the start point to the end point of the input voice along the time progress can be specified as the vibrato period.

By the way, the note period estimation means first calculates the sound pressure fluctuation time point when the sound pressure in the sound pressure transition becomes equal to or lower than the sound pressure at the note start timing after the note start timing after the in-period timing is removed. It may be configured to be specified as a note end timing paired with the note start timing.

According to the end timing estimation means configured as described above, the sound generation end timing can be directly estimated from the sound pressure transition.
When the note period estimating means is configured as described above, the time progress point indicating the note end timing may be estimated before removing the in-period timing, or the in-period timing is removed. It may be the one estimated after.

However, when the note period estimation means is configured as described above , there is a possibility that a section where the sound pressure monotonously decreases in the vibrato period is estimated as the note end timing.

Order to prevent this, the sound marks period estimating means, after the note start timing after the period in the timing has been removed, and the initial sound pressure variation point in the later end timing of vibrato periods, the sound marks the start timing and the pair It may be considered as the note end timing.

By the way, when a person sings, the sound pressure of the input voice rises without decreasing to the sound pressure at the note start timing, and utterance for the next sound may be started. Under such circumstances, sound marks the end timing itself may not be identified.

For this reason, in the present invention, the note period estimation means may be configured to specify a time point that is a set time length before the later start timing as a note end timing paired with the preceding start timing. However, the pre-start timing here refers to the note start timing that precedes the time progress among the note start timings adjacent to the time progress after the in-period timing is removed. This is the later note start timing in time progression.

  According to the note period estimating means configured as described above, even if the note end timing cannot be estimated from the input voice, the note end timing can be estimated from the note start timing. Thereby, according to the speech processing device of the present invention, it is possible to prevent a situation in which the note end timing is not estimated.

  In particular, in the speech processing apparatus of the present invention configured as described above, if the set time length is set to a time short enough to be regarded as 0 [s], the specified note end timing is after the end timing of the vibrato period. As a result, the entire vibrato period can be included in one independent note period.

Incidentally, the sound marks period estimating means in the present invention, in addition, the traveling time than the rear start timing before, the sound pressure in the sound pressure transition is, the sound pressure fluctuation when it becomes the lower the sound pressure or before the start time If present, it is desirable that the sound pressure fluctuation time point be specified as the note end timing paired with the previous start timing.

  According to the speech processing apparatus of the present invention configured as described above, even if the method for specifying the note end timing from the sound pressure transition and the method for specifying the note end timing from the note start timing are combined, One note end timing can be specified with respect to the note start timing.

Further, when the note period estimating means is configured as described above, there is a possibility that the note end timing paired with the last note start timing is not specified along with the time progress of the input voice.
Therefore, note duration estimating means of the present invention, the end along the time progression of the input speech, of the notes start timing after the period in the timing has been removed, the last note start timing along the time progression You may comprise so that it may identify as a note end timing which becomes a pair.

According to the speech processing device of the present invention configured as described above, in order to identify the end of the input speech along the time as the note end timing, the note end timing paired with the last note start timing along the time progression Can be prevented from being identified.
Furthermore, the speech processing apparatus of the present invention includes basic frequency detection means, discontinuity detection means, frequency correction means, and pitch estimation means.
The fundamental frequency determining means determines the speech fundamental frequency in each unit section along the time axis in the input speech, and the discontinuity detecting means determines the speech fundamental frequency detected by the fundamental frequency determining means. Among the frequency transitions arranged along the time axis, a discontinuous region in which the speech fundamental frequency is discontinuous between consecutive unit sections is detected.
Further, the frequency correction means is configured such that the fundamental frequency of each unit section corresponding to the discontinuous area detected by the discontinuity detecting means is the unit immediately before the discontinuous area, with respect to the frequency transition specified by the sound pressure transition specifying means. The frequency correction is performed so that the frequency immediately before the speech fundamental frequency of the section reaches the immediately following frequency that is the speech fundamental frequency of the unit section immediately after the discontinuous region. The voice fundamental frequency in the unit section after the frequency correction by the means is executed is specified as the pitch in the note period corresponding to the unit section.

The present invention may be implemented as a program that is executed by a computer.
However, the program of the present invention, from the input speech, the first predetermined period at the sound pressure changes the specific procedures for determining the sound pressure changes in the input speech, the identified sound pressure changes are monotonically increasing interval A start timing detection procedure that detects each point when the rate of increase in sound pressure first exceeds the specified value as time progresses, and a vibrato period that identifies the vibrato period based on the pitch transition Let the computer execute specific procedures. Furthermore, the program of the present invention uses the note start timing corresponding to the vibrato period as the in-period timing among the note start timings detected by the start timing detection procedure, and detects from the detection results in the start timing detection procedure. Identifying a note end timing paired with each of the note start timings after the removal of the intra-period timing in the intra-period timing removal procedure for removing the intra-period timing and the intra-period timing removal procedure, It is necessary to cause the computer to execute a note period estimation procedure for estimating each period between the note start timing and the note end timing as a note period.
Furthermore, in the program of the present invention, each of a plurality of periods defined so as to be continuous with each other over the entire pitch transition in the vibrato period specifying procedure is set as the second specified period. If the pitch transition in the specified period is the pitch change in the specified period, and the fluctuation range of the pitch in the period pitch transition is less than or equal to the predefined specified range, the increase in pitch will increase in the period transition Increase / decrease detection procedure for detecting the interval and the decrease interval in which the pitch decreases, and if the number of the increase interval and the decrease interval detected in the increase / decrease detection procedure is equal to or greater than a predetermined number, It is necessary to cause the computer to execute a period specifying procedure for specifying the second specified period corresponding to the transition as the vibrato period.

  If the present invention is a program made in this way, for example, it is recorded on a computer-readable recording medium such as a DVD-ROM, CD-ROM, hard disk, flash memory, etc., and loaded into the computer as needed to start. In addition, the computer can be used by being acquired and started up via a communication line as necessary. And by making a computer perform each procedure, the computer can be functioned as an audio | voice processing apparatus described in Claim 1.

It is a block diagram which shows schematic structure of a music search system. It is the flowchart which showed the process sequence of the music search process. It is the flowchart which showed the process sequence of the pitch estimation process. It is explanatory drawing which showed the detection method of the correlation peak typically. It is explanatory drawing which showed typically the determination method of the audio | voice basic frequency f0. It is the flowchart which showed the process sequence of the reliability calculation process. It is explanatory drawing which illustrated the derivation process of f0 candidate reliability. It is the flowchart which showed the processing procedure of f0 correction processing. It is explanatory drawing for demonstrating the operation example of f0 correction process. It is the flowchart which showed the process sequence of the start / end timing estimation process. It is explanatory drawing which illustrated the estimation process of the start / end timing. It is explanatory drawing which illustrated the estimation process of the start / end timing. It is the flowchart which showed the processing procedure of the music transcription process. It is explanatory drawing which illustrated the determination process of the note pitch in music transcription processing. It is the flowchart which showed the process sequence of the transcription result collation process.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
First, FIG. 1 is a block diagram showing a schematic configuration of a music search system provided with a voice processing device to which the present invention is applied.
<About the music search system>
The music search system 1 searches for a music (hereinafter referred to as an intended expected music) that is estimated to be intended by the user when inputting the voice from the input voice inputted by the user. It is.

  For this reason, as shown in FIG. 1, the music search system 1 records the input voice with the server 40 that stores the music data prepared in advance for each music, and collates the scored result with the music data. And an audio processing device 20 for searching for an expected song. The voice processing device 20 is connected to the server 40 via a network (for example, a dedicated line or WAN).

Among these, the server 40 is a known service server device mainly composed of an information processing apparatus including a storage device 41 for storing music data and a known microcomputer 42 having at least a ROM, a RAM, and a CPU. It is.
<About music data>
Next, music data stored in the storage device 41 will be described.

  This music data includes music information that is data for identifying the music, time information that indicates the time required from the start of performance of the music to the end of performance, and a guide melody that is data relating to the melody of the music. doing.

The song information includes at least song number data for specifying a song and song name data indicating the song name of the song.
The guide melody is well-known data that represents the pitch and the tone value of each constituent sound that forms the main melody (hereinafter referred to as a reference melody) of the music. Specifically, the tone lengths of the constituent sounds in the present embodiment are represented by a tone output start time and a tone output end time. However, the musical sound output start time here is the time from the start of the performance of the music until the output of the constituent sound starts, and the musical sound output end time is the time until the output of the constituent sound ends. This is the time from the start of the performance of the music. That is, the time length between the musical sound output start time and the musical sound output end time is the sound length of the constituent sound.

Hereinafter, in the guide melody, information representing the pitch and the note value of each constituent sound is referred to as reference note data. However, the reference note data is such that the pitch and the note value of each constituent sound are associated with the order of time progression in the reference melody of the constituent sound.
<About the audio processor>
Next, the voice processing device 20 will be described.

  Returning to FIG. 1, the audio processing device 20 includes a communication unit 21, a display unit 22, an operation receiving unit 23, a microphone 24, an audio input unit 25, an audio output unit 26, a speaker 27, A storage unit 28 and a control unit 30 are provided.

  Among these, the communication unit 21 connects the voice processing device 20 to a network (for example, a dedicated line or WAN), and communicates with the outside (that is, the server 40) via the connected network. It is.

  And the display part 22 is a known display apparatus comprised from the liquid crystal display etc., for example. The operation receiving unit 23 includes a known input device such as a keyboard and a pointing device (for example, a mouse).

  The microphone 24 is a well-known device for inputting sound. The voice input unit 25 is configured as an AD converter that samples the voice (analog signal) input via the microphone 24 and inputs the sampled value (sample value) to the control unit 30. Hereinafter, the sound converted into the sampling value by the sound input unit 25 is referred to as sound data.

  Furthermore, the audio output unit 26 is configured to output a control signal based on a command from the control unit 30 to the speaker 27. And the speaker 27 is comprised so that the control signal from the audio | voice output part 26 may be converted into a sound and emitted.

  The storage unit 28 is a storage device (for example, a hard disk drive) configured to hold the stored content even when the power is turned off and to be able to read and write the stored content, and the server 40 via the program and the communication unit 21. The music data acquired from is stored.

Next, the control unit 30 is configured around a known microcomputer having at least a ROM 31, a RAM 32, and a CPU 33.
Of these, the ROM 31 stores programs and data that need to retain stored contents even when the power is turned off. The RAM 32 temporarily stores programs and data. The processing program from the storage unit 28 is transferred and stored in the RAM 32.

  And CPU33 performs each process (various calculation) according to the processing program memorize | stored in ROM31 or RAM32, and each part 21,22,23,25 (24), 26 (27), 26 which comprises the audio | voice processing apparatus 20 is carried out. The control for 28 is executed.

In the present embodiment, as the processing program, based on the input voice input by the user through the microphone 24, voice note data obtained by recording the input voice is generated, and the generated voice note data is used as the reference note data. There is prepared for the control unit 30 (more precisely, the CPU 33) to execute a music search process for searching for an intended expected music in accordance with the result collated.
<About music search processing>
Next, a music search process executed by the control unit 30 will be described.

Here, FIG. 2 is a flowchart showing a processing procedure of the music search processing.
The music search process is started when an activation command is received via the operation reception unit 23 after at least one piece of audio data based on the input voice input via the microphone 24 is stored in the storage unit 28. The input voice here is continuous (continuous) for a certain time or more as time progresses.

As shown in FIG. 2, when the music search process is started, first, one piece of voice data is acquired from the voice data stored in the storage unit 28 in S110.
In S120, well-known downsampling, DC component removal, noise removal processing, compressor processing, and normalization are executed as pre-processing on the audio data acquired in S110. Hereinafter, the audio data that has been pre-processed in S120 is referred to as processed audio data.

  In S130, a pitch estimation process for estimating the pitch (sound fundamental frequency f0) of the input voice in the unit section is executed for each unit section defined along the time progress of the input voice in the processed voice data. To do.

  Furthermore, in S150, start / end timing estimation processing is performed for estimating the start timing and end timing of each sound generation period, which is a period in which utterance is continued at a predetermined sound pressure or higher in the input voice. Hereinafter, the start timing and end timing estimated in the start / end timing estimation processing are referred to as a sound generation start timing and a sound generation end timing, respectively.

  Subsequently, in S190, a period that can be regarded as one note (hereinafter referred to as a note period) is estimated based on the sound generation start timing and the sound generation end timing estimated in S150, and the pitch (hereinafter referred to as a note period) is estimated. , Which is referred to as a note pitch) is performed based on the sound fundamental frequency f0 for each unit section estimated in S130. As a result of this transcription processing, the duration of each note period (that is, the note length or the note value obtained by quantizing the note length) and the note pitch are associated with each other, that is, input as voice note data. Data in which voice is converted into musical notes is generated.

  In S210, the voice note data generated in S190 is collated with reference note data, and an intended expected song is identified based on the collation result, and the identified expected song is used by the voice processing device 20. The transcription result collation process notified to the person is executed.

Thereafter, the music search process ends.
<Pitch estimation processing>
Next, the pitch estimation process started in S130 of the music search process will be described.

Here, FIG. 3 is a flowchart showing a processing procedure of the pitch estimation processing.
As shown in FIG. 3, when the pitch estimation process is started, the processed voice data is subjected to frequency analysis in S310. As this frequency analysis, in this embodiment, a sample value of a predetermined number of samplings in the processed audio data is subjected to FFT (Fast Fourier Transform). Note that as many sample values as the number of samplings are repeatedly acquired from the start to the end of the processed audio data while overlapping a part along the time progress. Thereby, the amplitude spectrum (that is, the distribution of frequency components) of the input speech is derived for each unit interval corresponding to the number of samplings.

In S330, based on the amplitude spectrum derived in S310, an autocorrelation value representing the probability that the frequency component of the amplitude spectrum is a fundamental frequency component is derived.
Specifically, the product sum of the amplitude value in each frequency component of one amplitude spectrum and the amplitude value in the frequency component increased from the frequency component in the amplitude spectrum by a specified frequency width is derived as an autocorrelation value. ing. For this reason, the autocorrelation value that is derived every time the specified frequency width is displaced becomes a large value when the fundamental frequency component or the harmonic component of the fundamental frequency matches when the specified frequency width is displaced.

In S330, an average value of all autocorrelation values derived from the amplitude spectrum (hereinafter referred to as autocorrelation average value) is also derived for each amplitude spectrum (that is, unit interval).
Further, in S350, the autocorrelation value derived in the previous S330 is smoothed and differentiated to detect a section f0 candidate representing a frequency that is a candidate for the speech fundamental frequency f0 in each unit section.

  The section f0 candidate is a frequency corresponding to the maximum value (hereinafter referred to as a correlation peak) in the locus of the autocorrelation value, as shown in FIG. The locus of autocorrelation values here is obtained by arranging autocorrelation values for each specified frequency along the frequency axis.

  However, in this embodiment, only the correlation peak whose autocorrelation value is equal to or greater than the autocorrelation average value is set as the section f0 candidate (that is, in the example shown in FIG. 4, the fourth correlation peak is less than the autocorrelation average value). Therefore, the frequency corresponding to the fourth correlation peak is not detected as the section f0 candidate). Furthermore, in S350 in the present embodiment, the value of the autocorrelation value in the frequency component that is not detected as the section f0 candidate is 0.

This step S350 is repeatedly executed until all the unit sections defined in the processed voice data are finished.
In subsequent S370, a reliability calculation process is performed to calculate f0 candidate reliability representing the likelihood of each of the section f0 candidates detected in S350 as the speech fundamental frequency f0. The f0 candidate reliability derived by this reliability calculation process becomes a larger value as the likelihood is higher.

  This reliability calculation process is executed for each unit section. For this reason, f0 candidate reliability is calculated about each section f0 candidate derived | led-out from one amplitude spectrum by one reliability calculation process.

  In S390, based on the f0 candidate reliability for each unit section derived in S370, the speech fundamental frequency f0 in the unit section is determined. The section f0 candidate determined as the speech fundamental frequency f0 corresponds to the f0 candidate reliability having the highest value among all the f0 candidate reliability derived in S370, as shown in FIG. (In the example shown in FIG. 5, the first section f0 candidate is determined as the voice fundamental frequency f0).

  However, in S390 of the present embodiment, the value of the f0 candidate reliability that is less than the predetermined reliability threshold is 0 (in the example illustrated in FIG. 5, the f0 candidate reliability of the third section f0 candidate is Since it is less than the reliability threshold, the value of the f0 candidate reliability is 0). And if all the f0 candidate reliability in a unit area is less than a reliability threshold value, the audio | voice fundamental frequency f0 in the unit area will be determined to 0 [Hz]. That is, when the f0 candidate reliability is low, the section f0 candidate corresponding to the f0 candidate reliability is not determined as the speech fundamental frequency f0.

Further, in S410, it is determined whether or not the steps of S370 and S390 have been executed for all the unit sections defined in the processed voice data.
As a result of the determination in S410, if S370 and S390 are not executed for all unit sections, the process returns to S370. In S370 thus shifted, the f0 candidate reliability is calculated for the next unit interval along the time progress in the processed speech data from the unit interval for which the f0 candidate reliability was calculated in the previous S370, Thereafter, the process proceeds to S390.

And when execution of S370 and S390 is completed about all the unit sections, it progresses to S430.
In S430, f0 correction processing for correcting the sound fundamental frequency f0 for each unit section determined in S390 is executed. This f0 correction processing is executed for a discontinuous region in which the voice fundamental frequency f0 can be regarded as discontinuous in a frequency transition distribution in which the voice fundamental frequency f0 for each unit section is arranged along the time progression in the input voice. The

  In S450, the voice fundamental frequency f0 in each unit section after the correction is performed in S430 is quantized in semitone units. Thereby, the voice fundamental frequency f0 is attracted for each semitone. Since this quantization is a well-known process, a detailed description thereof is omitted here.

Thereafter, the pitch estimation process is terminated, and the process proceeds to S150 of the music search process.
<Reliability calculation process>
Next, the reliability calculation process started in S370 of the pitch estimation process will be described.

Here, FIG. 6 is a flowchart showing a processing procedure of the reliability calculation processing.
As shown in FIG. 6, when the reliability calculation process is started in S370 of the pitch estimation process, in S3710, it is included in the specific frequency band from all the sections f0 candidates in one unit section. The autocorrelation value of the section f0 candidate (hereinafter referred to as a specific f0 candidate) is extracted. However, the specific f0 candidate in the present embodiment corresponds to the lowest frequency among the sections f0 candidates included in the specific frequency band. The specific frequency band is a frequency band from a lower limit frequency to an upper limit frequency that is automatically defined by deriving an autocorrelation value.

  In S3720, one autocorrelation value of a section f0 candidate (hereinafter referred to as a harmonic f0 candidate) included in the harmonic range of the specific f0 candidate corresponding to the autocorrelation value extracted in S3710 is acquired. However, the overtone range is a frequency range defined so as to sandwich the overtone component around the overtone component of the specific f0 candidate corresponding to the autocorrelation value acquired in S3710.

  In S3730, the autocorrelation value of the specific f0 candidate extracted in S3710 is subtracted from the autocorrelation value of the overtone f0 candidate acquired in S3720. Then, the subtraction result is newly defined (that is, changed) as the autocorrelation value of the harmonic overtone f0 candidate acquired in S3720.

  Subsequently, in S3740, it is determined whether or not the step of S3730 has been executed for the autocorrelation values of all overtone f0 candidates in one unit section. As a result of the determination, if the step of S3730 is not executed for the autocorrelation values of all overtone f0 candidates, the process returns to S3720.

  In S3720 thus shifted, the section f0 candidate included in the next higher harmonic range of the section f0 candidate corresponding to the autocorrelation value acquired in the previous S3720 is set as the harmonic f0 candidate, and the autocorrelation value of the harmonic f0 candidate is set. Is acquired and it progresses to S3730.

  In other words, by repeating the steps from S3720 to S3740, as shown in FIG. 7A, the autocorrelation value of the harmonic overtone f0 candidate is specified f0 from the value derived in S330 in the previous pitch estimation process. The candidate autocorrelation value is changed to a subtracted value.

  In S3750, the autocorrelation value is multiplied by an attenuation coefficient. As shown in FIG. 7B, the attenuation coefficient increases as the frequency corresponding to the autocorrelation value to be multiplied decreases, and decreases as the frequency increases.

  However, the autocorrelation values multiplied by the attenuation coefficient are all in the unit interval including the autocorrelation value of the specific f0 candidate and the autocorrelation values of all overtone f0 candidates changed by repeating the steps from S3720 to S3740. Auto-correlation value of the section f0 candidate.

  In S3760, the autocorrelation value after the attenuation coefficient is multiplied in S3750 is multiplied by the spectrum amplitude value of the section f0 candidate corresponding to each autocorrelation value. Then, the multiplication result is derived as the f0 candidate reliability for each section f0 candidate.

  Note that the autocorrelation values corresponding to frequency components other than the section f0 candidate (hereinafter referred to as non-candidate frequencies) are set to 0 in S350 in the previous pitch estimation process. For this reason, even if the autocorrelation value of the non-candidate frequency is multiplied by the attenuation coefficient in S3750 or the calculation of the f0 candidate reliability in S3760 is performed, the calculation result becomes zero.

Therefore, only the f0 candidate reliability for the section f0 candidate in the unit section is calculated by the multiplication of the attenuation coefficient in S3750 and the calculation of the f0 candidate reliability in S3760.
Then, this reliability calculation process is complete | finished and it returns to S390 of a pitch estimation process.

In other words, in the reliability calculation process of the present embodiment, the autocorrelation value of each frequency component is multiplied by the attenuation coefficient, so that the autocorrelation of the section f0 candidate in the high frequency band in which the harmonic component of the speech fundamental frequency f0 is highly likely to be included. The value is suppressed. Therefore, the f0 candidate reliability obtained by multiplying the autocorrelation value whose value is suppressed by the amplitude value of the section f0 candidate corresponding to each autocorrelation value becomes larger as it corresponds to the frequency component of the fundamental frequency.
<About f0 correction processing>
Next, the f0 correction process started in S430 of the pitch estimation process will be described.

Here, FIG. 8 is a flowchart showing the processing procedure of the f0 correction processing.
When this f0 correction process is started as shown in FIG. 8, first, in S4310, one unit section is selected from all the unit sections in which the frequency analysis was executed in S310 of the previous pitch estimation process. Select. In S4310, each time a transition is made to S4310, the unit sections are selected one by one along the time progress in the processed voice data from the start of the processed voice data.

Subsequently, in S4320, it is determined whether or not the voice basic frequency f0 in the unit section selected in the previous S4310 is 0 [Hz].
As a result of the determination, if the audio fundamental frequency f0 is 0 [Hz], the process proceeds to S4330. In S4330, the section counter is incremented by 1, and the process returns to S4310.

  That is, by executing the steps from S4310 to S4330, a unit interval (hereinafter referred to as a non-regular frequency interval) in which the audio fundamental frequency f0 is 0 [Hz] continues along the time progress of the processed audio data. The number to do is measured.

On the other hand, if the result of determination in S4320 is that the audio fundamental frequency is a frequency other than 0 [Hz], the flow proceeds to S4340.
In S4340, the voice basic frequency f0 that has been set as the first section f0 until the transition to S4340 this time is set as the second section f0. The frequency f0 is set as the first section f0. In other words, when the process proceeds to S4340, the voice fundamental frequency f0 in the unit section close to the voice start is set as the second section f0 among the voice fundamental frequencies f0 acquired along with the time progress of the processed voice data. The voice fundamental frequency f0 in the unit interval close to the end is set as the first interval f0.

Subsequently, in S4350, it is determined whether or not the count value, which is the value of the section counter, is equal to or greater than a first specified value specified in advance.
If the result of determination in S4350 is that the count value is greater than or equal to the first specified value, processing proceeds to S4360. That is, if the number of non-normal frequency sections that are continuous along the time progress of the processed audio data is equal to or greater than the first specified value, the continuous non-normal frequency sections are detected as discontinuous regions in the frequency transition distribution. . Hereinafter, the non-regular frequency interval that continues for the first specified value or more is referred to as a long-term discontinuous region.

  In S4360, in the time progress of the processed voice data, the basic voice frequency from the unit section immediately after the unit section corresponding to the second section f0 to the unit section immediately before the unit section selected in the latest S4310. Correction is made so that f0 becomes the second interval f0. Thereafter, the process proceeds to S4390.

That is, in S4360, the sound fundamental frequency f0 in the non-regular frequency section forming the long-term discontinuous region is changed from 0 [Hz] to the second section f0.
By the way, if the result of determination in S4350 is that the count value is less than the first specified value, processing proceeds to S4370. In S4370, it is determined whether or not the count value is 1 or more. As a result of the determination, if the count value is 1 or more, the process proceeds to S4380. That is, if the number of non-regular frequency sections that are continuous along the time progress of the input speech is one or more and less than the first specified value, the non-normal frequency sections that are consecutive are regarded as non-uniform in the frequency transition distribution. Detect as a continuous area. Hereinafter, one or more non-regular frequency intervals that are continuous with less than the first specified value are referred to as short-term discontinuous regions.

  Then, in S4380, the basic voice frequency f0 of the unit section corresponding to the short-term discontinuous region is corrected so as to linearly reach the first section f0 in order from the second section f0 with a constant fluctuation range. To do. Thereafter, the process proceeds to S4390.

  That is, in S4380, the speech fundamental frequency f0 in the non-regular frequency section forming the short-term discontinuous region is changed from 0 [Hz] to a frequency on a straight line connecting the second section f0 and the first section f0.

In subsequent S4390, the section counter is initialized (in this case, the value is 0).
Thereafter, in S4400, it is determined whether or not all unit sections defined in the processed voice data have been selected in S4310. As a result of the determination, if an unselected unit section exists, the process returns to S4310.

  If the count value is less than 1 as a result of the determination in S4370, the process proceeds to S4410. That is, in the frequency transition distribution, if there is no non-regular frequency section between the unit section corresponding to the second section f0 and the unit section corresponding to the first section f0, the process proceeds to S4410.

  In S4410, it is determined whether or not the sound skip flag has been set. Note that the sound skip flag indicates that the start time of the harmonic overtone detection region, which is one of the discontinuous regions in the frequency transition distribution, has been detected if set.

  That is, in the steps after S4410, in the frequency transition distribution, the ratio between the voice basic frequencies f0 in the unit sections adjacent to each other along the time progress of the processed voice data exceeds a special range representing a preset ratio range. Thus, a discontinuous region where the frequency transition is discontinuous (that is, a harmonic overtone detection region) is detected. At the same time, in step S4410 and subsequent steps, the sound fundamental frequency f0 corresponding to the unit section forming the harmonic overtone detection area is corrected.

  If the sound skip flag is not set as a result of the determination in S4410, the process proceeds to S4420. In S4420, it is determined whether or not the result of dividing the second section f0 by the first section f0 (hereinafter referred to as the first frequency ratio) exceeds the special range.

  If the result of determination in S4420 is that the first frequency ratio exceeds the special range, processing proceeds to S4430. In S4430, a sound skip flag is set. That is, when the ratio of the speech fundamental frequencies f0 in the adjacent unit sections along the time progress of the processed speech data exceeds the special range, the unit section after the time progress is selected among the adjacent unit sections. The start time of the harmonic overtone detection area.

In S4440, the second section f0 is set as the third section f0. Thereafter, the process proceeds to S4400.
If the first frequency ratio is within the special range as a result of the determination in S4420, it is determined that the harmonic overtone detection area is not started in the frequency transition distribution, and the process proceeds to S4400. In S4400, if there is an unselected unit section among all the unit sections (S4400: NO), the process returns to S4310.

  By the way, as a result of the determination in S4410, if the sound skip flag has been set, the process proceeds to S4450. In S4450, it is assumed that the harmonic overtone detection area is continuing in the frequency transition distribution, and the sound skip counter is incremented by one.

  Thereafter, in S4460, it is determined whether or not the result of dividing the third section f0 by the first section f0 (hereinafter referred to as the second frequency ratio) exceeds the special range. As a result of the determination, if the second frequency ratio exceeds the special range, the process proceeds to S4470.

  In step S4470, it is determined whether or not the sound skip value, which is the value of the sound skip counter, is equal to or greater than a second predetermined value specified in advance. As a result of the determination, if the skip value is less than the second specified value, it is determined that the harmonic overtone detection area is continuing in the frequency transition distribution, and the process proceeds to S4400.

  By the way, if the result of determination in S4460 is that the second frequency ratio is within the special range, it is determined that continuation of the harmonic overtone detection area has ended in the frequency transition distribution, and the flow proceeds to S4480. In other words, the harmonic overtone detection area starts from the time when the basic frequency f0 in the unit interval adjacent to the frequency transition distribution fluctuates beyond a special range (hereinafter referred to as a specific fluctuation) in the frequency transition distribution. This is a region up to a point in time when the sound fundamental frequency f0 in the unit interval adjacent to the time interval returns to within a special range based on the third interval f0. However, the harmonic overtone detection area is an area where the number of unit sections constituting the area is less than the second specified number.

  In S4480, the basic voice frequency f0 of the unit section corresponding to the harmonic overtone detection area is corrected so as to linearly reach the first section f0 in order from the third section f0 with a constant fluctuation range. . Thereafter, the process proceeds to S4490.

  Note that, as a result of the determination in S4470, if the skip value is equal to or greater than the second specified value, the region composed of adjacent unit sections after the corresponding specific variation is not a discontinuous region, but is a speech fundamental frequency f0 in the input speech. In step S4490, the process proceeds to step S4490. In S4490, the sound skip counter is initialized, the sound skip flag is canceled, and the process proceeds to S4400.

  In S4400, if there is an unselected unit section among all the unit sections (S440: NO), the process returns to S4310. If there is no unselected unit section when the process proceeds to S4400, the f0 correction process is terminated, and the process proceeds to S450 of the pitch estimation process.

Next, an operation example when the f0 correction process in the present embodiment is executed will be described.
Here, FIG. 9A is a diagram showing a frequency transition distribution before executing the f0 correction process, and FIG. 9B is a diagram showing a frequency transition distribution after executing the f0 correction process. It is.

  When the f0 correction processing is performed on the speech fundamental frequency f0 in each unit section showing the frequency transition distribution as shown in FIG. 9A, first, the units along the time progress of the input speech in the frequency transition distribution A section is selected (S4310). The basic voice frequency f0 in the selected unit interval is a frequency other than 0 [Hz] up to the basic audio frequency f0_t1 in the unit interval t1, and the basic audio frequency f0 in the unit interval that continues in time progress. The ratio between them is within a special range. For this reason, the voice fundamental frequency f0 determined in S390 of the pitch transition process is maintained without performing frequency correction from the start time in the frequency transition distribution to the unit interval t1.

However, the ratio between the sound fundamental frequency f0_t1 in the unit interval t1 and the sound fundamental frequency f0_t2 in the unit interval t2 exceeds the special range.
For this reason, when the unit section t2 is selected in S4310, a negative determination is made in S4420, and a sound skip flag is set. Next, the ratio of the voice basic frequency f0_t3 in the unit interval t3 selected in S4310 to the voice basic frequency f0_t1 exceeds the special range. Therefore, an affirmative determination is made in S4460, and the skip value at this point is less than the second specified value (in the description of the operation of the f0 correction process, the second specified value is 2 or more). In S4470, a negative determination is made.

  Then, in S4310, the audio fundamental frequency f0_t4 in the unit interval t4 selected next along the time progress of the processed audio data has a ratio with the audio basic frequency f0_t1 within the special range. Therefore, a negative determination is made in S4460, and a section from the unit section t2 to the unit section t3 is detected as a harmonic overtone detection area. As shown in FIG. 9B, the sound fundamental frequencies f0_t2 and f0_t3 in the harmonic overtone detection area detected in this way are sequentially changed from the sound fundamental frequency f0_t1 to the sound fundamental frequency f0_t4 while changing within a certain fluctuation range. And correct so that it reaches linearly.

  Here, returning to FIG. 9A, in the f0 correction process, the selection of the unit section is repeated along with the time progress of the input voice. At this time, in the frequency transition distribution shown in FIG. 9A, in the region between the unit interval t5 and the unit interval t9, the voice basic frequency f0 (f0_t5 to t9 in the drawing) is 0 [ Hz].

  Therefore, in the f0 correction process, when the unit sections t5 to t10 are selected in S4310, each time the unit sections t5 to t10 are selected, the process proceeds to S4330, and the section counter is set to 5. Increase to. Note that, in S4310, the basic voice frequency f0_t10 in the unit interval t10 that is selected next along the input voice is a frequency other than 0 [Hz], and thus an affirmative determination is made in S4320. Since the count value is less than the first specified value (in the description of the operation of the f0 correction process, the first specified value is 6 or more) and is 1 or more, a negative determination is made in S4370. Therefore, the unit interval t5 to the unit interval t10 are detected as short-term discontinuous regions. The basic voice frequencies f0_t5 to t9 in the short-term discontinuous area detected in this way are sequentially changed from the basic voice frequency f0_tA to the basic voice frequency f0_t10 while changing within a certain fluctuation range, as shown in FIG. 9B. And correct so that it reaches linearly.

That is, in the f0 correction process of the present embodiment, a harmonic overtone detection region, a short-term discontinuous region, and a long-term discontinuous region are detected as discontinuous regions in the frequency transition distribution.
Then, when a harmonic overtone detection area or a short-term discontinuous area is detected as a discontinuous area, the f0 correction process performs speech in a unit section immediately before the harmonic overdetection area or the short-term discontinuous area is sandwiched in time progress. Corrections are made in order from the fundamental frequency f0 so as to reach the speech fundamental frequency f0 in the immediately following unit section while varying within a certain fluctuation range. On the other hand, when a long-term discontinuous area is detected as a discontinuous area, in the f0 correction process, a long-term discontinuous area is formed with the voice fundamental frequency f0 in the unit interval immediately before the time progression for the long-term discontinuous area. The basic voice frequency f0 in the unit section is assumed.
<Start / end timing estimation process>
Next, the start / end timing estimation process started in S150 of the music search process will be described.

Here, FIG. 10 is a flowchart showing a processing procedure of the start / end timing estimation processing.
As shown in FIG. 10, when this start / end timing estimation process is started, first, in S510, for each unit section for which frequency analysis was performed in S310 of the previous pitch estimation process, each unit section The sound pressure at is derived. The derived sound pressure is the sum of the spectrum amplitude values in the amplitude spectrum derived in the previous S310.

  Subsequently, in S520, based on the sound pressure for each unit section derived in S510, a sound pressure transition representing a sound pressure transition along the time progress of the input speech is derived. At the same time, in S520, the derived sound pressure transition is smoothed by the moving average. However, the moving average in the present embodiment is implemented by repeatedly defining a prescribed number of unit sections so as to overlap each other along the time progression in the sound pressure transition. The prescribed number of unit sections that are repeatedly defined is achieved by displacing the unit sections one by one. As a result, the sound pressure transition after smoothing (hereinafter referred to as the smoothed sound pressure transition) has the corresponding sound pressure in all unit sections as the sound pressure transition before smoothing. become.

  In S530, as shown in FIG. 11A, in the smoothed sound pressure transition, the noise sound pressure having a predetermined magnitude is subtracted from each sound pressure corresponding to each unit section. At this time, for a sound pressure at which the subtraction result is a negative value (minus), the value is set to zero.

  In S540, one unit section is selected from all the unit sections in the sound pressure transition. At the same time, in S540, the sound pressure in the selected unit section is acquired. In S540, each time the unit section moves to S540, the unit section is sequentially selected from the start of the processed audio data along the time progress of the processed audio data.

  In S550, the sound pressure that has been the first sound pressure Pv1 until the current transition to S550 is set as the second sound pressure Pv2, and the sound pressure in the unit section selected in S540 when the transition to S550 is performed is the first. Set as one sound pressure Pv1. That is, when the process proceeds to S550, the sound pressure in the unit section close to the start of the sound is set as the second sound pressure Pv2 among the sound pressures acquired along the time progress of the processed sound data, and close to the end of the sound. The sound pressure in the unit section is the first sound pressure Pv1.

Further, in S560, the first sound pressure Pv1 is divided by the second sound pressure Pv2 (hereinafter, this calculation result is referred to as a sound pressure increase rate).
Subsequently, in S570, it is determined whether or not the sound pressure increase rate derived in S560 is greater than or equal to a predefined threshold value Th. As a result of the determination in S570, if the sound pressure increase rate is equal to or greater than the specified threshold Th, the process proceeds to S580. In S580, the sound generation counter is incremented by one.

  Subsequently, in S590, it is determined whether or not the pronunciation count value, which is the value of the pronunciation counter, is greater than or equal to a first threshold value defined in advance. If the result of the determination is that the pronunciation count value is less than the first threshold value, S600 is performed. Proceed to In S600, it is determined whether or not the sound generation count value is equal to or greater than a second threshold value that is defined in advance as one value smaller than the first threshold value. As a result of the determination in S600, if the pronunciation count value is less than the second threshold value, the process returns to S540, and steps S540 to S590 are repeated.

  On the other hand, as a result of the determination in S600, if the pronunciation count value is equal to or greater than the second threshold value, that is, when the steps of S540 to S590 are repeated, an affirmative determination is made in S570 continuously for the second threshold value. , The process proceeds to S610. That is, by making an affirmative determination in S600, a unit interval in which the sound pressure increase rate is equal to or greater than the specified threshold Th is a region in which the number of the first threshold value plus 1 (hereinafter referred to as a start determination target interval). Is detected).

  In step S610, the first unit section along the time progress of the input voice among the unit sections forming the start determination target section is specified as the sound generation start timing. At the same time, the sound pressure at the specified sounding start timing (hereinafter referred to as sounding start sound pressure) is acquired. Furthermore, in S610, the specified sound generation start timing and the acquired sound generation start sound pressure are stored in the storage unit 28. 10 to 12, the start timing is expressed as “ST”.

  Note that, as a result of the determination in S590, if the pronunciation count value is greater than or equal to the first threshold value, the process proceeds to S630 without executing steps S600 and S610. That is, in the smoothed sound pressure transition, if the sound pressure increase rate after the sounding start timing is continuously greater than or equal to the specified threshold Th from the sounding start timing, a negative determination is made in S590.

  By the way, if the sound pressure increase rate is less than the prescribed threshold Th as a result of the determination in S570, the sound generation counter is initialized (in this case, 0) in S620. That is, the measurement of the number of unit sections in which the sound pressure increase rate equal to or greater than the specified threshold Th continues. Thereafter, the process proceeds to S630.

  In S630, it is determined whether or not the first sound pressure Pv1 is equal to or lower than the sound generation start sound pressure (hereinafter referred to as end determination sound pressure) stored in the storage unit 28 in the latest S610. As a result of the determination, if the first sound pressure Pv1 is less than the end determination sound pressure, the process proceeds to S640.

  In S640, the unit section corresponding to the first sound pressure Pv1 is stored in the storage unit 28 as the sound generation end timing. Thereafter, the process proceeds to S650. 10 to 12, the end timing is expressed as “ET”.

  As a result of the determination in S630, if the first sound pressure Pv1 is equal to or higher than the end determination sound pressure, it is determined that the unit section selected in S540 is not the sound generation end timing, and S640 is not executed. , The process proceeds to S650.

  In S650, it is determined whether or not all unit sections defined in the processed audio data have been selected in S540. As a result of the determination, if there is an unselected unit section, the process returns to S540. On the other hand, as a result of the determination in S650, if there is no unselected unit section in S540, the process proceeds to S660.

  That is, as shown in FIG. 11B, when the smoothed sound pressure transition includes a monotonically increasing section of the sound pressure in which the sound pressure increase rate is continuously equal to or higher than the specified threshold, the start / end timing estimation process In step S540 to S650, the first unit section in the monotonically increasing section is specified as the sound generation start timing (first, second, third, and fourth sound generation STs in the figure). However, the monotonically increasing section referred to here is a unit section that continues for the number of unit sections forming the start determination target section.

  Furthermore, by repeating the steps S540 to S650, in the smoothed sound pressure transition, the sound pressure corresponding to each unit section in the unit sections after the sounding start timing along the time progress of the processed speech data is The unit section that first becomes equal to or lower than the end determination sound pressure is specified as the sound generation end timing (first and second sound generation ETs in the figure).

  Here, returning to FIG. 10, in S <b> 660, the storage unit 28 uses the final unit interval along the time progression of the processed audio data as the sound generation end timing among the unit intervals set in the processed audio data. To remember.

  In S670, the basic voice frequency f0 in the determination target section is acquired from all the unit sections defined in the processed voice data. The determination target section from which the voice fundamental frequency f0 is acquired in S670 is composed of a predetermined number of unit sections. The prescribed number of unit sections are repeatedly defined so as to be continuous and overlap each other along the time progress in the processed voice data.

  In S680, the fluctuation range of the voice fundamental frequency f0 is derived based on the voice fundamental frequency f0 in the determination target section acquired in S670. The fluctuation range derived in S680 is the difference between the maximum audio basic frequency f0 and the minimum audio basic frequency f0 in the determination target section.

  In S690, it is determined whether or not the fluctuation width derived in S680 is less than a predetermined width that is a predetermined frequency width. As a result of the determination, if the fluctuation range is less than the specified range, the process proceeds to S700.

  In S700, a frequency trajectory is derived by arranging all the voice basic frequencies f0 in the determination target section along the time progress of the processed voice data. At the same time, the derived frequency locus is smoothed and differentiated to detect an extreme value in the frequency locus.

  In S710, it is determined whether or not an extreme value in the frequency trajectory has been detected as a result of the smoothing differentiation in S700. If the extreme value is detected as a result of the determination, the process proceeds to S720.

  In S720, the number of extreme values in the determination target section detected in S700 is totaled. In S730, it is determined whether or not the vibrato value, which is the number of extreme values tabulated in S720, is equal to or greater than a predetermined third threshold value. If the result of determination in S730 is that the vibrato value is greater than or equal to the third threshold value, processing proceeds to S740.

  That is, by executing the steps from S670 to S730, the sum of the increasing interval in which the speech fundamental frequency f0 increases and the decreasing interval in which the speech fundamental frequency f0 increases in the determination target interval in which the fluctuation range of the speech fundamental frequency f0 is less than the specified width. Is determined as the vibrato period. In addition, this vibrato period means the period when the user of the voice processing device 20 uttered by vibrato.

  In step S740, the vibrato values counted in step S720 are initialized (in this case, 0). Further, in S750, the sounding start timing corresponding to the vibrato period (hereinafter referred to as the intra-period timing) is deleted (removed) from the sounding start timings stored in the storage unit 28. Thereafter, the process proceeds to S770.

  As a result of the determination in S690, when the fluctuation range of the voice basic frequency f0 within the determination target section is equal to or greater than the specified width, or as a result of the determination in S710, no extreme value is included in the determination target section. Then, the process proceeds to S760. Furthermore, if the result of determination in S730 is that the vibrato value is less than the third threshold, the process proceeds to S760.

That is, if the determination target section defined in S670 is not the vibrato period, the process proceeds to S760. In S760, after the vibrato value is initialized, the process proceeds to S770.
In S770, it is determined whether or not all the unit sections set in the processed audio data are defined as the determination target sections. As a result of the determination, if all unit sections are not defined as determination target sections, the process returns to S680, a new determination target section is set, and the process proceeds to S680. Then, S680 to S770 are repeated until all unit sections are defined as determination target sections.

  For example, by executing the start / end timing estimation process, the sounding start timing (first to fourth sounding start timings) and the sounding end timing (first and second sounding end times) as shown in FIG. (Timing) is specified, and the determination target section including the third sound generation start timing and the fourth sound generation start timing is specified as the vibrato period. In such a case, since the third sound generation start timing and the fourth sound generation start timing are removed as the in-period timing, as shown in FIG. 12B, the first sound generation start timing and the second sound generation start timing are Only two are left. Note that all the sound generation end timings are left without being removed.

If it is determined in S770 that all unit sections are defined as determination targets, the start / end timing estimation process is terminated, and the process proceeds to S190 of the music search process.
That is, in the start / end timing estimation processing of the present embodiment, the sound generation start timing and the sound generation end timing are detected based on the sound pressure transition of the input sound, and the time base of the sound basic frequency f0 in the input sound is met. The vibrato period is specified from the transition (that is, the frequency trajectory). In the start / end timing estimation process, the sounding start timing corresponding to the specified vibrato period is deleted, and only the sounding start timing corresponding to the outside of the vibrato period is left.
<About transcription processing>
Next, the music transcription process started in S190 of the music search process will be described.

Here, FIG. 13 is a flowchart showing the processing procedure of the music transcription processing.
As shown in FIG. 13, when the music recording process is started, first, in S910, one unit section is selected from all the unit sections in which the frequency analysis was performed in S310 of the previous pitch estimation process. Select. In S910, each time a transition is made to S910, the unit sections are sequentially selected from the start of the processed audio data along the time progress of the processed audio data.

  Subsequently, in S920, it is determined whether or not the unit section selected in S910 is the sounding start timing. As a result of the determination, if the selected unit section is not the sounding start timing, the process proceeds to S930.

  In S930, it is determined whether or not the unit section selected in S910 is the sound generation end timing. As a result of the determination, if the selected unit section is not the sound generation end timing, the process returns to S910. That is, if the unit section selected in S910 is not the sounding start timing or the sounding end timing, the steps from S910 to S930 are repeated.

  On the other hand, as a result of the determination in S920, if the unit section selected in S910 is the sounding start timing, the process proceeds to S940. In S940, the sound generation start timing that was the first start timing before the transition to S940 this time is set as the second start timing, and the unit section selected in S910 when moving to S550 (ie, the sound generation start timing) Is set as the first start timing. That is, when proceeding to S940, the sounding start timing close to the utterance start is set as the second sounding start timing along the time progress of the processed sound data, and the sound pressure in the unit section close to the sound end is set to the first. It is the start timing. In FIG. 13, the start timing is denoted as ST.

  In S950, it is determined whether a start acquisition flag (hereinafter referred to as start acquisition F) has been set. If the start acquisition flag is not set as a result of the determination, the process proceeds to S960. In S960, a start acquisition flag is set. Thereafter, the process returns to S910.

  By the way, as a result of the determination in S930, if the unit section selected in S910 is the pronunciation end timing, the process proceeds to S970. That is, the transition to S970 is a case where the sound generation start timing and the sound generation end timing to be paired with the sound generation start timing are acquired along the time progress of the processed audio data. In S970, the start acquisition flag is canceled and the process proceeds to S980.

  If the start acquisition flag is set as a result of the determination in S950, the process proceeds to S980. That is, when two sound generation start timings exist without interposing the sound generation end timing between the sound generation start timings along the time progress of the processed sound data, an affirmative determination is made in S950.

In S980, the note period is specified based on the sounding start timing or sounding end timing acquired at the time of shifting to S980.
Specifically, in this embodiment, when the determination in S930 is affirmative and the process proceeds to S980, the first start timing is set as the note start timing, and the sound generation end timing is set as the note end timing. On the other hand, if the determination in S950 is negative and the process proceeds to S980, the second start timing is set as the note start timing, and the first start is performed in accordance with the time progress of the processed audio data from the first start timing. The time point before the timing by the set time length is set as the note end timing. In any case, the period between the note start timing and the note end timing is specified as the note period. In S980 in the present embodiment, the length of the specified note period is derived as the sound length.

  Subsequently, in S990, the pitch in all unit intervals corresponding to the note period specified in S980 (that is, the voice fundamental frequency f0 quantized in S450 of the pitch estimation process, hereinafter also referred to as a quantization frequency). ) To get. That is, the quantization frequency is acquired by the number of unit intervals constituting the note period.

  In S1000, the first pitch frequency and the second pitch frequency are specified based on the quantization frequency acquired in S990, and the first pitch number and the second pitch number are tabulated. The first pitch frequency specified in S1000 is a quantization frequency having the highest ratio in the note period specified in S980. The second pitch frequency is a ratio in which the ratio in the note period is two. The second highest quantization frequency. In S1000 of the present embodiment, when there are a plurality of second pitch frequencies in the note period specified in S980, the highest frequency is set as the second pitch frequency.

  The first pitch number counted in S1000 is the number of unit intervals corresponding to the first pitch frequency in the unit intervals included in the note period specified in S980. The second pitch number is the number of unit intervals corresponding to the second pitch frequency in the unit intervals included in the note period specified in S980.

  Next, in S1010, it is determined whether or not the second pitch frequency specified in S1000 is higher than the first pitch frequency. As a result of the determination, if the second pitch frequency is higher than the first pitch frequency, the process proceeds to S1020.

  In S1020, it is determined whether or not the second pitch number is equal to or higher than a pitch determination threshold value. The pitch determination threshold used for this determination is a value obtained by multiplying a predetermined ratio A (in the present embodiment, 1 / 2.3) and the first pitch number. As a result of the determination in S1020, if the second pitch number is equal to or higher than the pitch determination threshold, the process proceeds to S1030.

  In S1030, the pitch corresponding to the second pitch frequency is specified as the pitch in the note period specified in S980 (that is, the note pitch). Then, voice note data is generated with the specified note pitch and the note length derived in S980 as the note note length. Thereafter, the process proceeds to S1050.

  By the way, as a result of the determination in S1010, if the second pitch frequency is equal to or lower than the first pitch frequency, or if the second pitch number is less than the pitch determination threshold as a result of the determination in S1020, the process proceeds to S1040. Proceed with

  In S1040, the pitch corresponding to the first pitch frequency is specified as the pitch (that is, the note pitch) in the note period specified in S980. Then, voice note data is generated with the specified note pitch and the note length derived in S980 as the note note length. Thereafter, the process proceeds to S1050.

For example, after repeating the steps from S910 to S980, in S980, the period between the first note start timing and the first note end timing as shown in FIG. 14A is specified as the first note period. To do. Second pitch frequency f02 _t1 _hi in the identified first note period is the frequency higher than the first pitch frequency f01 _t1. The second pitch number is “3”, and is a value obtained by multiplying the first pitch number “5” by a specified ratio A (A = 1 / (2.3) in the present embodiment). Is also big.

In this case, as shown in FIG. 14B, the note pitch for the first note period is the pitch corresponding to the second pitch frequency f02 _t1 _hi in S1030 (the first note pitch in the figure). ).

Note that the second pitch frequency f02 _t1 _low also, the ratio in the first note is the same as the second pitch frequency f02 _t1 _hi. However, since towards the second pitch frequency f02 _t1 _hi has a high frequency, the pitch of the first note period, the second pitch frequency f02 _t1 _hi.

Furthermore, after repeating the steps from S910 to S980, in S980, the period between the second note start timing and the second note end timing as shown in FIG. 14A is specified as the second note period. To do. Second pitch frequency f02 _t2 _hi in the specified second note period is the frequency lower than the first pitch frequency f01 _t2. Further, the second pitch number is “3”, and a value obtained by multiplying the first pitch number “4” by a specified ratio A (A = 1 / (2.3) in the present embodiment). Is also small.

In this case, as shown in FIG. 14B, the note pitch (the second note pitch in the figure) for the second note period is the pitch corresponding to the first pitch frequency f01 _t2 in S1040. Specified.
In S1050, it is determined whether or not all unit sections defined in the processed audio data have been selected in S910. As a result of the determination, if there is an unselected unit section, the process returns to S910, and S910 to S1050 are repeated.

On the other hand, if there is no unselected unit section as a result of the determination in S1050, the music transcription process is terminated and the process proceeds to S210 of the music search process.
In other words, in this music recording process, if there is a sounding start timing and a sounding end timing that should be paired with the sounding start timing as time progresses in the processed audio data, the sounding start timing is set to the note start time. Timing is set, and the sound generation end timing is set as a note end timing. In addition, when the two sound generation start timings exist without interposing the sound generation end timing between the sound generation start timings along the time progress of the processed audio data, the previous sound generation start along the time progress is started. The timing is the note start timing, and the sound generation start timing after the time progression is the note end timing. In any case, the period between the note start timing and the note end timing is specified as the note period.

In addition to this, in this musical notation process, if the second pitch frequency is higher than the first pitch frequency and the second pitch number is equal to or higher than the specified ratio A with respect to the first pitch number, The pitch corresponding to the second pitch frequency is specified as the note pitch in the note period. At the same time, in this music recording process, when the second pitch frequency is lower than the first pitch frequency, or when the second pitch number is less than the specified ratio A with respect to the first pitch frequency. The pitch corresponding to the first pitch frequency is specified as the note pitch in the note period.
<Transcription result matching process>
Next, the transcription result matching process started in S210 of the music search process will be described.

Here, FIG. 15 is a flowchart showing the processing procedure of the transcription result collation process.
As shown in FIG. 15, when the transcription result collation process is started, in S1210, the voice note data generated in the previous transcription process is defined in advance along the time progress of the processed voice data. A word is formed (ie, grouped) for each specified number of notes. This wording is performed so that part of the voice note data overlaps each other. Hereinafter, each of the voiced note data converted into words is referred to as word note data.

  Furthermore, in S1220, the music (hereinafter referred to as note collation) in which the word note data is collated with the reference note data (that is, the guide melody) from the music corresponding to the music data acquired from the server 40 and stored in the storage unit 28. (Referred to as music).

  In S1230, one word note data is acquired from all the word note data generated in S1210. However, when the word note data is acquired, data including voice note data close to the start of the voice is acquired in the time progress of the processed voice data.

  In step S1240, the reference note data corresponding to the note collation music determined in step S1220 is converted into words for a predetermined number of notes that are continuous with time. When the reference note data corresponding to the specified number of notes is converted into words, the reference note data for the constituent sounds close to the start of the reference melody is executed in the time progression of the reference melody. Hereinafter, reference note data corresponding to the prescribed number of notes acquired by wording in S1240 is referred to as comparative note data.

  Subsequently, in S1250, the word note data acquired in S1230 is collated with the comparison note data acquired in S1240. As a result of the collation, if the word note data and the comparison note data match (S1260: YES), the process proceeds to S1270.

  In S1270, a note coincidence degree and a cumulative in-music coincidence degree, which will be described in detail later, are derived, and the derived accumulated in-music coincidence degree is stored in association with the constituent sound numbers, and then the process proceeds to S1280. The number of the component sound associated with the cumulative in-music coincidence is the one associated with the first component sound along the time progression of the reference melody among the prescribed number of component sounds forming the comparative note data It is.

On the other hand, if the word note data and the comparison note data do not match as a result of the collation in S1250 (S1260: NO), the process proceeds to S1280.
In S1280, all the reference note data are worded, and it is determined whether or not the word note data acquired in S1230 is collated with the comparison note data generated by the wording. As a result of the determination, if the word note data is not collated with all the comparison note data, the process returns to S1240. In step S1240, the reference note data worded in the previous step S1240 and the reference note data corresponding to the prescribed number of notes so that a part of the reference note along the time progression of the reference melody overlaps are worded. get. That is, new comparison note data is generated, and the process proceeds to S1250.

Thereby, S1240 to S1280 are repeatedly executed until the collation of one word note data is completed for all the reference note data in one music piece.
As a result of the determination in S1280, if all reference note data is worded and the word note data is collated with the comparison note data generated by the wording, the process proceeds to S1290. In S1290, all the word note data is acquired, and it is determined whether or not the comparison note data has already been collated.

  As a result of the determination in S1290, if all the word note data is not collated with the comparison note data, the process returns to S1230. In S1230, one word note data is acquired from word note data that has not been compared with the comparison note data. However, when acquiring the word note data, the word note data consisting of the voice note data close to the start of the voice is acquired as the input voice progresses over time.

  Thereafter, the steps from S1230 to S1290 are repeated until an affirmative determination is made in S1290. Hereinafter, one flow from S1300 to S1290 is referred to as a separate note collation cycle. In addition, one flow from S1240 to S1280 from the acquisition of word note data to the acquisition of new word note data in another note verification cycle is referred to as the same note verification cycle.

  If an affirmative determination is made in S1260 in the process of repeating the same note matching cycle, the process proceeds to S1270. In S1270 thus shifted, the comparison note data matched with the word note data in the current separate note collation cycle is compared with the comparison note data matched with the word note data in the previous separate note collation cycle and the reference melody. It is determined whether it is continuous over time (hereinafter referred to as note connection determination). Specifically, the comparison note that is determined to match the word note data when proceeding to S1270 this time among the constituent note numbers associated with the note matching degree in the previous separate note matching cycle. If there is a number indicating that it is a constituent sound one before the time progression in the reference melody than the number of the constituent sound forming the data, the determination result in the note connection determination is affirmed.

  If the determination result of the note connection determination is affirmative, a value obtained by raising the power of the initial specified value is derived as the note coincidence, with the number of different note collation cycles successively determined to be positive as the “power exponent”. On the other hand, if the determination result of the note connection determination is negative, the initial specified value itself is derived as the note matching degree.

  That is, the note coincidence degree becomes larger as the word note data along the time progression of the processed voice data continuously matches the comparison note data along the time progression in the reference melody of the note collation music.

Further, the sum of the derived note coincidence is derived as the in-music cumulative coincidence.
If a positive determination is made in S1290, the process proceeds to S1300. In S1300, the cumulative value in the music for the note collation music determined in S1220 is correlated with the song name data corresponding to the note collation music in the storage unit 28. Remember. In other words, the cumulative coincidence in music associated with the song name data in S1300 has the largest value among the cumulative in-music coincidence derived by repeating another note collation cycle for one note collation music. It is.

  Subsequently, in S1310, it is determined whether or not all the music corresponding to the music data stored in the storage unit 28 has been determined as the note collation music. As a result of the determination, if all music pieces have not been determined as note collation music pieces, the process returns to S1220. In S1220 thus shifted, a new music piece is determined as a note collation music piece from among the music pieces that have not been decided as a note collation music piece, and the process proceeds to S1230. That is, the steps from S1230 to S1310 are repeated until the reference of note data in all music data stored in the storage unit 28 is matched with the word note data.

If it is determined in step S1310 that all the music pieces stored in the storage unit 28 have been determined as note collation music pieces, the process advances to step S1320.
In S1320, the music corresponding to the in-music cumulative coincidence having the maximum value among the cumulative in-music coincidence stored in the storage unit 28 in S1300 is specified as the expected expected music. Further, in S1320, the song name data for the specified expected song is acquired, the song name corresponding to the obtained song name data is displayed on the display unit 22, and the song name is output from the speaker 27 by voice. That is, the song title of the expected song is notified.

Then, after that, the musical score matching process is terminated, and the music search process is terminated.
That is, in the transcription result collating process of the present embodiment, the voice note data generated by the musical transcription process is collated with reference note data prepared in advance for each music piece. And, as a result of the collation, the more the number of comparison note data that the voice note data continuous along the time progression of the processed voice data matches continuously along the time progression in the reference melody of the note collation music, Deriving the cumulative value of the cumulative value in the song. And in the transcription result collation process of this embodiment, the music corresponding to the highest value in the derived in-music cumulative coincidence is detected as the intended expected music.
[Effect of the embodiment]
As described above, in the start / end timing estimation process of the present embodiment, the sounding start timing corresponding to the vibrato period is deleted from the detected sounding start timings, and the sounding start timing corresponding to outside the vibrato period is deleted. Leave only.

  Then, if there is a remaining sounding start timing and a sounding end timing to be paired with the sounding start timing, the sounding start timing and the sounding end timing are set as the note start timing and the note end timing, respectively, in the music transcription process. It is timing. On the other hand, if the remaining sounding start timing is present without interposing the sounding end timing between the two sounding start timings, the two sounding start timings of the two sounding start timings before the time progression are obtained in the music recording process. The sound generation start timing is the note start timing, and the sound generation start timing after the time progress is the note end timing. Further, in the music recording process, a period between the note start timing and the note end timing is estimated as a note period.

  Therefore, according to the speech processing apparatus 20 of the present embodiment, it is possible to prevent the note period from which the period starts from being estimated from within the vibrato period. As a result, according to the speech processing device 20 of the present embodiment, it is possible to improve the estimation accuracy of the note period in the input speech.

According to the sound processing device 20 of the present embodiment, since the end of the input sound is set as the sound generation end timing, the note period is also applied to the last sound generation start timing (ie, note start timing) in the processed sound data. Can be estimated.
[Other Embodiments]
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects.

  For example, the pitch estimation process executed in the music search process is not limited to that described in the embodiment. The pitch estimation processing in the above embodiment may be any method as long as it detects the voice fundamental frequency f0 of the input voice in the unit section for each unit section defined in the processed voice data.

  Moreover, in the start / end timing estimation process of the above embodiment, the detection of the sound generation end timing is performed after the pair of sound generation start timings, but the detection of the sound generation end timing is not limited to this, For example, it may be performed after the end of the vibrato period.

  Furthermore, in the above embodiment, the detection of the end time of the sound generation is after the sound generation start timing along the time progress of the processed sound data, and the unit section after the predetermined number of unit sections from the sound generation start timing You may perform after that.

  In S530 in the start / end timing estimation process of the above embodiment, the predetermined specified value is the noise sound pressure, but the noise sound pressure is not limited to this. For example, the average sound pressure from the start point along the time progression of the processed voice data to the first sounding start timing along the time progression may be the noise sound pressure, and among the specified value and the average sound pressure, A large value may be used as the noise sound pressure.

  By the way, in the music search process in the above-described embodiment, the audio data that is input via the microphone 24 and then stored in the storage unit 28 is the processing target. For example, audio data acquired from the server 40 or another audio processing device 20 may be used. In this case, in the audio processing device 20, the microphone 24 and the audio input unit 25 may be omitted.

  On the contrary, in the music search process in the above embodiment, the audio data immediately after being sampled by the audio input unit 25 may be directly processed. That is, in the music search process, the voice input via the microphone 24 may be processed in real time.

In addition, the audio processing device 20 in the above embodiment may not include the speaker 27 and the audio output unit 26.
In the above embodiment, the music search process is executed by the audio processing device 20, but the music search process may be executed by the server 40. In this case, the audio data needs to be transferred from the audio processing device 20 to the server 40.

In addition, the music search system 1 may be configured only from the voice processing device 20. In this case, the music data needs to be stored in the storage unit 28 in advance.
In the music search process in the above-described embodiment, the transcription result matching process is executed in S210. However, S210 may be omitted as the content executed as the music search process. That is, the audio processing device 20 in the above embodiment may be configured as a so-called music recording device.
[Correspondence between Embodiment and Claims]
Finally, the correspondence between the description of the above embodiment and the description of the claims will be described.

  The function obtained by executing S510 and S520 in the start / end timing estimation process of the above embodiment corresponds to the sound pressure transition specifying means of the present invention, and the function obtained by executing S540 to S610 is This corresponds to the start timing detection means of the invention.

  The function obtained by executing S670 to S730 in the start / end timing estimation process of the above embodiment corresponds to the vibrato period specifying means of the present invention, and the function obtained by executing S750 is the present invention. This corresponds to the timing removal means within the period.

  In addition, the function obtained by performing S910-S980 in the music recording process of the said embodiment is equivalent to the note period estimation means of this invention.

  DESCRIPTION OF SYMBOLS 1 ... Music search system 20 ... Audio | voice processing apparatus 21 ... Communication part 22 ... Display part 23 ... Operation reception part 24 ... Microphone 25 ... Audio | voice input part 26 ... Audio | voice output part 27 ... Speaker 28 ... Memory | storage part 30 ... Control part 31 ... ROM 32 ... RAM 33 ... CPU 40 ... Server 41 ... Storage device 42 ... Microcomputer

Claims (7)

From the time the input speech continuously along with the progress, to estimate the note duration representing each period that can be regarded as a single note, a speech processing device for transcription identify and the pitch of the musical note period,
A sound pressure transition specifying means for specifying a sound pressure transition representing a transition along a time progression of a sound pressure in the input sound from the input sound;
Each note period start timing is a note start timing, and in a section where the sound pressure transition specified by the sound pressure transition specifying means is monotonically increasing, the sound in the first specified period specified by the sound pressure transition Start timing detection means for detecting each time when the rate of increase in pressure first becomes equal to or greater than a predetermined value as time progresses, as the note start timing;
A vibrato period specifying means for specifying the vibrato period based on a pitch transition representing a transition along a time progression of a pitch in the input voice, with a period uttered by vibrato in the input voice as a vibrato period; ,
Among the note start timings detected by the start timing detecting means, the note start timing corresponding to the vibrato period specified by the vibrato period specifying means is set as the in-period timing, and the detection by the start timing detecting means is performed. An intra-period timing removing means for removing the intra-period timing from the result;
A note end timing paired with each note start timing after the timing within the period is removed by the timing removal means within the period, and between the note start timing and the note end timing as a pair Note period estimating means for estimating each period as the note period, and
The vibrato period specifying means is:
A plurality of periods defined so as to be continuous with each other over the entire pitch transition and along the time progress are defined as a second defined period, and the pitch transition in the second defined period is defined as a period pitch transition. If the pitch fluctuation range in the period pitch transition is equal to or less than a predetermined range, the increase section in which the pitch increases and the decrease section in which the pitch decreases in the period transition are detected. An increase / decrease detection means to
If the number of increase and decrease sections detected by the increase / decrease detection means is equal to or greater than a predetermined number, a period specification that specifies the second specified period corresponding to the period pitch transition as the vibrato period And a voice processing apparatus. The note period estimating means includes
The sound pressure fluctuation time at which the sound pressure in the sound pressure transition first becomes equal to or lower than the sound pressure at the note start timing after the note start timing after the timing within the period is removed is defined as the note start timing. The voice processing device according to claim 1, wherein the voice processing device is specified as the note end timing of a pair. The note period estimating means includes
Of the note start timings that are adjacent to each other along the time progress after the timing within the period is removed, on the time progress, the previous note start timing is set as the previous start timing, and the subsequent note start timing is set as the subsequent start timing, 3. The voice processing according to claim 1, wherein a time point that is a preset time length before the subsequent start timing is specified as the note end timing paired with the previous start timing. 4. apparatus. The note period estimating means includes
If there is a sound pressure fluctuation time point at which the sound pressure in the sound pressure transition becomes equal to or lower than the sound pressure at the previous start timing before the subsequent start timing, the sound pressure fluctuation time point is The voice processing device according to claim 3, wherein the voice processing device is specified as the note end timing paired with the start timing. The note period estimating means includes
The end of the input voice along the time progress is specified as the note end timing paired with the last note start timing along the time progress among the note start timings after the timing within the period is removed. The speech processing apparatus according to any one of claims 1 to 4, wherein the speech processing apparatus is characterized. Fundamental frequency determination means for determining a speech fundamental frequency in the unit interval for each unit interval continuous along the time axis in the input speech;
Discontinuous detection for detecting a discontinuous region in which a speech fundamental frequency is discontinuous between successive unit sections among frequency transitions in which the speech fundamental frequency detected by the fundamental frequency determining means is arranged along a time axis. Means,
The basic frequency of each unit section corresponding to the discontinuous area detected by the discontinuity detecting means is the frequency fundamental specified by the sound pressure transition specifying means. A frequency correction means for performing frequency correction for correcting so as to reach the immediately following frequency that is the speech fundamental frequency of the unit section immediately after the discontinuous region from the immediately preceding frequency that is the frequency;
The pitch estimation means for specifying a voice fundamental frequency in a unit section after the frequency correction by the frequency correction means is performed as a pitch in the note period corresponding to the unit section. The speech processing apparatus according to any one of claims 1 to 5. It is a program for causing a computer to function as a voice processing device that specifies and records a note period that represents each period that can be regarded as one note from input speech that is continuous over time, and a pitch in the note period. And
A sound pressure transition identification procedure for identifying a sound pressure transition representing a transition along a time progression of a sound pressure in the input voice from the input voice;
The sound in the first specified period defined in the sound pressure transition in the section where the sound pressure transition specified in the sound pressure transition specifying procedure is monotonically increasing with the start timing of each of the note periods as the note start timing. A start timing detection procedure for detecting, as the note start timing, each time point at which the rate of increase in pressure becomes equal to or greater than a predetermined value defined in advance along time,
A vibrato period specifying procedure for specifying a vibrato period based on a pitch transition representing a transition along a time progression of a pitch in the input voice, with a period uttered by vibrato in the input voice as a vibrato period; ,
Among the note start timings detected in the start timing detection procedure, the note start timing corresponding to the vibrato period specified in the vibrato period specifying procedure is set as the in-period timing, and the detection in the start timing detection procedure is performed. From the results, a timing removal procedure within a period for removing the timing within the period;
The note end timing paired with each of the note start timings after the timing within the period is removed in the timing removal procedure within the period, and between the note start timing and the note end timing as a pair Causing the computer to execute a note period estimation procedure for estimating each period as the note period;
further,
The vibrato period specifying procedure is:
A plurality of periods defined so as to be continuous with each other over the entire pitch transition and along the time progress are defined as a second defined period, and the pitch transition in the second defined period is defined as a period pitch transition. If the pitch fluctuation range in the period pitch transition is equal to or less than a predetermined range, the increase section in which the pitch increases and the decrease section in which the pitch decreases in the period transition are detected. An increase / decrease detection procedure to
If the number of increasing and decreasing intervals detected by the increase / decrease detection procedure is equal to or greater than a predetermined number, a period specifying that specifies the second specified period corresponding to the period pitch transition as the vibrato period A program characterized by causing a computer to execute the procedure.