JP2004334240A - Sound signal analysis device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To determine validity/invalidity of a section, estimated to correspond to a normal part of a musical sound, i.e. a single note except for a fluctuated part, even when the pitch or a level of an input sound from a microphone or the like fluctuates delicately. <P>SOLUTION: A sound signal analysis device inputs an arbitrary sound signal consisting of one or a plurality of note time series connections, and detects the section estimated to correspond to the note one by one from among the input signals, respectively. It arranges the detected section on a grid divided at time intervals, corresponding to the prescribed note length according to the time series, respectively, assigns one which is nearest the grid position on one prescribed end from either the start or the finish end of each normal section for each normal section respectively, and as a result, selects one normal section of the longest time length as a valid note, when a plurality of the normal sections are assigned at the same grid position. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

The present invention relates to a section in which a musical sound exists (effective section) and a steady state of the musical sound based on a sound signal whose pitch or note is undetermined, such as an audio signal or a musical sound signal input by a microphone or the like. The present invention relates to a sound signal analyzing apparatus and method capable of automatically analyzing a part (note name of a scale) and a note length thereof, and further relates to a recording medium storing a program therefor.
The analysis result according to the present invention can be output as electronic musical score information in the form of MIDI information or the like as necessary. Therefore, the present invention automatically converts an audible melody input by human voice or the like. The present invention relates to a technology that can be converted into a score.

Recently, a computer performance system that generates performance information such as MIDI information using a computer or the like and reproduces performance sounds based on the performance information has attracted attention as a new musical performance apparatus.
In this type of computer performance system, a method for inputting data for generating performance information includes a real-time input method, a step input method, a numerical value input method, a score input method, and the like.

  The real-time input method is a method of converting operation information of a performance operator such as a keyboard actually played by a player like a tape recorder into performance information in real time. The numerical value input method is a method in which performance information such as a pitch (pitch), a sound length, and the strength of a sound is directly input as numerical data from a computer keyboard. The musical score input method is a method in which simplified musical notation symbols and the like are arranged in a musical score (5-line notation) on a display using a function key of a computer, a mouse, or the like. The step input method is a method in which a musical score is input on a MIDI keyboard or a software keyboard, and a sound length is input using a function key of a computer, a mouse, or the like.

Of the above-mentioned input methods, the real-time input method can store the actual performance operation state as it is as performance information, so it is easy to express human delicate nuances in performance, and it is possible to input in a short time It has the advantage of being However, this method requires the player to have a high level of musical instrument playing ability, and is not suitable for beginners.
Therefore, as a performance information generator that allows beginners to easily input performance information in a short time by taking advantage of the real-time input method, human voices or musical sounds of natural musical instruments are directly input through a microphone, and the input sound is input. In some cases, performance information is generated in accordance with the performance information. That is, this can easily generate a MIDI signal simply by inputting a human voice or a sound (single sound) such as a guitar from a microphone, and can control a MIDI device without using a MIDI keyboard or the like.

  In a conventional performance information generating apparatus, MIDI information is generated by performing the following processing for a pitch change of a sound input from a microphone. That is, the first method detects a change in pitch in semitone units and generates only note information of that pitch. In the second method, a pitch change is detected in semitone units, and note information of the pitch and pitch bend information (pitch change information) relating to a pitch change therebetween are generated. The third method generates pitch bend information that can change the pitch of an input signal in the range of one octave up and down without detecting a note. To generate note information (note-on or note-off), the level of the input sound is compared with a predetermined reference value, and when the level of the input sound becomes higher than the reference value, the note-on is reduced. A note-off has occurred at the point in time.

However, in the case where the pitch change is detected in semitone units as in the first and second methods, if the pitch of the input sound is slightly changed, a lot of unintended note information (note on or note off) is generated. There is. When the pitch change is generated by the pitch bend information as in the third method, the pitch change can be made to follow the pitch bend information faithfully, but is not suitable for purposes such as transcription. Further, when note information is generated according to the input level, there is a problem that a large number of unintended note information is generated according to the fluctuation of the level of the input sound.

  By the way, in the real-time input method, since a plurality of sounds are input to the microphone in a time series at arbitrary time intervals, it is required to efficiently analyze a portion where the sound exists. That is, it is not preferable to constantly analyze the pitch and the like with respect to the signal input through the microphone, since a useless analysis process is performed even during a time when no sound is actually input. Therefore, it is efficient to extract a section (effective section) where a sound actually exists from a signal input to the microphone and to perform a complicated analysis process such as pitch analysis only on the extracted effective section. is there. The conventional effective section extraction method for this purpose simply extracts the effective section by comparing a predetermined reference level with the input signal level. Therefore, when the level of the input sound fluctuates subtly, the reference level is particularly high. If it fluctuates in the vicinity, there is a problem that the extraction of the effective section becomes inaccurate.

  SUMMARY OF THE INVENTION It is an object of the present invention to provide a sound signal analyzing apparatus and method capable of analyzing a stationary part of a musical sound, that is, a part corresponding to one note, based on an input sound from a microphone or the like. For example, even when the pitch or level of an input sound from a microphone or the like is slightly fluctuated, a sound signal analysis capable of analyzing a stationary part of a musical sound other than the fluctuated part, that is, a part corresponding to one note. An apparatus and method are provided. More specifically, the present invention effectively analyzes a stationary part of an input sound signal, and accurately analyzes the pitch of the sound based on the result. This analysis result can be output as electronic musical score information in the form of MIDI information or the like, if necessary, so that audible melody input by human voice or actual musical instrument performance can be automatically output. It is an object of the present invention to provide a technique capable of accurately performing musical score conversion.

  The sound signal analyzer of the present invention according to claim 1, wherein input means for inputting an arbitrary sound signal formed of a time series of one or a plurality of notes, and one of the input sound signals. A section detecting means for respectively detecting a section estimated to correspond to one note; and arranging the detected section on a grid divided at time intervals corresponding to a predetermined note length in accordance with the time series. One grid position closest to a predetermined one of the start or end ends of each section is assigned to each section. As a result, if a plurality of sections are assigned to the same grid position, Means for selecting one section having a long time length as a valid note.

When detecting sections that are estimated to correspond to notes one by one from the input sound signal, all of the detected sections do not necessarily correspond to valid notes. Some may be invalid. In order to determine an invalid section, a grid divided at time intervals corresponding to a predetermined note length (for example, a minimum allowable note length) is advantageously used. That is, each of the detected sections is arranged on a grid divided at time intervals corresponding to a predetermined note length in accordance with the time series, and one grid position closest to the start end of each section is assigned to each section. In contrast, when a plurality of sections are assigned to the same grid position, one section having the longest time length is selected as a valid section.

  Further, according to the present invention, the degree of coincidence between the waveforms of adjacent sections is calculated, a portion where the degree of coincidence is larger than a predetermined value is detected as a voiced section, and a section having a high degree of coincidence within the detected voiced section is calculated. , The degree of coincidence is sequentially calculated with the waveforms in the sections on both sides of the waveform, and a steady section is detected based on the degree of coincidence, and the pitch and the like of the sound signal are analyzed / detected in the steady section. You may do so. Since a section having a high degree of coincidence in a voiced section serves as a reference for a vowel of a voice, a change in a vowel can be detected based on the degree of coincidence determined based on the vowel. The thus detected steady section may be recognized as a vowel, that is, a section inferred as one note.

  Furthermore, in the present invention, each of the above-described configurations, that is, the invention, can be configured not only as a device invention of a voice analysis device, but also as a method invention of a voice analysis method. Further, the embodiment of the present invention can be implemented in the form of a computer program, and the present invention can be implemented in the form of a recording medium storing such a computer program. Included in the range.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
FIG. 2 is a hardware block diagram showing the configuration of an electronic musical instrument incorporating a musical sound information analyzer and a performance information generator according to the present invention.

The electronic musical instrument is controlled by a microcomputer including a microprocessor unit (CPU) 1, a program memory 2, and a working memory 3.
The CPU 1 controls the operation of the entire electronic musical instrument. A program memory 2, a working memory 3, a performance data memory 4, a key press detection circuit 5, a microphone interface 6, a switch detection circuit 7, a display circuit 8, and a sound source circuit 9 are supplied to the CPU 1 via a data and address bus 1E. Each is connected.

  The program memory 2 stores various programs (system programs, operation programs, and the like) of the CPU 1, various data, and the like, and is configured by a read-only memory (ROM).

The working memory 3 temporarily stores performance information and various data generated when the CPU 1 executes a program. A predetermined address area of a random access memory (RAM) is assigned to each of the working memories 3. It is used as a buffer, table, etc.
The performance data memory 4 stores performance information (MIDI data) generated based on a sound input from a microphone or the like.

  Further, the CPU 1 may be connected to a hard disk device 1H or the like and may store therein various data such as automatic performance data and chord progression data, and may further store the operation program. In addition, these operation programs are stored in the hard disk device 1H without storing the operation programs in the ROM 2, and by reading them into the RAM 3, the same operation as when the operation programs are stored in the ROM 2 is performed on the CPU 1. Can be done. In this way, an operation program can be easily added or upgraded. Also, a removable external storage medium 1G, for example, a CD-ROM or the like may be provided. Various data and an arbitrary operation program may be stored in the external storage medium 1G, for example, a CD-ROM. The operation program and various data stored in the CD-ROM can be read out by a CD-ROM drive (not shown), and can be transferred and stored in the hard disk drive 1H. This makes it possible to easily perform new installation and version upgrade of the operation program.

  The communication interface 1F is connected to a data and address bus 1E, and can be connected to various communication networks such as a LAN (local area network) and the Internet via the communication interface 1F. Data may be exchanged. Thereby, when the operation program and various data are not stored in the hard disk device 1H, the operation program and various data can be downloaded from the server computer. In this case, a command requesting downloading of an operation program and various data is transmitted from the electronic musical instrument, which is a musical sound generation device serving as a client, to a server computer via a communication interface and a communication network. The server computer transmits predetermined operation programs and data to the electronic musical instrument 1 via the communication network in response to the command. The electronic musical instrument receives these operation programs and data via the communication interface and stores them in the hard disk device. Thus, downloading of the operation program and various data is completed.

  The keyboard 10 has a plurality of keys for selecting a pitch of a musical tone to be pronounced, has a key switch corresponding to each key, and has a key pressing speed detecting device and a key pressing key as necessary. It has touch detection means such as a pressure detection device.

  The key press detection circuit 5 includes a circuit composed of a plurality of key switches provided corresponding to respective keys of the keyboard 10 for designating the pitch of a musical tone to be generated. When a key is released, a key-on event is output, and when a key is newly released, a key-off event is output. In addition, a process of generating touch data by determining a key pressing operation speed or pressing force at the time of key pressing is performed, and the generated touch data is output as velocity data. As described above, data such as a key-on event, a key-off event, and velocity are represented by data conforming to the MIDI standard (hereinafter, referred to as “MIDI data”), and include data indicating a key code and an assigned channel.

  The microphone 1A converts a voice signal or a musical instrument sound into a voltage signal and outputs the voltage signal to the microphone interface 6. The microphone interface 6 converts an analog voltage signal from the microphone 1A into a digital signal and outputs the digital signal to the CPU 1 via the data and address bus 1E.

  The numeric keypad and various switches 1B are configured to include various operators such as a numeric keypad for inputting numerical data, a keyboard for inputting character data, and a start / stop switch for a note conversion process (sound signal analysis process and performance information generation process). Is done. In addition, various controls for selecting, setting, and controlling the pitch, tone, effect, and the like are also included, but the details thereof are known and will not be described.

  The switch detection circuit 7 detects the operation state of each operator of the numeric keypad & various switches 1B, and outputs switch information corresponding to the operation state to the CPU 1 via the data and address bus 1E.

The display circuit 8 displays various information such as the control state of the CPU 1 and the contents of the setting data on the display 1C. The display 1C is composed of a liquid crystal display panel (LCD), a CRT, or the like, and its display operation is controlled by the display circuit 8.
A GUI (Graphical User Interface) is configured by the numeric keypad & various switches 1B and the display 1C.

  The tone generator 9 is capable of simultaneously generating a tone signal on a plurality of channels, receives data and MIDI data on a given tone track via an address bus 1E, and generates a tone signal based on the data. And outputs it to the sound system 1D.

The tone generator circuit 9 can simultaneously generate tone signals on a plurality of channels by using a single circuit in a time-division manner to form a plurality of tone channels, or a single tone channel is constituted by a single circuit. It may be of such a type as to be performed. Further, any tone signal generation method in the tone generator 9 may be used. For example, a memory reading method (waveform memory method) for sequentially reading out tone waveform sample value data stored in a waveform memory in accordance with address data that changes in accordance with a pitch of a musical tone to be generated, or a phase angle parameter An FM method for executing a predetermined frequency modulation operation as data to obtain tone waveform sample value data, or an AM method for executing a predetermined amplitude modulation operation using the above address data as phase angle parameter data to obtain tone waveform sample value data, etc. May be appropriately adopted. In addition to these methods, a physical model method of synthesizing a musical tone waveform by an algorithm that simulates the sounding principle of a natural musical instrument, a harmonic synthesizing method of synthesizing a musical tone waveform by adding a plurality of harmonics to a fundamental wave, A formant synthesis method for synthesizing a musical tone waveform using a formant waveform having a specific spectral distribution, an analog synthesizer method using VCO, VCF, and VCA may be employed. In addition, the tone generator circuit is not limited to the one that uses the dedicated hardware, and the tone generator circuit may be configured using a DSP and a microprogram. Alternatively, the tone generator circuit may be configured using a CPU and software programs. It may be.
The tone signal generated by the tone generator 9 is generated via a sound system 1D including an amplifier and a speaker.

  Next, an example in which the electronic musical instrument according to the present invention operates as a sound signal analyzer and a performance information generator will be described.

FIG. 1 is a diagram showing a main flow when the electronic musical instrument of FIG. 2 operates as a performance information generating device. The main flow is executed sequentially in the following steps.
Step 11: First, an initial setting process is performed, and an initial value is set to each register and flag in the working memory 3 in FIG.
At this time, when the musical note processing start switch on the numeric keypad & various switches 1B is turned on, a series of processing from step 12 to step 18 is performed.

  Step 12: This step is performed when it is determined that the musical note processing start switch is turned on, and is input from the microphone 1A via the microphone interface 6 in response to the turn-on operation. A voltage waveform of a voice signal or a musical instrument sound is sampled at a predetermined period (for example, 44.1 kHz), and the result is stored in a predetermined area in the working memory 3 as a digital sample signal. Since this sampling process is performed by a conventionally known method, the details are omitted here.

  Steps 13 to 16 correspond to the note conversion process corresponding to the ON operation of the note conversion start switch. In this note conversion process, sampled audio signals and digital sample signals of musical instrument sounds are analyzed in various ways and converted into a pitch sequence, that is, MIDI data that can be displayed in a musical score.

  Step 13: Based on the digital sample signal obtained as a result of the audio sampling processing in step 12, an effective section detection process for detecting a section where a musical sound is present, that is, an effective section is performed. Details of the valid section detection processing will be described later.

  Step 14: As a result of the effective section detection processing in step 13, a stable section detection processing for subdividing each detected effective section into stable sections having a more stable level is performed. The details of the stable section detection processing will be described later.

  Step 15: As a result of the stable section detection processing in step 14, a steady section detection processing for detecting a steady part (a part corresponding to one note) of a musical sound present in each detected stable section is performed. The details of the stationary section detection processing will be described later.

  Step 16: A pitch sequence determination process for allocating the most optimal note for each steady section obtained as a result of the processes of steps 13 to 15 is performed. That is, in this step, MIDI data is generated. Details of the pitch sequence determination processing will be described later.

  Step 17: A score creation process for creating a score based on the MIDI data generated by the process of step 16 is performed. Since the score creation processing can be easily realized by the conventional technique, the details are omitted.

  Step 18: Perform an automatic performance process based on the MIDI data generated by the process of step 16. Since the automatic performance process can be easily realized by the conventional technique, the details are omitted.

  FIG. 3 is a diagram showing details of the valid section detection processing in step 13 of FIG. Hereinafter, how the effective section is detected from the digital sample signal obtained in step 12 will be described with reference to FIGS. 7 and 8.

  Step 31: An average sound pressure level is calculated based on the digital sample signal obtained in step 12. FIG. 7 is a diagram illustrating an example of a waveform value of an audio signal sampled at a sampling frequency of 44.1 kHz, that is, a digital sample signal. FIG. 7 shows waveform values for about 20 points. In step 31, the average of the sample amplitude values over a predetermined number of samples (for example, the number of samples corresponding to a time corresponding to 10 msec) is obtained, and the average is set as the average sound pressure level. Therefore, in the case of the sampling period of 44.1 kHz, the predetermined number of samples is “441”, and the average value of a certain sample point is the total value of each point 10 msec before the last point, that is, This is a value obtained by dividing the sum of the waveform values 441 points before the final point by 441. Since there is no waveform value for 441 points from the point 0 to the point 440, the average of the waveform values from the point 0 to the corresponding point is set as the average value of the point. In this way, time-series average sound pressure level information is obtained for each sample timing.

  FIG. 7 illustrates a case where the average value of the waveform values for 15 points is calculated as the average sound pressure level for convenience of explanation. Therefore, up to the first 15 points, the total value of the waveform values up to that point is divided by the number of points. The sum of the waveform values is obtained by summing the absolute values.

  FIG. 8A is a graph showing the average sound pressure level value obtained in this way, with the sampling point on the horizontal axis. Hereinafter, the curve formed by the average sound pressure level in FIG. 8A is referred to as an average sound pressure level curve.

  When an average sound pressure level is obtained at every 15 points as shown in FIG. 7, a low-pass filter with a cutoff frequency of about 10 Hz is applied to smooth the level fluctuation. Therefore, when actually averaging the waveform values for 441 points, it is desirable to apply a low-pass filter having a cutoff frequency of about 80 to 100 Hz to smooth the level fluctuation.

  Also, here, a case has been described where the average value of a certain sampling point is obtained, and the average sound pressure level is obtained by summing the waveform values of a predetermined number of points before that point. The waveform values at a predetermined number of points before and after may be totaled, or the waveform values at a predetermined number of points after the sampling point may be totaled.

  Step 32: The average sound pressure level curve obtained in step 31 as shown in FIG. 8A is classified into an effective section or an invalid section based on a predetermined threshold value. In this process, the threshold value is set to 20% of the maximum waveform value in the average sound pressure level curve. It goes without saying that other values may be used as threshold values. For example, the average value of the average sound pressure level curve is set as the threshold value, 80% of the average value is set as the threshold value, and the half value of the maximum value of the average sound pressure level curve is set as the threshold value. You may.

  The threshold value is indicated by a dotted line as shown in FIG. Therefore, the intersection point between the dotted line (threshold) and the average sound pressure level curve is a boundary between the valid section and the invalid section, a section larger than the dotted line (threshold) is a valid section, and a section smaller than the dotted line (threshold) is the invalid section. It becomes. In FIG. 8B, the valid section is indicated by a circle, and the invalid section is indicated by a cross.

  Step 33: If the minimum length required for human recognition of the pitch is 0.05 msec, the invalid section smaller than the minimum length is changed to the valid section from the invalid sections determined in step 32. For example, when the sampling period is 44.1 kHz, invalid sections of 2205 or less in sampling number are changed to valid sections. In FIG. 8B, the third and fifth invalid sections from the left correspond to this short invalid section. Therefore, as a result of the processing in step 33, FIG. 8B becomes as shown in FIG. 8C, and the valid section is extended. In this processing, the invalid sections existing at the beginning and end of the entire section correspond to short invalid sections, but are represented by a special area which is not changed to a valid section just because it is short. are doing.

  Step 34: A process of changing a short valid section of 0.05 msec or less from the pattern of the valid section and the invalid section obtained as a result of the processing of the step 33 to the invalid section is performed. This process is performed in the same manner as in step 33. In FIG. 8C, the rightmost effective section corresponds to the short effective section. Therefore, as a result of the processing in step 34, FIG. 8C becomes as shown in FIG. As is clear from FIG. 8D, the effective sections are all four sections from the first section to the fourth section. Note that the mark at the end of the section is regarded as the fourth effective section.

  Step 35: The average value of the average sound pressure level curve of the effective section specified in step 34 is obtained, and if the average value is smaller than a predetermined value, a final effective section in which that part is regarded as an invalid section is checked. This average value is obtained by dividing the sum of the average sound pressure level values of the points existing in the effective section by the effective section length. The average value of the average sound pressure levels obtained in this manner is shown below each section in FIG. The first section is 60, the second section is 25, the third section is 45, and the fourth section is 15. If the average value of the average sound pressure level falls below 30% of the maximum waveform value of the section, the section is regarded as an invalid section. Here, since the second section and the fourth section correspond to each other, each section becomes an invalid section. FIG. 8E is a diagram showing the valid section and the invalid section specified by the valid section check processing in step 35.

  Step 36: A process of extending the valid section specified by the processes of steps 31 to 35 is performed. For example, as shown in FIG. 8 (F), 15% of the maximum waveform value is set as the extension permission level, and a line is drawn on that portion, and the boundary line for specifying the valid section is extended to the line of the extension permission level. . That is, the extension processing is performed while checking whether the average sound pressure level curve has fallen below the extension permission level while checking the rise and fall of the average sound pressure level curve from the end of each effective section to the outside. At this time, when the descending is reversed to the ascending or when the descending falls below the extended permission level, the area up to that point is regarded as an effective section.

  FIG. 8G is a diagram showing another example of the effective section extension processing. The extension permission level is set to 5% of the maximum waveform value, and the position where the lowering of the average sound pressure level curve ends is set as the end of the effective section. Alternatively, the position at which the ascent starts may be the end. According to the extension processing, the extension width of the first section and the third section is larger than in the case of FIG. In this way, an effective section that can be recognized as a pitch by a human is finally determined.

If the extension permission level is low and the effective section is at a short distance, the extension position at the end of a certain effective section may be close to the extension position at the beginning of the next effective section. It may also be at the same position. In addition, the boundary position changes depending on whether a part where the descent ends or a part where the ascent starts is delimited. As a result of the extension processing, when the valid sections overlap, both intermediate positions may be set as the boundary positions.
8 (F) and 8 (G), the case where the effective section is extended before and after is described. However, only the forward direction or the backward direction may be performed. Further, when extending forward and backward, the extension permission level may be made different between the forward direction and the backward direction.

FIG. 4 is a diagram showing details of the stable section detection processing in step 14 of FIG. Hereinafter, a stable section detection process for detecting an area with a stable level is performed on the average sound pressure level curve in the effective section obtained in step 13. The operation of the stable section detection processing will be described with reference to FIG. FIG. 9 shows a case where a stable section is detected for the first effective section from point A to point B in FIG.
Step 41: The slope frequency is calculated based on the average sound pressure level curve in the effective section detected in FIG. In this process, as shown in FIG. 9B, the calculation width for calculating the inclination is set to, for example, 100 points, and the shift amount of the calculation width is set to, for example, 50 points. While calculating the inclination frequency. Assuming that the point A is the sample point “000”, the slope between the sample points “000” and “100” is calculated, and then the slope is shifted by 50 points between the sample points “050” and “150”. Obtain the slope frequency. For example, when the average sound pressure level of the sample point “000” is “325” and the average sound pressure level of the sample point “100” is “1576”, the slope frequency is (1576-325) / 100 = 12. It becomes 51. Thereafter, the slope degrees are calculated in order, such as between sample points "100" and "200", between "150" and "250", and between "200" and "300". FIG. 9B shows an example of the calculated inclination degree. As is clear from the figure, the gradient between the sample points "000" and "100" is 12.51, the gradient between the sample points "050" and "150" is 32.42, and the sample points "100" and "100". The slope between sample points "200" and "250" is 20.12, the slope between sample points "150" and "250" is 11.84, the slope between sample points "200" and "300" is 5.24, and the sample point " The slope between sample points "300" and "400" is 4.82, the slope between sample points "300" and "400" is 2.34, and the slope between sample points "350" and "450" is 3.89. , The inclination between sample points “400” and “500” is 5.36. These gradients will be stored as the gradients at the former sample point. That is, the slope frequency of the sample point “000” is 12.51, and the slope frequency of the sample point “050” is 32.42, and the slope frequency is stored for each sample point.
In this way, the gradient frequencies in all the sections from the point A to the point B are calculated, and the stable section extraction processing in the next step 42 is performed.

Step 42: Based on the gradient calculated in step 41, a stable section is extracted. That is, if the slope frequency at each sample point is less than or equal to a predetermined value (for example, 10) as a stable portion, and the number of sample points regarded as the stable portion is equal to or more than a predetermined number, that is, for a predetermined time, Then, the continuous stable portion is defined as a stable section. The predetermined time is set to, for example, about 2000 sample points in consideration of the tempo. In the case of the average sound pressure level curve as shown in FIG. 9A, three sections a, b, and c as shown in FIG. 9C are searched as stable sections.
Step 43: Based on the existence of the stable section extracted in step 42, the human first notices that there is a starting point of the sound which is the trigger of the note near the starting point of the stable section. Here, in order to determine the vicinity of the starting point of the note, the stable section extracted in step 42 is extended.
When extending this stable section, that is, when determining the start point of a note, for the first stable section a, the point A is necessarily the start point of the note in the stable section a, and for the last stable section c, Inevitably, the point B is the end point of the note in the stable section c. However, the note end point of the stable section a and the note start point of the stable section b cannot be easily obtained. Therefore, the sample point having the largest gradient between the end point of the stable section and the start point of the next stable section is determined as the note end point of the stable section and the note start point of the next stable section. The note end point of each stable section a and the note start point of stable section b are point C as shown in FIG. 9D, and the note end point of stable section b and the note start point of stable section c are point D.
In the above description, the case where the sample point having the largest inclination is the note start point of the stable section and the note end point of the next stable section has been described. However, the present invention is not limited to this. The sample point when the stability degree first exceeds a predetermined value (threshold value) up to the start point of the stable section may be set to the note end point (note start point), or the start point of the stable section. A sample point in the case where the value falls below a predetermined value (threshold value) immediately before may be set as a note end point (note start point), or a sample point obtained by the above three methods may be compositely calculated and newly calculated. Alternatively, a note end point (note start point) may be obtained. The sections AC, CD, and DB obtained in this manner become stable sections corresponding to the respective levels. That is, in the case of FIG. 9, the stable section corresponding to the level of the stable section a is AC, the stable section corresponding to the level of the stable section b is CD, and the stable section corresponding to the level of the stable section c is DB.

FIG. 5 is a diagram showing details of the stationary section detection processing in step 15 of FIG. How the steady section is detected from the stable sections obtained in step 14 will be described in detail with reference to FIGS. 10 to 17.
When analyzing musical audio signals such as voices and musical sounds, it is important to know where the stationary part is. This is because, in a tone other than the rhythm type, the pitch is determined by the periodicity of the stationary part, and the tone value is determined by using the stationary part as a skeleton. In this embodiment, the stationary part is a section corresponding to one note when expressed as a musical score, and attention is paid to changes in the three major elements of tone, pitch, and velocity, and humans are assigned one sound. Means to detect on the time axis a section recognized as.
Hereinafter, the stationary section detection processing will be described according to the steps in FIG.

  In order to detect the stationary section, it is necessary to first detect the reference position of the period of the sound signal waveform. The method of detecting the reference position is generally roughly divided into one of the zero cross position detection method and the peak position detection method. In order to detect the reference position of the period by the 0 cross position detection method, it is difficult to detect the overtone without removing as much as possible of the overtone with a filter or the like, and it is necessary to divide the band. In the case of the peak position detection method as well, it is desirable to remove overtones as much as possible, but since it is not as severe as the 0 cross position detection, it is only necessary to apply a band-pass filter using the soundable frequency band of the voice or instrument as a cutoff frequency, and the band is divided. There is no particular need to perform such processing. Therefore, the peak position detection method is simpler in procedure, and a method that can obtain a reasonable result is desirable. Therefore, in this embodiment, the case where the reference position of the cycle is detected by the peak position detection method will be described.

Step 51: Pass a first-order band-pass filter (first-order BPF) to delete a predetermined harmonic. This is to apply a band-pass filter using a band capable of generating sound as a cutoff frequency. In the case of voice, the band that can be pronounced by humans is about 80 to 1000 Hz, which is necessary for almighty analysis without limiting the user. However, when the number of users is limited, it is possible to reduce errors due to overtones and improve the detection accuracy by narrowing the soundable band to some extent. In the case of a guitar, the frequency is about 80 to 700 Hz. However, if the pitch frame is determined in advance, the accuracy is improved. The accuracy can be improved by setting in advance the differences between the instruments. FIG. 10A shows a part of the audio waveform after the primary BPF processing.
Step 52: A peak reference position detection process, which is a reference for one cycle, is performed on the musical tone waveform signal obtained by the first-order BPF process in step 51 using a peak position detection method. This peak position detection method is performed by a known method. The peak level of the musical tone waveform is detected and held by a predetermined time constant circuit, and the held value is used as a threshold voltage. When the voltage exceeds the threshold voltage, the next peak level is obtained. , And by repeating the sequence sequentially, a peak position as shown in FIG. 10A can be detected. FIG. 10A is a diagram showing a state of a threshold hold voltage when the peak position is detected.
A peak position as shown in FIG. 10B is detected from the audio waveform of FIG. In FIG. 10B, the peak reference positions P1, P2, P3, and P6 all appear regularly at predetermined positions. However, the peak reference positions P4 and P5 have peaks at positions including errors due to slight disturbance of the audio waveform. Is appearing.
This is because the peak position appears continuously as shown in FIG. 10A because the band of the cutoff frequency of the first-order BPF processing in step 51 covers a wide range.

Step 53: Based on the peak reference position detected in step 52, a basic section starting from a certain peak reference position and a section up to the next peak reference position immediately after the basic section (hereinafter, referred to as a moving section). A comparison is made as to whether the waveforms in the two sections between are the same.
Considering the peak reference position shown in FIG. 10B, a section d is from the peak reference position P1 to the peak reference position P2, and a section e is from the peak reference position P2 to the peak reference position P3. At this time, since both sections d and e are larger than the minimum band length and smaller than the maximum band length, the section d is a basic section, the section e is a moving section, and is subjected to a waveform comparison process described later.
Next, the section e becomes the basic section, and the section from the peak reference position P3 to the peak reference position P4 becomes the section f. At this time, since both sections e and f are larger than the minimum band length and smaller than the maximum band length, the section e becomes a basic section and the section f becomes a moving section, and is subjected to a waveform comparison process described later.
However, since the section from the peak reference position P4 to the peak reference position P5 is smaller than the minimum bandwidth, the section g from the next peak reference position P5 to the peak reference position P6 is not a section to be compared. f is a waveform comparison target.
As a result of the waveform comparison processing, the section f and the section g are recognized as different waveforms from the other sections d and e. First, a working memory (RAM) is provided with a data area in which a coincidence flag or a non-coincidence flag is written, using the peak reference position information as an address. Then, in the case as shown in FIG. 10B, since it is determined that the section d and the section e match, a match flag is written in the data area related to the peak reference position information P2 corresponding to the section e. On the other hand, since it is determined that the section e does not match the section f, a mismatch flag is written in the data area related to the peak reference position information P3 corresponding to the section f. Since the section from the peak reference position P4 to the peak reference position P5 is smaller than the minimum bandwidth, a mismatch flag is written in the data area related to the peak reference position information P4 and P5. It is assumed that a match flag has been written in the data area relating to the peak reference position information P1 and P6. FIG. 10C shows the state of the match flag and the non-match flag sequentially written together with the peak reference position information.

The waveform comparison processing is performed by a method of calculating an error rate described later.
FIG. 13 is a diagram for explaining a method of calculating an error rate performed in the waveform comparison processing.
First, it is assumed that two waveforms whose error rates are to be calculated are a comparison wave 1X and a comparison wave 2X as shown in FIG. This waveform is a range divided by the peak reference position in FIG. First, the amplitude values of the comparison wave 1X and the comparison wave 2X are normalized such that the maximum amplitude value becomes 100%. First, the comparison wave 1X becomes the comparison wave 1Y, and the comparison wave 2X becomes the comparison wave 2Y. Here, since the comparative wave 2X has a shorter length in the time axis (horizontal axis) than the comparative wave 1X, the comparative wave 2X extends so as to have the same time width as the comparative wave 1X. That is, the time axis of the comparison wave 2Y is extended to make the comparison wave 2Z.
The error rate is calculated between the comparison wave 1Y and the comparison wave 2Z.
FIG. 13 is a diagram showing specific values when calculating the error rate between the comparison wave 1Y and the comparison wave 2Z. In the drawing, a case will be described in which the error rate is calculated for the waveforms of the first cycle of the comparison wave 1Y and the comparison wave 2Z, that is, for 24 samplings.
The difference is calculated for the same sampling position of the comparison wave 1Y and the comparison wave 2Z, and the sum of the absolute values of the difference is calculated. In the case of FIG. 13, the total value of the absolute values is 122. By dividing this by the number of samplings 24, the error rate is obtained. In this case, the error rate is 5. Therefore, if the threshold value of the same waveform is set to 10, the error rate 5 in the case of FIG. 13 is 10 or less, so that the same waveform is processed. In FIG. 13, each waveform is normalized with 1000 as the maximum level.

Step 54: Using the result of the waveform comparison process of step 53, the sections in which the error rate is smaller than a predetermined value (for example, 10) are connected to each other to be a pseudo-match section, and extracted from each match section. The maximum value and the minimum value of the pitch are detected, and the cutoff frequency band is determined based on the maximum value and the minimum value. For example, it is assumed that the minimum value of the pitch in a plurality of matching sections obtained as a result of the waveform comparison processing is 235 points, and the maximum value is 365 points. Assuming that the minimum value is reduced by 10% and the maximum value is increased by 10% in order to give a margin to this matching section, the matching section is changed from about 212 points to about 402 points. This corresponds to a frequency band of an audio signal of 110 Hz to 208 Hz when the sampling frequency is 44.1 kHz. Therefore, the cutoff frequency band is from 110 Hz to 208 Hz.
Step 55: Using the new cut-off frequency band determined in Step 54, the signal is passed through a second-order band-pass filter (second-order BPF) to remove unnecessary harmonics. For example, in the case described above, the cutoff frequency band is in a range from 110 Hz to 208 Hz. This can reduce errors due to overtones and improve the detection accuracy.
Step 56: The same processing as the peak reference position detection processing of step 52 is performed.
Step 57: The same processing as the waveform comparison processing of step 53 is performed.
By the series of processing from step 55 to step 57, low frequency and harmonics that cause errors are cut, and more accurate peak reference position detection processing and waveform comparison processing become possible. A section is obtained. By the waveform comparison process of step 57, the waveform of the effective section characterized by the coincidence flag and the non-coincidence flag as shown in FIG. 10 (C) is divided into three stationary sections X, Y and Z as shown in FIG. Such a pitch sequence is required.

Step 58: The pitch sequence as shown in FIG. 10D obtained by the processing up to step 57 may be used, but in order to further improve the accuracy, the pitch data at each peak reference position is interpolated to obtain one sample. Interpolation is performed so that one pitch data is obtained for each point. In this case, as apparent from FIG. 10 (A), for the sample points before the first peak reference position and the sample points after the last peak reference position, there is no pitch data for interpolation, so that the pitch Interpolation cannot be performed. Therefore, pitch data at the first peak position is directly applied to the sample points before the first peak reference position, and pitch data at the last peak position is applied to the sample points after the last peak reference position. Then, between the respective peak reference positions, the values of both pitch data are linearly interpolated and applied. For example, in FIG. 10B, assuming that the pitch data at the peak reference position P1 is PD1 and the pitch data at the peak reference position P2 is PD2, at an arbitrary sample point PV between the peak reference positions P1 and P2. The pitch data is obtained by the following equation.
(PD2-PD1) x (PV-PA) / (P2-P1)

Step 59: Band-pass filter processing is performed using the pitch data for each sample point obtained by the processing of step 58. That is, since the pitch data changes with the passage of time, the so-called time-varying bandpass filter (BPF) processing in which the cutoff frequency band also varies with time is performed. As a result, the musical tone waveform signal is transformed into a waveform close to a sine waveform. By performing peak position detection processing on such a waveform, ideal peak position detection can be performed. In addition, since the comparison process can be performed based on this, the error can be minimized, so that the same waveform (same vowel) section can be found with high accuracy.
Step 5A: The same processing as the peak reference position detection processing in step 52 is performed on the musical tone waveform that has undergone the time-varying BPF processing in step 59.
Step 5B: The same processing as the waveform comparison processing in step 53 is performed on the musical tone waveform that has undergone the time-varying BPF processing in step 59.

In the above description, in the steps 52, 56 and 5A of the stationary section detection processing in FIG. 5, the case where the peak reference position is detected by focusing only on the plus side of the musical sound waveform has been described. As described above, a musical tone waveform having a pitch may have strong peaks on both the plus side, the minus side, and both the plus side and the minus side. Therefore, as described above, when a peak appears strongly on the plus side, the pitch can be detected by focusing on the plus side and detecting the peak reference position. In this case, there is no problem if a sharp peak appears on both sides, but it may appear strongly on the negative side. In such a case, it is needless to say that it is better to detect the peak reference position by paying attention to the strong appearance side. If the peak reference position is detected by paying attention to the direction in which the peak appears weakly, the detection of the peak reference position itself becomes ambiguous, and the pitch may not be detected as expected. The phenomenon that the peak appears to be biased to either the plus side or the minus side varies variously depending on the conditions of the human being or the musical instrument at each time, so if you focus on either side and detect the peak reference position At present it is not possible to say that it is good.
Therefore, the tone waveform is checked in advance so that a strong peak may appear on either side, and whether the peak appears strongly on the plus side or the minus side is detected, and based on the tone waveform on the detected side. Then, the peak reference position and the pitch may be detected. For example, it is assumed that the musical tone waveform in the stable section after the primary BPF processing in step 51 in FIG. 5 is as shown in FIG. In the case of this tone waveform, a strong peak appears on the minus side, and a weak peak appears on the plus side. When the peak reference position is detected on both sides of this musical tone waveform, the peak intensity is different, but stable peaks appear on both sides. It is possible to detect the reference position. Therefore, in the case of the musical tone waveform, there is no problem even if the stationary section detection processing of FIG. 5 is performed by focusing on the plus side. However, depending on the musical sound waveform, in the case of a waveform having a relatively short period and a long repetition, the peak may gradually become dull. In some cases, an accurate peak reference position cannot be detected. Therefore, even in the case of the waveform as shown in FIG. 11, it is desirable to pay attention to the negative side where the strongest peak appears.
Therefore, for each stable section detected by the stable section detection processing of FIG. 4, it is detected whether the maximum of the absolute value of the musical tone waveform exists on the plus side or the minus side in the entire section, and attention is paid to the side to be detected. Then, the peak reference position may be detected. In FIG. 11, since the maximum of the absolute value exists on the minus side, it is desirable to detect the peak reference position by paying attention to the minus side. Thus, the peak reference position can be detected without being disturbed by overtones or the like.

In the above-described embodiment, the peak level of the musical tone waveform is detected and held by a predetermined time constant circuit, and the held value is set as a threshold hold voltage, which is then equal to or higher than the threshold hold voltage. Is held as the next peak level, and the peak position is detected by sequentially repeating this. However, according to this method, it is determined whether or not a desired peak reference position can be detected depending on how much the time constant is set. In the case of a sound, there is a problem that an orderly peak often does not easily appear. Therefore, in the above-described embodiment, the determination whether or not the detected peak reference position is an accurate one that can be used in the subsequent frequency band determination processing is performed by the waveform comparison processing based on the detected peak position. Had gone by. This means that the peak position detected by the above-described peak reference position detection processing may not be very accurate. Therefore, when detecting the peak level of the musical sound waveform, the time constant is set to a relatively small value, a large number of possible peak reference positions are extracted from the musical sound waveform, and based on the extracted peak reference position. A waveform comparison process may be performed to sequentially determine the peak reference position. In this case, if the peak positions are detected by paying attention to the plus side of the musical tone waveform as shown in FIG. 14, three peak positions are extracted in one cycle. When the waveform comparison processing is performed based on the three peak positions per one cycle, the time required for the processing becomes very long. Therefore, here, first, based on the section determined to have the same waveform, the subsequent detection processing of the same waveform section is efficiently performed.
For example, in the case of a musical sound waveform as shown in FIG. 14, Pa to Po are detected as peak positions. Accordingly, the waveform comparison processing is first performed for the following 16 combinations starting from the peak position Pa.
(Pa-Pb) and (Pb-Pc), (Pa-Pb) and (Pb-Pd),
(Pa-Pb) and (Pb-Pe), (Pa-Pb) and (Pb-Pf),
(Pa-Pc) and (Pc-Pd), (Pa-Pc) and (Pc-Pe),
(Pa-Pc) and (Pc-Pf), (Pa-Pc) and (Pc-Pg),
(Pa-Pd) and (Pd-Pe), (Pa-Pd) and (Pd-Pf),
(Pa-Pd) and (Pd-Pg), (Pa-Pd) and (Pd-Ph),
(Pa-Pe) and (Pe-Pf), (Pa-Pe) and (Pe-Pg),
(Pa-Pe) and (Pe-Ph), (Pa-Pe) and (Pe-Pi)
As a result, it is determined that the waveforms of the section (Pa-Pd) and the section (Pd-Pg) match. As a result, the peak position Pa becomes the pitch reference position PPa, and the other peak positions Pb and Pc are excluded from the candidates. Then, waveform comparison processing is performed for 16 combinations in the same manner starting from the peak position Pd, and the peak position Pd becomes the pitch reference position PPd. Hereinafter, similarly, the pitch reference positions are successively detected.
When the same waveform section is detected from the 16 patterns, the error rates of all 16 patterns are calculated, and the smallest one having an error rate equal to or less than a predetermined value (for example, 10) is set as the same waveform section. Alternatively, when an error rate that is less than or equal to a predetermined value (for example, 10) appears in the sequentially calculated error rates, it may be set as the same waveform section.

In this way, if the error rate calculation processing is performed for a large number of combinations to extract the same waveform section, it takes a considerable amount of time. Therefore, here, it is determined that the waveforms have the same waveform as described above. Based on the section, the detection processing of the subsequent same waveform section is performed. That is, among the above 16 combinations,
(Pa-Pb) and (Pb-Pd), (Pa-Pb) and (Pb-Pe),
(Pa-Pb) and (Pb-Pf), (Pa-Pc) and (Pc-Pd),
(Pa-Pc) and (Pc-Pg), (Pa-Pd) and (Pd-Pe),
(Pa-Pd) and (Pd-Pf), (Pa-Pe) and (Pe-Pf),
(Pa-Pe) and (Pe-Pg)
No comparison processing is performed for the nine cases. This is because the ratio of the waveform section lengths to be compared is almost twice, and it is impossible to obtain the same waveform without comparison, so that those comparisons are not performed in advance.
Therefore, here, the waveform comparison processing is performed for the following seven combinations.
(Pa-Pb) and (Pb-Pc), (Pa-Pc) and (Pc-Pe),
(Pa-Pc) and (Pc-Pf), (Pa-Pd) and (Pd-Pg),
(Pa-Pd) and (Pd-Ph), (Pa-Pe) and (Pe-Ph),
(Pa-Pe) and (Pe-Pi)
Then, it is determined that the waveforms of the section (Pa-Pd) and the section (Pd-Pg) match as in the case described above. As a result, the peak position Pa becomes the pitch reference position PPa, and the other peak positions Pb and Pc are excluded from the candidates. Next, the waveform comparison processing is performed for the seven combinations in the same manner starting from the peak position Pd. Here, the section to be compared next is determined based on the section (Pd-Pg). limit. That is, the waveform comparison processing is performed on the sections (Pg-Pi), (Pg-Pj), and (Pg-Pk) in which the section length of the section (Pd-Pg) becomes ± α. Here, for example, a length of about a quarter of the section (Pa-Pd) is used as α. It goes without saying that other appropriate values may be used for α. As a result of this waveform comparison processing, the section (Pd-Pg) and the section (Pg-Pj) are determined to be the same waveform section. Therefore, since the waveform comparison processing may be performed for the three combinations thereafter, the arithmetic processing becomes very easy.

Step 5C: Extend the steady section obtained by the processing from step 51 to step 5B. That is, as a result of performing the processing from step 51 to step 5B, it is good that each of the steady sections X, Y, and Z is separated by one non-matching section as shown in FIG. When the stationary section is divided by a plurality of non-coincidence sections as in C), each non-coincidence section must be connected to the normal section to extend the normal section. This step 5C is for performing the extension processing of this steady section.
For example, when one stable section is determined to have the same waveform section of the first same vowel portion XX and the second same vowel portion YY as shown in FIG. The first periodic section S1 of the first vowel part XX that is in contact with the beginning of the section and the last periodic section E2 of the second vowel part YY that is in contact with the end of the stable section may be extended along the stable section. . However, the non-coincidence sections N1 to N6 between the first vowel portion XX and the second vowel portion YY cannot be simply extended, and are extended as follows.
First, based on the expansion error rate having a larger error rate tolerance than the waveform comparison processing of steps 53, 57, and 5B, the last period section E1 of the first same vowel part XX and the non-coincidence sections N1, N2, N3, The comparison is performed in the order of N4, N5, and N6, and the unmatched section determined to be smaller than the error rate for expansion is incorporated into the first vowel part XX and expanded. Similarly, the head period section S2 of the second vowel portion YY is compared with the non-coincidence sections N6, N5, N4, N3, N2, and N1 in that order. XX and YY to be expanded. In the case of FIG. 15A, the mismatched sections N1 and N2 are incorporated in the first vowel part XX, and the mismatched section N6 is incorporated in the second vowel part YY. As a result, as shown in FIG. Let's say
The unmatched sections N3, N4, and N5 remaining without being incorporated in each of the homologous vowel sections are incorporated in any of the homologous vowel sections in the following manner. An error rate is obtained by performing a waveform comparison process between the mismatched section N3 and the mismatched section N2 incorporated in the first vowel section XX, and the mismatched section N5 and the mismatched section incorporated in the second same vowel section YY. An error rate is obtained by performing a waveform comparison process with N6, the two error rates are compared, and the smaller error rate (the one with a higher degree of coincidence) is incorporated as the same vowel part and expanded. As a result, since the error rate between the mismatched section N2 and the mismatched section N3 is smaller, the mismatched section N3 is incorporated into the first same vowel portion XX as shown in FIG. This time, the error rate between the non-matching section N2 and the non-matching section N4 is determined, and the smaller error rate is compared with the error rate between the mismatching section N6 and the mismatching section N5. In this way, as shown in FIG. 15C, the non-matching sections N3 and N4 are incorporated into the first vowel part XX, and the mismatching section N5 is incorporated into the second vowel part YY.
When the mismatching section N3 is incorporated into the first same vowel section as described above, the mismatching section N3 is regarded as the first same vowel section as shown in FIG. Needless to say, the mismatched section N4 and the mismatched section N3 at this time may be set as comparison target sections.
When incorporating the smaller error rate into one of the vowel sections, an upper limit value is set for the error rate. If the error rate exceeds the upper limit, the mismatched section is incorporated into the same vowel section. It may not be necessary.
As described above, the gap section is incorporated into the same vowel section on both sides, and the extended processing of the steady section ends.

In the above-described waveform comparison processing, the case where the waveform comparison processing in step 5B is performed on the musical tone waveform subjected to the time-varying BPF processing in step 59 has been described. Since the comparison process is performed on a waveform close to the waveform, even the feature of each vowel may be filtered, and the significance of extracting the vowel section may be reduced. Therefore, a waveform for peak position detection and a waveform for waveform comparison processing may be separately prepared, and the peak position detection and waveform comparison processing may be respectively performed based on the waveforms. That is, the waveform after the time-varying BPF processing is used as it is as the waveform for peak position detection, and the BPF processing for leaving a frequency band waveform several times as long as the frequency component used for the time-varying BPF processing is used for the waveform comparison processing. The waveform obtained by performing the above is used.
For example, when the frequencies of the respective waveform section lengths are obtained based on the peak reference positions detected by the peak reference position detection processing in step 5A, it is assumed that the following frequency sequence is obtained.
134.6 Hz, 135.2 Hz, 145.7 Hz, 135.7 Hz, ...
Therefore, this frequency sequence is used as a basic frequency sequence, and a time-varying BPF process is performed for each frequency band with a frequency band that is an integral multiple of this as a cutoff frequency, and a waveform obtained thereby is synthesized. That is, in the case of the above-described frequency train, the frequency train is twice as large as the fundamental frequency train.
269.2 Hz, 270.4 Hz, 291.4 Hz, 271.4 Hz, ...
As a triple frequency train,
403.8 Hz, 405.6 Hz, 437.1 Hz, 407.1 Hz, ...
As a quadruple frequency train,
538.4 Hz, 540.8 Hz, 582.8 Hz, 542.8 Hz, ...
As described above, the time-varying BPF processing is performed separately using a frequency sequence that is an integral multiple of the fundamental frequency sequence as a cutoff frequency.
The synthesized waveform obtained by synthesizing the waveforms after the BPF processing corresponding to the respective frequency trains thus obtained is used as the target waveform of the waveform comparison processing in step 5B.
As a result, when detecting a vowel section, an accurate vowel section can be accurately detected in accordance with a change in timbre (vowel).
It goes without saying that bandpass filter processing may be performed in which the basic frequency is set to the lowest frequency and the integral frequency of the basic frequency is set to the highest frequency, and this may be used as a target waveform for the waveform comparison processing.

Step 5D: The stationary section obtained by the processing from step 51 to step 5C is subdivided in consideration of the change in pitch and stability, and the final stationary section is determined. In the stationary section detection processing up to step 5C, since the waveform is expanded and compared, even if the pitch change of the voice waveform is caused by a continuous vowel such as "Oh", it is regarded as one and the same sound. It is a mechanism. Therefore, in the case of the musical sound waveform of the musical instrument sound, a situation may occur in which the pitch change of the continuous musical instrument sound cannot be found. Therefore, in the present embodiment, the state of the pitch change is checked for each stationary section obtained by the processing up to step 5C, and it is determined whether or not it is necessary to divide the pitch in accordance with the state. When it is determined that it is necessary, the steady section is divided into smaller steady sections.
That is, the frequency at the peak reference position is calculated by calculating the length (period length) between the peak reference positions in a certain stationary section and dividing the sampling frequency by that. Based on the value of the frequency of each waveform composing a respective stationary section, a difference (i.e., the ratio) between the frequency f ₀ of the previous waveform segment and the frequency f ₁ of the waveform segment, and quantified with a linear axis which corresponds to the note When expressed as a relative value x based on the value, ie, the “cent value of the pitch”, the following expression is obtained.
^{_{f 1 / f 0 = 2 (}} x / 12)
Expressing this in logarithm and solving x gives
_{x = log (f 1 / f} 0) / log (12 √2)
It is obtained by the following formula. In addition, ^"12 √2" is a 2 of the 12 root. As is well known, this corresponds to a formula that converts the difference or ratio (ie, pitch) of two frequencies to a cent value. However, in the expression of a general cent value, the pitch of a semitone is expressed by 100 cents. However, for x according to the above equation, the numerical value “1” corresponds to the pitch of the semitone, and the pitch of the semitone is “1”. , The value including the value after the decimal point. However, since this is only a matter of the scale of the decimal point, the relative value x may be considered to substantially correspond to a cent value, that is, relative pitch information. . In the above equation, the relative value x has a plus or minus sign depending on which of f ₁ and f ₀ is larger, but since this sign is not necessary for detecting a pitch stable section, The absolute value expression | x | from which this is removed is referred to as "note distance". FIG. 16A shows an example of a function of the temporal change of the “note distance” (hereinafter referred to as a note distance variation curve) obtained in this manner, in which the vertical axis represents the “note distance” and the horizontal axis represents the “note distance”. Time. A section where the note distance variation curve is flat corresponds to a section where the pitch is stable.

If the note distance variation curve as shown in FIG. 16A is differentiated and a portion having a large falling or rising is defined as a break, two pitch stable sections PS1 and PS2 are detected.
Although the pitch stable section may be obtained in this manner, in this embodiment, the dynamic border curve is calculated, and the pitch stable section is detected based on the calculated dynamic border curve. Here, the dynamic border curve is calculated based on the note distance variation curve. For example, the dynamic border at a certain sample point PX is an average value of the note distance variation curve from the start position to the sample point PX. And multiplying it by a predetermined constant. Note that an offset value may be added to this. In the case of FIG. 16A, the dynamic border curve becomes like the curve AC1. By comparing the dynamic border curve AC1 with the note distance variation curve NC1, a section where the note distance variation curve NC1 is smaller than the dynamic border curve AC1 is defined as a pitch stable section. At this time, when the dynamic border curve AC1 becomes smaller than the note distance variation curve NC1, the calculation of the dynamic border curve AC1 is stopped, the value is kept, and the note distance variation curve NC1 and its When the held value becomes equal, the value of the dynamic border curve AC1 up to the previous time is reset, and the calculation of the dynamic border curve AC1 is started again from the beginning in the same manner. This is shown in FIG. 16 (B). Then, pitch stable sections PS3 and PS4 as shown in FIG. 16B are obtained. In the case of the note distance variation curve NC1 as shown in FIG. 16, the pitch stable section detected by the differentiation processing and the pitch stable section detected by the dynamic border curve do not change much. However, in the case of the note distance variation curve NC2 as shown in FIG. 17, a clear difference appears.

In the case of the note distance variation curve NC2 in FIG. 17, the pitch is unstable in the latter half of the same vowel section. Therefore, when a pitch stable section is detected by the slope of the curve NC2, the pitch as shown in FIG. Many stable sections PS5, PS6, PS7, and PS8 appear. However, in the case of the note distance variation curve of FIG. 17, the human ear responds dullly (sparsely) to a change in pitch at a portion where the pitch is unstable. A large number of stable pitch sections are not sensed, and two rough pitch stable sections as shown in FIG. 16B are sensed.
On the other hand, if a pitch stable section is detected by the above-described dynamic border curve, it is possible to cause a reaction similar to that of a human ear. That is, when a dynamic border curve of the note distance variation curve NC2 in FIG. 17 is obtained, a curve AC2 as shown in FIG. 17B is obtained. Therefore, the portion of the note distance variation curve NC2 smaller than the dynamic border curve AC2 becomes the pitch stable sections PS9 and PSA, and is grasped as roughly two sections as shown in FIG. That is, in a section in which the pitch is stable (stable section PS9 in FIG. 17), when a certain pitch change occurs thereafter, the human feels that the division of the sound has changed, that is, a new sound has started. Conversely, in a section where the pitch is unstable (stable section PSA in FIG. 17), a slight change in the pitch is hardly perceived by the human ear, and is not recognized as a new sound, that is, a section. Therefore, in a section where the pitch is unstable, the change of the pitch is not recognized unless the change of the pitch is relatively large. In order to enable detection of such a pitch stable section close to the human ear, a dynamic border as described above is used as a curve, and a stable section or an unstable section is dynamically detected in a series of processes. We are dividing.
The stable pitch section detected in this way is a steady section finally detected by the steady section detection process of FIG. 5, that is, a section corresponding to one note when expressed as a musical score.

FIG. 6 is a diagram showing details of the pitch sequence determination processing in step 16 of FIG. The pitch sequence determination process is a process for determining an optimal pitch sequence for each steady section detected in step 15. Hereinafter, the pitch sequence determination processing will be described with reference to FIGS.
When voices or musical sounds are finally converted to note information, the melody greatly changes depending on the pitch to which a certain specific frequency is rounded, and in many cases, the desired detection cannot be performed. Therefore, in this embodiment, the pitch is determined mainly by the relative sound, and the pitch sequence is determined by selecting the most suitable pitch transition using the key.
An example of the pitch sequence determination processing will be described with reference to the flowchart of FIG.
Step 61: For each stationary section obtained by the stationary section detection processing of step 15 (FIG. 5), a representative frequency of the section is determined. FIG. 19A is a diagram illustrating an example of a finally obtained steady section. Here, assuming that a total of twelve sections have been detected, section numbers [0] to [12] surrounded by parentheses are assigned to each section.
What is important in determining the representative frequency of each stationary section is to identify the trend of the frequency from the periodic position of each stationary section and determine one unique frequency for the section. As a method for this, the first method uses the average frequency of the entire stationary section as the representative frequency of the section. In the second method, a period (frequency) near the middle of a stationary section is set as a representative frequency of the section. In the third method, the average frequency of the portion where the pitch is stable is set as the representative frequency of the section.
In this embodiment, the representative frequency is calculated using the note distance variation curve used in the subdivision processing based on the note distance in step 5D of FIG. 5 and the pitch stable section detected at that time. That is, the average value of the note distance variation curve in the section subdivided by the subdivision processing in step 5D of FIG. 5, that is, the pitch stable section is obtained. This average value is used as a static border. For example, in the case of the note distance variation curve NC2 as shown in FIG. 17, the static border in the pitch section PS9 is SB1, and the static border in the pitch section PSA is SB2. The sections in which the note distance variation curves NC2 in the pitch sections PS9 and PSA are smaller than the static borders SB1 and SB2 are defined as representative frequency detection sections F1 and F2, and the waveforms of the waveforms present in the representative frequency detection sections F1 and F2 are determined. Based on the pitch, the representative frequency of the steady section (pitch stable section) PS9 and PSA is determined.
For example, the number of waveform sections constituting the representative frequency detection section F1 in FIG. 18 is 12 as shown in FIG. 19B, and the cycle length of each waveform section is as illustrated. In this case, the average value of the cycle length in the representative frequency detection section F1 is 255.833. Here, since the cycle length is represented by the number of samplings, the sampling frequency is 44.1 kHz. Therefore, the representative frequency of this representative frequency detection section F1 is obtained by dividing the sampling frequency by the average value of the cycle length. Therefore, in the case of FIG. 19B, the frequency is 172.38 Hz. In this case, the value of the representative frequency treats two decimal places as valid. FIG. 19C is a diagram showing a result of calculating the representative frequency of each stationary section as shown in FIG. 19A.

Step 62: When the representative frequency of each stationary section is determined by the processing of step 61, the note distance between consecutive section numbers before and after each stationary section is determined based on the representative frequency. The determination of the note distance is obtained in the same manner as the arithmetic expression used in step 5D of FIG. FIG. 19C shows an example of the note distance calculated in this way.
Step 63: The calculated note distance is rounded to one decimal place, and the note distance is rounded to each pitch on the 12th scale. For example, in the case of FIG. 19C, each note distance is rounded off to become an integer in the right column. Since this integer indicates the difference in note number from the previous pitch, it is possible to complete the pitch sequence data by determining the initial pitch. The pitch sequence data shown in the rightmost column of FIG. 19C is data indicating a pitch transition when the initial pitch is set to 0.
That is, in the case of FIG. 19 (C), it becomes 0-2--4-5-2-3.
Step 64: Determine the pitch of the first sound. First, the simplest method assigns a 60 note number (note name C4) sound to the first sound as a default value. That is, in the case of the MIDI standard, since the limit of the note number is 0 to 127, the note having the note number 60 (note name C4) is assigned as the pitch of the first note. As a result, the pitch can be increased by 67 semitones on the high tone side (plus side) and by 60 semitones on the low tone side (minus side).
In this way, the data indicating the pitch sequence in the rightmost column in FIG. 19C is 60 (C4) -62 (D4) -64 (E4) -65 (F4) -62 (D4) -63 (D # 4).

Step 65: Modify the pitch sequence data determined in step 64. That is, the swing range of the pitch sequence data determined in step 64 is detected, and if the swing range is -60 or less on the low frequency side (minus side), the default value 60 is set according to the minimum swing width. Fix it. This correction is performed by shifting the default value upward so that the note with the minimum swing is 0 or more. For example, when the minimum swing is -64, the default value 60 is shifted upward by four notes according to the result of the calculation formula -60-(-64) = 4, and 64 is assigned as the first note. Even in the case where the sound swings +67 or more toward the higher pitch side (plus side), the default value 60 may be similarly corrected according to the maximum swing width. In addition, it is impossible to judge from the human utterance band that the swing width exceeds on both the low-pitched side (minus side) and the high-pitched tone (plus side), and such a case is excluded. In the case where such a case can occur, the range may be set in a range of 0 to 256.
In step 64, the case where the pitch of the first sound is determined as a default value (for example, 60) and the pitch sequence data is created has been described. However, the present invention is not limited to this. A frequency of a close genuine scale may be detected and applied to the scale.
For example, in the case of FIG. 19 (C), since the representative frequency of the section number [0] is 172.38 Hz, the pitch of the first sound is determined to be the note number 53 (note name F3) closest thereto.
Thus, the data indicating the pitch sequence in FIG. 19C is 53 (F3) -55 (G3) -57 (A3) -58 (A # 3) -55 (G3) -56 (G # 3). ...
It goes without saying that the pitch sequence may be assigned by various other methods.

Next, a second embodiment in which the electronic musical instrument according to the present invention operates as a sound signal analyzer and a performance information generator will be described.
The main flow when the electronic musical instrument according to the second embodiment operates as the sound signal analyzer and the performance information generator is the same as that in FIG. However, since the contents of each processing of Steps 13 to 15 in the main flow are different from those of the above-described first embodiment, the different points will be described in detail below.

FIG. 20 is a diagram showing details of the valid section detection processing in step 13 of FIG. 1, and corresponds to FIG. The valid section detection processing is processing for detecting a section in which a musical sound exists, that is, a valid section, based on the digital sample signal obtained as a result of the voice sampling processing in step 12. Hereinafter, the details of the valid section detection processing will be described with reference to FIG.
Step 201: A process for dividing the digital sample signal obtained in step 12 into a predetermined number of samples is performed. FIG. 23A is a diagram illustrating an example of a waveform value of an audio signal sampled at a sampling frequency of 44.1 kHz, that is, a digital sample signal. FIG. 23A shows waveform values for about 4408 points. FIG. 23D shows a waveform value of about 8816 points that is twice as large. In step 201, the digital sample signal is divided by a predetermined number of samples (for example, the number of samples corresponding to the maximum period when the lowest frequency of audio is 80 Hz). Therefore, when the sampling period is 44.1 kHz, the predetermined number of samples is “551 = 44100/80”. FIG. 23B corresponds to the waveform values of FIG. 23A, and is a diagram showing the state of each of the waveform sections S1 to S8 when the waveform values are divided every 551 samples.

Step 202: For each section divided by step 201, the maximum value of the digital sample signal waveform existing in the section is extracted. In FIG. 23 (C), the digital sample signal waveform of FIG. 23 (A) is indicated by a dotted line, and the maximum value of each waveform in each of the sections S1 to S8 is indicated by a black point.
Step 203: Interpolate (for example, linearly interpolate) the maximum value of each section obtained in Step 202 to create an auxiliary waveform. FIG. 23D shows an auxiliary waveform corresponding to a section approximately twice as long as FIGS. 23A to 23C. The auxiliary waveform obtained by linearly interpolating the maximum value of each section is shown in FIG. Is shown. Note that the digital sample signal waveform in FIG. 23A is shown by a dotted line.
In the next steps 204 to 206, an effective section is extracted based on the auxiliary waveform thus obtained.

Step 204: The auxiliary waveform as shown in FIG. 23D obtained in step 203 is classified into a valid section and an invalid section based on a predetermined threshold Th. In this processing, the threshold Th is set to a value about one third of the maximum waveform value. It goes without saying that a value other than this may be used as the threshold value Th. For example, the average value of the solid waveform in FIG. 23D may be set to the threshold value Th, or 80% of the average value may be set to the threshold value Th.
Therefore, the intersection of the threshold value Th and the auxiliary waveform is the boundary between the valid section and the invalid section, the section larger than the threshold value Th is the valid section, and the section smaller than the threshold Th is the invalid section.

Step 205: When the minimum length required for human recognition of the pitch is 0.05 msec, the invalid section smaller than the minimum length is changed from the invalid sections determined in step 202 to the valid section. For example, when the sampling period is 44.1 kHz, the invalid section of 2205 or less in sampling number corresponds to the invalid period of 0.05 msec or less, which is the minimum length necessary for human being to recognize the pitch. Change invalid invalid section to valid section. The invalid section shown in FIG. 23D corresponds to three invalid sections (about 1653 sampling numbers) in the waveform section, and thus corresponds to an invalid section smaller than the minimum length. Is done.
Step 206: A process of changing a short valid section of 0.05 msec or less from the pattern of the valid section and the invalid section obtained as a result of the processing of the step 205 to the invalid section is performed. This process is performed in the same manner as in step 205 described above.

FIG. 21 is a diagram showing details of the stable section detection processing in step 14 of FIG. 1, and corresponds to FIG. Hereinafter, a stable section detection process for detecting an area with a stable level is performed on the digital sampling signal in the effective section obtained by the effective section detection processing of FIG. The operation of the stable section detection processing will be described with reference to FIGS.
Step 211: The side where the peak of the waveform is strong is detected based on the digital sampling signal in the effective section detected by the effective section detection processing of FIG. That is, as shown in FIG. 24A, the absolute value of the peak value max of the plus (+) side waveform and the peak value min of the minus (−) side waveform of the digital sampling signal in the effective section are calculated. Is determined depending on whether the absolute value of is large. Note that a side having a strong peak may be determined by other methods. For example, the total of the absolute values of the top three to five peak values may be compared and determined.
Step 212: On the side where the peak is detected in step 211, the envelope is taken forward (in the time lapse direction), and the peak portion is detected. That is, as shown in FIG. 24B, an envelope is taken ahead of the waveform on the plus side, and the peak portion is detected. As a result, in the case of FIG. 24B, four points P1 to P4 are detected as peak portions.
Step 213: This time, the envelope is taken in the direction opposite to that of step 212 (the direction opposite to the elapsed time), and its peak is detected. Then, as shown in FIG. 24C, peak portions P1 to P4 are detected at the same position, but depending on the waveform, other peak portions PP are detected. This means that if the envelope is detected in only one direction in a waveform whose level gradually rises, the overtone peak is mistaken as a pitch peak, and a portion that is not actually a peak (such as a peak PP). ) May be erroneously detected as a peak. Therefore, as in steps 212 and 213, by detecting the peaks in different directions and detecting the peaks, it is possible to improve the detection accuracy of the peaks.

Step 214: A new waveform is generated by linearly interpolating the peaks detected by the processing of steps 212 and 213. FIG. 24D shows a new peak value interpolation curve generated by linearly interpolating the peak portions P1 to P4 detected in steps 212 and 213. In FIG. 24D, the waveform of the digital sample signal in FIG. 24A is indicated by a dotted line.
Step 215: Calculate the total slope between the peaks based on the peak value interpolation curve generated by the above processing. In this processing, as shown in FIG. 24D, the calculation width for calculating the inclination is set to, for example, 200 points, the shift amount of the calculation width is set to, for example, 100 points, and points corresponding to this shift amount are sequentially shifted. While calculating its inclination. Assuming that the first sample point a1 of the valid section is "100", the slope b1 between the sample points "100" and "300" is obtained by the formula: (a3 -a1) / 200. Next, a slope b2 between sample points "200" and "400" obtained by shifting each point by 100 points is obtained.

FIG. 25 shows an example of the inclination calculated in this manner. As is clear from the drawing, the slope b1 between the sample points "100" and "300" is 0.03, the slope b2 between the sample points "200" and "400" is 0.15, and the sample point "300"-" The slope b3 between "500" and "500" is 0.25, the slope b4 between sample points "400" and "600" is 0.50, the slope b5 between sample points "500" and "700" is 0.90, and the sample point " The slope b6 between "600" and "800" is 1.80, the slope b7 between sample points "700" and "900" is 1.90, and the slope b8 between sample points "800" and "1000" is 2.00. The slope b9 between sample points “900” and “1100” is 1.70, the slope b10 between “1000” and “1200” is 1.20, and the slope b10 between “1100” and “1300”. Oblique b11 is 0.70.
These slopes b1 to b11 are stored as the slopes at the former sample points a1 to a11. That is, assuming that the slope b1 of the sample point a1 is 0.03 and the slope b2 of the sample point a2 is 0.15, the slope is stored for each sample point.

  Next, a total inclination is calculated based on the values of the inclinations b1 to b11 thus obtained. The total slope is obtained by summing the last five slopes with respect to the sample point, and indicates the degree of the slope near the sample point. For example, the total slope c1 of the sample point a1 is calculated by summing the slope b1 of the sample point a1 and the slopes b2 to b5 four places behind it. That is, it is calculated by c1 = b1 + b2 + b3 + b4 + b5. In the case of FIG. 25, the total slope c1 of the sample point a1 is 1.83, the total slope of the sample point a2 is 3.60, the total slope c3 of the sample point a3 is 5.35, and the total slope of the sample point a4 is 7 .10, the total slope of the sample point a5 is 8.30, the total slope c6 of the sample point a6 is 8.60, and the total slope of the sample point a7 is 7.50.

In this way, the total inclination in all the sections is calculated, and the process of the next step 216 is performed. Here, a case has been described in which the sum of the five points is used as the total slope at the head, but the present invention is not limited to this, and a total slope in the middle of five points may be used. That is, the total slope c1 may be set to a value between the sample points a1 to a5, that is, the value of the sample point a3. In addition, if the position at five points is clear, the total inclination may be set to any value. Needless to say, the number of points is not limited to 5 and may be more or less.
By using the total inclination as described above, an appropriate inclined portion can be found without being deceived by a temporary inclination, so that an appropriate stable portion can be found.

Step 216: This time, a stable section is extracted based on the total slope calculated in step 215. That is, in a total slope curve formed by connecting the total slopes at each sample point by linear interpolation or other interpolation, a portion having a predetermined value (for example, a total slope value of 5) or less is defined as a stable section, and other portions are defined as stable sections. Is an unstable section.
Step 217: The maximum value of the waveform, that is, the maximum value of the peak value interpolation curve is detected for each section considered as a stable section by the processing of step 216, and when the maximum value is equal to or less than the predetermined value, the stable section is deleted. , Change to an unstable section.
Step 218: Based on the existence of the stable section extracted in this way, the human first notices that there is a starting point of a sound which is a trigger of a note near the starting point of the stable section. Therefore, in order to determine the vicinity of the starting point of the note, the note starting point of the portion that has been recognized as a stable section in steps 216 and 217 is detected, and the stable section is extended accordingly.

  FIG. 26 shows the concept of the processing from step 216 to step 218. When the total slope curve within the effective section is as shown in FIG. 26, three stable sections d1 to d3 are detected by dividing the curve by a predetermined value. Note that the stable section d2 is deleted because the maximum value of the peak value interpolation curve is equal to or less than the predetermined value as a result of the process of step 217. Therefore, in the case of the effective section in FIG. 26, two stable sections d1 and d3 exist, and an unstable section including the deleted stable section d2 exists between the two stable sections d1 and d3. Will be. It is necessary to connect the unstable sections to the stable sections d1 and d3 and perform the extension processing of the respective stable sections d1 and d3.

For the stable section d1, the start point of the effective section is necessarily the start point of the note in the stable section d1, and for the stable section d3, the end point of the valid section is necessarily the end point of the note in the stable section d3. It becomes. The note start point of the stable section d3 and the final point of the stable section d1 are obtained as follows. That is, from the unstable sections detected in step 216, an unstable section that is closer to the stable section for which the note start point is to be detected is determined and corresponds to the peak value of the total slope curve in the unstable section. The sample point to be used is set as the note start point of the stable section. Therefore, when the stable section d2 is deleted and two unstable sections exist as shown in FIG. 26, the peak of the total slope is determined in the unstable section closer to the stable section d3 for which the note start point is to be detected. The sample point f2 corresponding to the value is the note start point of the stable section d3. Accordingly, since the note end point of the stable section d1 is the sample point f2, the stable section d1 finally becomes the extended stable section e1 and the stable section d2 becomes the extended stable section e3.
In the case of FIG. 26, if the stable section d2 is not deleted by the process of step 217, the sample point f1 becomes the note end point of the stable section d1 and the note start point of the stable section d2, and the sample point f2 becomes This is the note end point of the stable section d2 and the note start point of the stable section d3.

  FIG. 22 is a diagram showing details of the steady-state section detection processing in step 15 of FIG. 1 and corresponds to FIG. How the steady section is detected from the stable sections determined by the stable section detection processing of FIG. 21 will be described in detail. Steps 221 to 229 in the steady-state section processing in FIG. 22 are almost the same as steps 51 to 59 shown in FIG. 5, so that the portion will be briefly described, and the other portions will be described in detail.

Step 221: Pass a first-order bandpass filter (first-order BPF) to delete a predetermined harmonic.
Step 222: A peak reference position detection process, which is a reference for one cycle, is performed on the musical tone waveform signal obtained by the first-order BPF process of step 221 using a peak position detection method.
Step 223: Based on the peak reference position detected in Step 222, a basic section starting from a certain peak reference position and a section up to the next peak reference position immediately after the basic section (hereinafter referred to as a moving section). The comparison of whether or not the waveforms of the two sections between are the same is performed by an error rate calculation method as shown in FIG.
Step 224: Using the result of the waveform comparison process of step 223, the sections in which the error rate is smaller than a predetermined value (for example, 10) are connected to each other to be a pseudo matching section, and extracted from each matching section. The maximum value and the minimum value of the pitch are detected, and the cutoff frequency band is determined based on the maximum value and the minimum value.
Step 225: Using the new cut-off frequency band determined in Step 224, pass through a second-order band-pass filter (second-order BPF) to remove unnecessary harmonics.
Step 226: The same processing as the peak reference position detection processing of step 222 is performed.
Step 227: The same processing as the waveform comparison processing of step 223 is performed.
By the series of processes from step 225 to step 227, low frequencies and harmonics that cause an error are cut, and a more accurate peak reference position detection process and a waveform comparison process can be performed. A matching section is obtained.
Step 228: The pitch data at each peak reference position obtained by the processing up to step 227 is interpolated, and linear interpolation is performed so that one pitch data is obtained for each sample point.

Step 229: Perform a time-varying bandpass filter (BPF) process using the pitch data for each sample point obtained by the process of step 228.
Step 22A: For the musical sound waveform that has undergone the time-varying band-pass filter processing in step 229, the side where the peak is strong is determined, and the musical sound waveform is formed by the period section determined based on the frequency change obtained in step 228. , And a position that becomes the maximum in each section is detected, and that position is defined as a peak reference position. That is, when the musical tone waveform obtained by the time-varying bandpass filter processing in step 229 is as shown in FIG. 27, each musical tone waveform is divided by period sections PR1 to PR5 determined based on the frequency change. The maximum values P1 to P6 in the divided sections PR1 to PR5 are the peak reference positions. When the peak reference position is detected by the peak reference position detection processing as in step 222 (step 52) or step 226 (step 56), in the case of the waveform shown in FIG. There is a problem that the positions P4 and P5 appear. However, when the musical sound waveform is divided by a period section as in step 22A and the peak reference position is detected therein, such an apparently incorrect peak is detected. No reference position is detected, and the accuracy of peak reference position detection is improved.

  Step 22B: Perform voiced section detection processing of the waveform based on the peak reference position detected by the peak reference position detection processing of step 22A. That is, in this voiced section detection process, as in step 223, based on the peak reference position detected in step 22A, a basic section starting from a certain peak reference position and a next peak reference position immediately after the basic section are determined. The comparison of whether or not the waveforms of the two sections between the section (hereinafter referred to as a moving section) are the same is performed by an error rate calculation method as shown in FIG. For this reason, in the voiced section detection process, if it is determined that the basic section and the moving section do not match, the section is not immediately set as a section of the voiced section. And a delimiter. As a result, it is possible to detect a "vowel-continuous portion" such as "a-a-a-" or "a-a-a-" as a voiced section.

For example, in the case of FIG. 28, assuming that the basic section P12 is from the peak reference position P1 to the peak reference position P2, the moving section P23 is from the peak reference position P2 to the peak reference position P3. In this case, it is assumed that it is determined that the basic section P12 and the moving section 23 match. It is assumed that the sections P23 and P34 are also determined to be identical. If it is determined that the next section P34 does not match the section P45, a waveform comparison process is performed between the section P34 and the section P56 next to the section P45. As a result, when it is determined that the section P34 and the section P56 match, even if the section P45 and the section 56 do not match, the section P34, the section P45, and the section P56 are assumed to match, and the next section P56 and the section P67 match. (Not shown). At this time, the sections P34 and P45, the sections P34 and P56, the sections P34 and P67 (not shown), the sections P34 and P78 (not shown), and the sections P34 and P89 (not shown) respectively do not match. Is determined, that is, when a mismatch occurs continuously for a predetermined number of times (for example, five times) or more, the section P34 is set as a section of a voiced section, and the same determination is performed for the next section P45 and the section P56. When the sections P45 and P56 do not match, the determination is made for the sections P45 and P67 (not shown). When the section 45 and the section P56 do not match, the process is not performed to determine whether or not the mismatch continues, but the determination is made for the next section P56 and the section P67. A matching process may be performed.
When the voiced section is determined in this way, the voiced section having a predetermined length or less (short voiced section) is deleted.

  By the above processing, stable sections are classified as voiced sections V1 to V3 separated by unstable sections as shown in FIG. Note that the voiced sections V1 to V3 correspond to a stable portion having a low value of the adjacent comparison error curve, and a portion having a high value of the adjacent comparison error curve corresponds to the unstable section. Therefore, based on this adjacent comparison error curve, expansion processing of the stable sections V1 to V3 is performed. This expansion process unconditionally extends the effective section that touches the start point and end point of the stable section to the start point and end point of the stable section, and compares the unstable section between two voiced sections with the adjacent comparison. The effective section is extended using the maximum value of the error as a breakpoint. Therefore, in the case of the adjacent comparison error curve as shown in FIG. 29, each voiced section V1 to V3 becomes an expanded voiced section V1E to V3E by the expansion processing. Note that FIG. 29 shows only one case where the slope of the adjacent comparison error in the extended voiced section is 0 (the bottom portion), but in actuality, the slope of the adjacent comparison error is 0. Needless to say, there may be a plurality of portions (base portions).

Step 22C: With respect to each extended voiced section obtained by the processing from step 221 to step 22B, a portion (base portion) where an error between adjacent sections, that is, a slope of an adjacent comparison error becomes 0, is detected, and the detected portion is used as a vowel reference. A process is performed to detect a section corresponding to the position of the vowel as a timbre section.
In the process of detecting the tone color section, a waveform section corresponding to the base portion is fixed as a basic section, and a plurality of waveform sections existing before and after the base section are sequentially set as moving sections, and a waveform comparison process is sequentially performed to obtain a comparison error. The comparison error thus obtained is called a reference comparison error.
That is, as shown in FIG. 30A, a waveform section m0 corresponding to the bottom portion of the adjacent comparison error is set as a basic section, and a plurality of moving sections m1, m-1, m2, m on both sides of the basic section. Perform waveform comparison processing between −2, m3, m−3, m4, m−4,. The basic section is a section corresponding to the minimum value of the adjacent comparison error, that is, a waveform section determined as having high coincidence as a result of the waveform comparison processing. The comparison error thus obtained becomes a reference comparison error curve as shown in FIG. This reference comparison error curve shows the same tendency as the adjacent comparison error curve in the vicinity of the waveform section m0 due to the fact that the waveform comparison processing is performed with the waveform section m0 as the reference, but the error rate is relatively high in the part far away. Increases and converges to the maximum error rate.
The waveform section in which the value (error rate) of the reference comparison error curve is equal to or less than a predetermined value is the timbre section TS1.
In the case where the reference comparison error curve is obtained, as in the voiced section detection process in step 22B, when the value of the reference comparison error curve becomes larger than a predetermined value, the reference comparison error curve is not immediately used as a delimiter of the timbre section. When a value larger than a predetermined value occurs continuously for a predetermined number of times or more, it is defined as a timbre section break.

  When the timbre section is determined in this way, if the length of the undecided section other than the timbre section in the extended voiced section is equal to or longer than a predetermined length, the same processing is performed on the extended voice section other than the determined timbre section. . That is, in the case of FIG. 30, when the timbre section TS1 as shown in FIG. 30B is determined, the extended voiced section other than the timbre section TS1, that is, the undecided section length is equal to or longer than a predetermined length. Similarly, a waveform section n0 corresponding to the base portion of the adjacent comparison error as shown in FIG. 30C is defined as a basic section, and the basic section and a plurality of moving sections n1, n−1, The waveform comparison processing is performed among n2, n-2, n3, n-3, n4, n-4,. The comparison error obtained in this manner becomes a reference comparison error curve as shown in FIG. The waveform section in which the value (error rate) of the reference comparison error curve is equal to or less than a predetermined value is the timbre section TS2. Therefore, in the case of the extended voiced section in FIG. 30, two tone color sections TS1 and TS2 are detected.

Step 22D: The timbre section obtained by the processing of step 22C is extended in the same manner as the stationary section extension processing of step 5C. That is, as a result of performing the processing from step 221 to step 22C, if the detected timbre section ST1 and the detected speech section ST2 are separated by one waveform section, the waveform section is directly changed to the timbre section ST1. And ST2, but if adjacent timbre sections are separated by a plurality of waveform sections, these waveform sections must be connected to the preceding and following timbre sections to extend the timbre section. No. The process of expanding the timbre section is performed by the same process as in FIG.
In this case as well, since the comparison process is performed on a waveform close to the sine waveform after the BPF process, even the features of each vowel are filtered, and the significance of extracting the same vowel section, that is, the same timbre, is diminished. There is a risk that it will. Therefore, a waveform for peak position detection and a waveform for waveform comparison processing may be separately prepared, and the peak position detection and waveform comparison processing may be respectively performed based on the waveforms. That is, the waveform after the time-varying BPF processing is used as it is as the waveform for peak position detection, and the BPF processing for leaving a frequency band waveform several times as long as the frequency component used for the time-varying BPF processing is used for the waveform comparison processing. The waveform obtained by performing the above is used.
It goes without saying that bandpass filter processing may be performed in which the basic frequency is set to the lowest frequency and the integral frequency of the basic frequency is set to the highest frequency, and this may be used as a target waveform for the waveform comparison processing.

  Subdivision processing is performed on the extended timbre section in this way in consideration of a change in pitch and stability, and a final interval section is determined. In the timbre section detection processing up to step 22C, since the waveform is expanded and compared, even if the pitch change of the voice waveform is caused by a continuous vowel such as "Oh", it is regarded as one and the same sound. It is a mechanism. Therefore, in the case of the musical sound waveform of the musical instrument sound, a situation may occur in which the pitch change of the continuous musical instrument sound cannot be found. Therefore, in this embodiment, the state of the pitch change is examined for each timbre section obtained by the processing up to step 22C, and it is determined whether or not further division is necessary according to the state. If it is determined that it is necessary, the timbre section is divided into smaller pitch sections. The process of dividing the timbre section into pitch sections is performed using a note distance variation curve as shown in FIG.

Step 22E: The interval detected by the process of step 22D may include a note that is too short to exist as a note. Therefore, in this step, one measure is equally divided into a grid in units of a predetermined note length (for example, eighth note length), and the pitch interval is applied to this grid to determine a note value. . Although the interval is assigned to the grid closest to the beginning of each interval, if two or more intervals are closest to one grid, the pitches in those intervals are set. The longer ones to the grid.
For example, FIG. 31 is a diagram showing an example of a musical interval corresponding to one measure divided by an eighth note length. In the figure, it is assumed that the pitch intervals finally determined in step 22D are like PT1 to PT5. In this case, the interval PT1 applies to the grid G2, the interval PT2 applies to the grid G4, and the interval PT3 applies to the grid G5. However, with respect to the grid G6, the interval PT4 and the interval PT5 are the intervals closest to the grid G6. Therefore, in this case, the longer pitch of the interval PT4 and the interval PT5, that is, the interval PT5 is applied to the Grigg G6.
Note that, since the interval PT3 is applied to the grid G5, the pitch of the interval PT2 is from the grid G4 to the grid G5. At this time, if the interval PT3 does not exist, the interval of the interval PT3 is set. The final position of the pitch of PT2 may be used as it is, or the interval of the interval PT2 may be applied to the nearest grid. In this case, note-off (rest) may be applied to a portion where no interval exists.
When there is no interval PT3, the last position of the pitch of the interval PT2 may be set to the grid G6, which is the start position of the next interval PT5. In this case, note-off (rest) does not exist.
After the pitch value is determined by the stationary section detection process of FIG. 22 in this manner, the optimum pitch sequence is assigned to each pitch value by the pitch sequence determination process of step 16 of FIG. This pitch sequence determination processing is the same as that of the first embodiment, and a description thereof will be omitted.

  According to the sound signal analyzer according to the embodiment, even when the pitch or level of the input sound from the microphone or the like is slightly fluctuated, it corresponds to a stationary part of a musical sound other than the fluctuated part, that is, one note. It is possible to provide a sound signal analyzer capable of analyzing a part to be reproduced.

FIG. 3 is a diagram showing a main flow when the electronic musical instrument of FIG. 2 operates as a performance information generating device. 1 is a hardware block diagram showing a configuration of an electronic musical instrument incorporating a musical sound information analyzer and a performance information generator according to the present invention. FIG. 3 is a diagram illustrating details of an effective section detection process in step 13 of FIG. 1. FIG. 3 is a diagram illustrating details of a stable section detection process in step 14 of FIG. 1. FIG. 3 is a diagram illustrating details of a steady section detection process in step 15 of FIG. 1. FIG. 2 is a diagram illustrating details of a pitch sequence determination process in step 16 of FIG. 1. FIG. 4 is a diagram illustrating an example of a waveform value of an audio signal sampled at a sampling frequency of 44.1 kHz, that is, a digital sample signal. FIG. 4 is a diagram illustrating a concept of an operation example of an effective section detection process in FIG. 3. FIG. 5 is a diagram illustrating a concept of an operation example of the stable section detection processing in FIG. 4. FIG. 6 is a diagram illustrating the concept of an operation example by the first and second order BPF processes and the waveform comparison process of FIG. FIG. 6 is a diagram illustrating a waveform example in a case where a strong peak of a tone waveform in a stable section appears on the minus side and a weak peak appears on a plus side after the primary BPF processing in step 51 of FIG. 5. FIG. 6 is a diagram showing a specific example of how a calculation method of an error rate performed in the waveform comparison processing of FIG. 5 is performed using two comparison waves. FIG. 12 is a diagram showing specific numerical values on how the error rate is calculated from the two comparison waves in FIG. 11 by the waveform comparison processing in FIG. 5. FIG. 6 is a diagram showing a specific example of how a peak reference position is extracted by the peak reference position detection processing of FIG. 5 when the time constant is set to be small. FIG. 6 is a diagram illustrating an operation example of a steady section extension process in step 5C in FIG. 5. FIG. 6 is a diagram illustrating an operation example of a subdivision process based on a note distance in step 5D of FIG. 5. FIG. 16 is a diagram illustrating another operation example of the subdivision processing based on the note distance in step 5D of FIG. 5. FIG. 7 is a diagram illustrating an example of an operation of determining a representative frequency from a portion of a steady section when a representative frequency determination process of each steady section in step 61 of FIG. 6 is performed. FIG. 7 is a diagram illustrating an operation example of how a representative frequency is detected from each stationary section in step 61 of FIG. 6. FIG. 13 is a diagram illustrating details of an effective section detection process according to another embodiment in step 13 of FIG. 1. FIG. 9 is a diagram showing details of a stable section detection process according to another embodiment in step 14 of FIG. 1. FIG. 11 is a diagram showing details of a stationary section detection process according to another embodiment of step 15 in FIG. 1. FIG. 21 is a diagram illustrating a concept of an operation example of an effective section detection process in FIG. 20. FIG. 22 is a diagram illustrating the concept of an operation example of processing from step 211 to step 215 in FIG. 21. FIG. 22 is a diagram illustrating an example of calculating a total inclination in step 215 of FIG. 21. FIG. 22 is a diagram illustrating the concept of an operation example of the processing in steps 216 and 218 in FIG. 21. FIG. 23 is a diagram illustrating a concept of an operation example of a peak reference position detection process in step 22A of FIG. 22. FIG. 23 is a diagram illustrating the first half of the concept of an operation example in the voiced section detection process in step 22B of FIG. 22. FIG. 23 is a diagram illustrating the latter half of the concept of the operation example in the voiced section detection process in step 22B of FIG. 22. FIG. 23 is a diagram showing a concept of an operation example in the tone color section detection processing in step 22C of FIG. 22. FIG. 23 is a diagram illustrating a concept of an operation example in a tone value determination process in step 22E of FIG. 22.

Explanation of reference numerals

DESCRIPTION OF SYMBOLS 1 ... CPU, 2 ... program memory, 3 ... working memory, 4 ... performance data memory, 5 ... key press detection circuit, 6 ... microphone interface, 7 ... switch detection circuit, 8 ... display circuit, 9 ... sound source circuit, 10 ... Keyboard, 1A microphone, 1B numeric keypad and various switches, 1C display, 1D sound system, 1E data and address bus

Claims (2)

Input means for inputting an arbitrary sound signal consisting of a chronological sequence of one or more notes,
Section detection means for detecting a section estimated to correspond to each note from the input sound signal,
According to the time series, the detected sections are respectively arranged on grids divided at time intervals corresponding to predetermined note lengths, and are closest to a predetermined one of the start or end ends of each section. Means for allocating one grid position to each section, and selecting one section having the longest time length as a valid note when a plurality of sections are allocated to the same grid position. Signal analyzer. A computer-readable recording medium having, as its storage content, a group of instructions for a program for analyzing a sound signal executed by the computer, and the program for analyzing the sound signal includes:
Inputting an arbitrary sound signal consisting of a time-series sequence of one or more notes;
Detecting a section estimated to correspond to each note from the input sound signal;
According to the time series, the detected sections are respectively arranged on grids divided at time intervals corresponding to predetermined note lengths, and are closest to a predetermined one of the start or end ends of each section. Allocating one grid position to each section, and selecting one section having the longest time length as a valid note when a plurality of sections are allocated to the same grid position as a result. A recording medium characterized by the above-mentioned.

Images