JP2604408B2 - Automatic music transcription method and device - Google Patents

Description

Description: BACKGROUND OF THE INVENTION The present invention relates to an automatic music transcription method and apparatus for creating musical score data from acoustic signals such as singing voices, humming voices, and musical instrument sounds, and more particularly to an audio transcription method for a predetermined section. The present invention relates to a process for correcting an identified pitch, which corrects a pitch identified as a pitch as necessary.

[Prior Art] In an automatic transcription system for converting an acoustic signal such as a singing voice, a humming voice, or a musical instrument sound into musical score data, a sound length, a pitch, a key, a time signature, and the like, which are basic information as a musical score from an acoustic signal. Detecting the tempo.

By the way, an acoustic signal is only a signal that continuously includes a repetitive waveform of a basic waveform, and the above-described information cannot be obtained immediately.

Therefore, in the conventional automatic transcription method, first, repetition information (hereinafter, referred to as pitch information) of a basic waveform representing a pitch of an acoustic signal and power information are extracted for each analysis cycle, and thereafter, the extracted pitch information is extracted. And / or dividing the audio signal into sections (segments) that can be regarded as the same pitch based on the power information (this processing is called segmentation), and then, based on the pitch information of the segment, the pitch along the absolute pitch axis as the pitch of the audio signal of each segment. Each information is obtained in the order of determining the tone of the acoustic signal based on the distribution of the pitch information around the pitch axis, and further determining the beat and tempo of the acoustic signal based on the segment.

[Problems to be Solved by the Invention] By the way, even if an attempt is made to identify a certain segment of an acoustic signal as a pitch on an absolute pitch axis, the pitch of an acoustic signal, particularly an acoustic signal generated by a person, is not stable and the same. Even when a pitch (one tone) is intended, the pitch fluctuates greatly. For this reason, the pitch identification processing is very difficult.

By the way, as a conventional pitch identification method, a method such as a simple operation to realize real-time processing, for example,
A method of identifying a pitch on the absolute axis closest to the average value of the pitch information in the segment and a pitch on the absolute axis closest to the median of the pitch information of the segment has been considered. According to such a method, even if the sound signal has fluctuation, the pitch difference between adjacent sounds on the musical scale is good, for example, in the case of a full scale like C and D in the C major scale. Although identification is possible, if the pitch difference between adjacent sounds is, for example, a semitone such as M and F in a C major scale, the identification may not be accurate due to fluctuations in the acoustic signal. For example,
Despite being intended for Mi in C major, it was sometimes identified as a fa.

The present invention has been made in consideration of the above points, and the pitch difference between adjacent pitches on a musical scale is either a whole tone or a semitone, even if a singer or the like intends the pitch to be on the absolute axis of the pitch. An object of the present invention is to provide an automatic music transcription method and apparatus that can perform identification accurately and can further improve the accuracy of final score data.

[Means for Solving the Problems] In order to solve the problems, in the first aspect of the present invention, pitch information representing a pitch and a power information of an acoustic signal, which is a repetition period of an input acoustic signal waveform, is extracted. Processing, segmentation processing for dividing an audio signal into sections that can be regarded as having the same pitch based on pitch information and / or power information, and identifying a pitch on the absolute pitch axis as a pitch for this section based on pitch information. A pitch identification process and at least a key determination process of determining a key of an audio signal based on a distribution state of pitch information, and in an automatic transcription method of converting an audio signal into musical score data, a pitch identified as a pitch of a section is determined. A process of extracting intervals of intervals in which the pitch difference adjacent to each other on the scale of the determined key is a semitone;
A process of summarizing pitch information between the identified pitch of the extracted section and the pitch of a semitone on the scale of the determined key in the pitch information of the extracted section, and according to the tabulated distribution status, The process of re-identifying the interval of the section is provided as a post-process of the key determination process.

Further, in the second aspect of the present invention, a pitch / power extracting means for extracting pitch information representing a pitch and a power information of the audio signal, which is a repetition period of the input audio signal waveform, Or segmentation means for dividing an audio signal into sections that can be regarded as having the same pitch based on power information, pitch identification means for identifying a pitch on the absolute pitch axis as a pitch of this section based on pitch information, and pitch information. An automatic music transcription device that partially converts a sound signal into musical score data, including a key determination unit that determines a key of the sound signal based on a distribution state of the sound signal.
A correction candidate section extracting means for extracting a section of a pitch in which a pitch difference adjacent to a pitch of a key whose pitch is determined as a pitch of a section is a semitone, and pitch information of the section in the pitch information of the extracted section. Pitch summation means for summing up pitch information between the identified pitch and a pitch that differs by a semitone on the scale of the determined key, and an identification pitch correction for re-identifying the interval in the section according to the tabulated distribution status Means are provided after the key determination means.

[Operation] For example, among the seven notes (doremi faso rashi) on the musical scale in C major, do and shi and mi and fa differ from the other pairs of adjacent pitches in that the pitch difference is a semitone. for that reason,
It is possible that an incorrect identification was made during pitch identification. Therefore,
In both the first and second aspects of the present invention, when the interval identified as the interval of the section is the above-described four tones, which are semitones in the key of the acoustic signal, the intervals are reviewed. I did it.

That is, in the first aspect of the present invention, after extracting the intervals of these four tones in the key in which the identified pitch is determined, the pitch information of the interval between the semitones is totaled, and the totaled distribution state is obtained. Accordingly, the interval of the section is re-identified.

According to the second aspect of the present invention, after the above-mentioned four-tone section in the key whose identified interval is determined is extracted as a candidate for the section to be corrected by the correction candidate section extracting means, the correction candidate section extraction means extracts The pitch information is summed up by the pitch summing means, and the identified pitch correcting means re-identifies the pitch in the section in accordance with the summed distribution status.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

Automatic transcription system First, an automatic transcription system to which the present invention is applied will be described.

In FIG. 3, a central processing unit (CPU) 1 controls the whole of the apparatus, and performs a musical notation processing shown in FIG. 4 stored in a main storage device 3 connected via a bus 2. Execute the program. In addition to the CPU 1 and the main storage device 3, a keyboard 4 as an input device, a display device 5 as an output device, an auxiliary storage device 6 used as a working memory, and an analog / digital converter 7 are connected to the bus 2. I have.

An audio signal input device 8 including, for example, a microphone is connected to the analog / digital converter 7. The acoustic signal input device 8 captures an acoustic signal such as singing or humming uttered by a user or a musical sound uttered from a musical instrument and converts the signal into an electric signal. The electric signal is converted into an analog / digital converter. 7 is output.

When a process is instructed by the keyboard input device 4, the CPU 1 starts the transcription process, executes a program stored in the main storage device 3, and converts the sound converted into a digital signal by the analog / digital converter 7. The signals are temporarily stored in the auxiliary storage device 6, and thereafter, these sound signals are converted into musical score data by executing the above-described program and output to the display device 5 as necessary.

Next, the transcription process performed by the CPU 1 after capturing the audio signal will be described in detail with reference to the flowchart shown in FIG.

First, the CPU 1 performs an autocorrelation analysis of the acoustic signal to extract pitch information of the acoustic signal at each analysis cycle, and also performs a sum-of-squares process to extract power information at each analysis cycle, and then performs noise removal and smoothing processing. And other post-processing (steps SP1, SP
2). Thereafter, for the pitch information, the CPU 1 calculates a shift amount of the pitch axis of the acoustic signal with respect to the absolute pitch axis according to the distribution state, and executes a tuning process of shifting the obtained pitch information according to the shift amount. (Step SP3). That is, the pitch information is corrected so that the difference between the pitch axis and the absolute pitch axis of the singer or musical instrument that has generated the acoustic signal is reduced.

Next, the CPU 1 obtains a continuous period of pitch information in which the obtained pitch information is considered to indicate the same pitch,
A segmentation is performed to divide the acoustic signal into segments for each sound, and a segmentation is performed based on the obtained change in the power information (step SP
4, SP5). Based on these two pieces of segment information obtained, the CPU 1 calculates a reference length corresponding to a time length of a quarter note, an eighth note, etc., and executes the segmentation again based on this reference length (step SP6). .

The CPU 1 identifies the pitch of the segment as a pitch on the absolute pitch axis that can determine that the pitch information is the closest based on the pitch information of the segment thus segmented, and further identifies the pitch of the identified continuous segment. Segmentation is again performed based on whether or not are the same (steps SP7 and SP8).

Thereafter, the CPU 1 obtains a product sum of a frequency of occurrence of a pitch for each segment obtained by summing the pitch information after tuning and a predetermined weighting coefficient determined according to the key, and based on the maximum information of the product sum. Then, for example, the key of the music of the input audio signal is determined such as C major or A minor, and then, if the identified pitch is a predetermined scale, the pitch is reviewed for pitch information to confirm and correct the pitch. (Steps SP9 and SP10). Next, the CPU 1 executes a review of the segmentation based on whether or not there is the same continuous segment from the finally determined pitch, and whether or not there is a power change between the continuous segments, Final segmentation is performed (step SP11).

Once the pitch and segment are determined in this way,
The CPU 1 extracts measures from the viewpoint that the music starts from the first beat, the last sound of the phrase does not extend to the next measure, and there is a break in each measure, and determines the time signature from the measure information and the segmentation information. The tempo is determined from the determined time signature information and the length of the bar (step SP1).
2, SP13).

Then, the CPU 1 organizes the information on the determined pitch, pitch, key, beat, and tempo to finally create the musical score data (step SP14).

Correction processing of the identified pitch Next, regarding the correction processing of the pitch identified by the above-described pitch identification processing of step SP7 (step SP10),
This will be described in detail with reference to the flowchart of FIG.

As described above, before executing the pitch correction, the CPU 1 first obtains, for example, an average value of pitch information in the segment obtained by the segmentation, and obtains a semitone on the absolute pitch axis having the closest average value. The pitch of the segment is identified for each of the different pitches (step SP20), and thereafter, a histogram is created for all 12 pitches of the pitch information, and a weighting coefficient for each pitch determined for each key and generation of each pitch The product sum with the frequency is obtained, and the key that gives the maximum product sum is determined as the key of the acoustic signal (step SP21).

In the correction process, the CPU 1 first confirms that the process has not been completed for the last segment, and determines the difference between the identified interval of the segment to be processed and the adjacent interval on the scale of the determined key. Is a semitone (e.g., C, F, C, C in C major), and if not, the next segment to be processed is performed without correcting the pitch. And returns to step SP22 (steps SP22 to SP24).

On the other hand, if the identified intervals of the processing segment are those intervals, the CPU 1 aggregates the pitch information between the identified interval of the segment and the intervals that differ by a semitone on the scale of the determined key. (Step SP25). For example,
If the pitch of the processing segment is "mi" in C major, the distribution of pitch information between the pitch information corresponding to "mi" and "fa" of the processing segment is determined.
Therefore, even in the pitch information of this segment, pitch information that is not between the semitones is not counted.

Next, the CPU 1 determines whether there is a lot of pitch information larger than the pitch information in the middle of the semitone, or a lot of pitch information smaller than the pitch information in the middle of the semitone. (Step SP26).

When the identification is reviewed and corrected in this manner, the processing segment is shifted to the next segment, and the process returns to step SP22.

The reason for reconsidering the pitch in the case where the difference between adjacent pitches in the determined key in the determined key is a semitone is because the pitch difference is a semitone, the possibility of identification error may occur. Is large.

The above process is repeated to review the pitch for all the segments, and when the review process for the last segment is completed, a positive result is obtained in step SP22, and the processing program ends.

FIG. 2 shows an example of correction of the identified pitch, in which the determined key is C major and the pitch identified based on the average value of the pitch information is "mi". I have. Since the identified pitch of this segment is "mi", the process proceeds to the correction process, and only the pitch information between "mi" and "fa", that is, the pitch information of the period T1, is totaled.
The pitch information above and below the pitch information value PC between "mi" and "fa" is counted. In this case, there is more pitch information in this period T1 than the pitch information value PC. The pitch of the segment is re-identified.

Therefore, according to the above-described embodiment, when the pitch difference between adjacent pitches in the determined key is a semitone, the pitch of the segment is further reviewed in detail. Can be accurately identified.

Other Embodiments In the above-described embodiment, an example in which the average value of the pitch information of the segment is reviewed to be identified to the closest interval has been described. However, the interval identified by another interval identification method is similarly reviewed. You can also.

Further, in the above-described embodiment, the pitch is re-identified based on the pitch information larger and smaller than the intermediate pitch information of the two pitches to be reviewed. However, the pitch may be re-examined by another method. good. For example, the pitch information of the processing segment may be reviewed based on the average value or the most frequent pitch information between the two intervals to be reviewed.

Further, in the above-described embodiment, the CPU 1 executes all the processing shown in FIG. 4 according to the program stored in the main storage device 3. However, a part or all of the processing is performed by a hardware configuration. May be executed. For example, as shown in FIG. 5 in which the same reference numerals are given to the corresponding parts in FIG. 3, after the sound signal from the sound signal input device 8 is amplified through the amplifier circuit 10, the pre-filter is further added.
The digital signal is supplied to an analog / digital converter 12 via an analog-to-digital converter 11 and converted into a digital signal. The acoustic signal converted to the digital signal is subjected to autocorrelation analysis by a signal processor 13 to extract pitch information, and a square sum processing is performed. And extract power information
It may be provided to the software processing system by the CPU1. As a signal processor 13 used in such a hardware configuration (10 to 13), a processor (for example, a processor that can process a signal in a voice band in real time and is provided with an interface signal with a host CPU 1)
NEC Corporation μPD7720) can be applied.

[Effect of the Invention] According to the present invention as described above, when the pitch difference between adjacent pitches in the scale of the key whose identified pitch is determined is a semitone, the pitch of the segment is further reviewed in detail. As a result, the pitch of each segment can be accurately identified, and an automatic transcription method and apparatus that can improve the accuracy of final score data can be obtained.

[Brief description of the drawings]

FIG. 1 is a flowchart showing a process of correcting an identified pitch according to an embodiment of the present invention, FIG. 2 is a schematic diagram for explaining the process of correcting the identified pitch, and FIG. 3 is an automatic music transcription to which the present invention is applied. FIG. 4 is a block diagram showing the configuration of the system, FIG. 4 is a flowchart showing the automatic transcription process, and FIG. 5 is a block diagram showing another configuration of the automatic transcription system. 1 ... CPU, 3 ... main storage device, 6 ... auxiliary storage device,
7 ... A / D converter, 8 ... Acoustic signal input device.

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Masaki Fujimoto 5-7-15 Shiba, Minato-ku, Tokyo Inside NEC Technical Information System Development Co., Ltd. (72) Inventor Masanori Mizuno 5-7-1 Shiba, Minato-ku, Tokyo No. 15 Examiner, NEC Technical Information Systems Development Co., Ltd. Shigeo Arai

Claims (2)

(57) [Claims] 1. A process for extracting pitch information representing a pitch and power information of the sound signal, which is a repetition period of an input sound signal waveform, and processing the sound signal based on the pitch information and / or the power information. A segmentation process of dividing the interval into segments that can be regarded as the same interval, a pitch identification process of identifying the interval on the absolute pitch axis as a pitch of the segmented section based on the pitch information, and according to a distribution state of the pitch information. At least a key determination process of determining a key of the sound signal, wherein the automatic music transcription method of converting the sound signal into musical score data, wherein the pitch identified as the pitch of the section is adjacent to the scale of the determined key. A process of extracting a section of a pitch in which a matching pitch difference is a semitone, and a tone of a key determined as an identification pitch of the section in pitch information of the extracted section. A process of summarizing pitch information between pitches that differ by a semitone on the floor and a process of re-identifying a pitch in that section according to the tabulated distribution status are provided as post-processes of the above-described key determination process. Automatic transcription method characterized by the following. 2. A pitch / power extracting means for extracting pitch information representing a pitch and power information of the audio signal, which is a repetition period of an input audio signal waveform, and includes: Segmentation means for classifying an acoustic signal into sections that can be regarded as having the same pitch based on the pitch information; pitch identification means for identifying a pitch on the absolute pitch axis as a pitch of the divided section based on the pitch information; and distribution of the pitch information. In an automatic transcription apparatus that partially includes a key determination unit that determines a key of the audio signal according to a situation and converts the audio signal into musical score data, a pitch identified as a pitch of the section is determined. Correction candidate section extraction means for extracting a section of a pitch in which the pitch difference between adjacent pitches is a semitone on the scale of the key, and the section of pitch information of the extracted section Pitch summing means for summing up pitch information between the identified pitch of the pitch and the pitch of the determined key that differs by a semitone, and the identified pitch for re-identifying the pitch of the section according to the tabulated distribution status An automatic transcription apparatus, wherein a correction unit is provided at a stage subsequent to the key determination unit.