JP2604412B2 - Automatic music transcription method and device - Google Patents

Abstract

PURPOSE:To prevent that segmentation is performed erroneously, by finely dividing a section wherein power data is larger than a threshold value at the rising change point of power data and subsequently looking at segmentation again on the basis of the length of a segment. CONSTITUTION:A CPU 1 compares the power data at each analytical point with a threshold value to set not only the section larger than the threshold value as the segment of an effective section but also the section smaller than said threshold value as the segment of an ineffective section and subsequently finely divides the effective section at an extracted rising change point. Thereafter, the CPU 1 counts the lengths of both segments regardless of effectiveness and ineffectiveness and connects the segment whose length is shorter than a predetermined length to the front segment to form one segment. By this method, it can be prevented that segmentation is performed erroneously by a noise component and processing on and after such as intervale identification processing can be well executed.

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an automatic music notation method and apparatus for creating musical score data from acoustic signals such as singing voices, humming voices, and instrument sounds, and more particularly, to a method for converting an audio signal into the same pitch. This relates to a segmentation process for dividing into sections that can be regarded as sections.

[Prior Art] In an automatic transcription system for converting an acoustic signal such as a singing voice, a humming voice, or a musical instrument sound into musical score data, a sound length, a pitch, a key, a time signature, and the like, which are basic information as a musical score from an acoustic signal. Detecting the tempo.

By the way, an acoustic signal is only a signal that continuously includes a repetitive waveform of a basic waveform, and the above-described information cannot be obtained immediately.

Therefore, in the conventional automatic transcription method, first, repetition information (hereinafter, referred to as pitch information) of a basic waveform representing a pitch of an acoustic signal and power information are extracted for each analysis cycle, and thereafter, the extracted pitch information is extracted. And / or dividing the sound signal into sections (segments) that can be regarded as the same pitch based on the power information (this process is called segmentation), and then determines the pitch along the absolute pitch axis of the sound signal of each segment from the pitch information of the segment. Then, the tone of the sound signal is determined based on the determined pitch information, and the information is obtained in the order of determining the beat and tempo of the sound signal based on the segment.

Therefore, the interval, time signature, tempo, and the like are determined based on the segment (tone length), and thus the segmentation processing is particularly important in creating musical score data.

[Problems to be Solved by the Invention] As described above, the segmentation is an important element in creating the score data, and if the accuracy of the segmentation is low, the accuracy of the finally obtained score data also becomes extremely low. The accuracy of the segmentation process itself from the power information should be improved both when performing the final segmentation from both the segmentation result based on the pitch information and the segmentation result based on the power information, or when performing the final segmentation based on the power information. Is desired.

The present invention has been made in view of the above points, and an object of the present invention is to provide an automatic transcription method and apparatus that can perform segmentation based on power information satisfactorily and improve the accuracy of musical score data. Is what you do.

[Means for Solving the Problem] In order to solve the problem, in the first aspect of the present invention, a process of extracting power information of an input audio signal is the same as a process of extracting an audio signal based on the extracted power information. A segmentation process for converting an acoustic signal into musical score data at least including a segmentation process for segmenting into a section that can be regarded as a musical interval. Processing: extracting a rising change point of power information for an effective section; processing of subdividing an effective section by the extracted rising transition point; and processing of an effective section and an invalid section including the subdivided effective section. Processing for extracting length: processing for connecting a section whose extracted length is shorter than a predetermined length to any of the preceding and following sections.

Further, in the second aspect of the present invention, the power extraction means for extracting power information from the input audio signal, and the segmentation means for dividing the audio signal into sections that can be regarded as having the same pitch based on the extracted power information, In an automatic transcription apparatus for converting an audio signal into musical score data provided partially, a division processing unit for dividing a segmentation unit into an effective section in which power information is equal to or more than a predetermined value and an invalid section in which power information is equal to or less than a predetermined value: A change point extracting unit that extracts a rising change point of power information; and a subdivision unit that subdivides an effective section based on the extracted rising change point: lengths of an effective section and an invalid section including the subdivided effective section. And a section correction section for connecting a section whose extracted length is shorter than a predetermined length to one of the preceding and following sections.

[Operation] In the first aspect of the present invention, power information is divided into an effective section having a power value equal to or greater than a predetermined value and an invalid section having a power value equal to or less than the predetermined value so as to reduce the influence of power fluctuation and noise of the audio signal. In addition, since an effective section including two or more sounds may occur, the power rises at the time of sound transition, and the effective rising section is subdivided by extracting a power rising change point. Further, when the section is erroneously classified due to noise or the like, since the length of the section is short, a section whose length is shorter than a predetermined length is extracted and connected to one of the preceding and following sections to form one section. I did it.

Also, in the second aspect of the present invention, similarly, the power information is divided into an effective section having a predetermined value or more and an invalid section having a predetermined value or less by the division processing section so as not to be affected by power fluctuation or noise. . In addition, the rising change point of the power in the effective section is extracted by the change point extracting section, and the effective section is subdivided by the subdivision section at the rising change point so that a section including two or more sounds is not generated. Further, the length of each section is extracted by a section length extracting unit, and a section whose length is shorter than a predetermined length is connected to one of the preceding and following sections by a section correcting unit to form one section. Was prevented from occurring.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

Automatic transcription system First, an automatic transcription system to which the present invention is applied will be described.

In FIG. 4, a central processing unit (CPU) 1 controls the whole of the apparatus, and performs a transcription process shown in FIG. 5 stored in a main storage device 3 connected via a bus 2. Execute the program. In addition to the CPU 1 and the main storage device 3, the bus 2 includes a keyboard 4 as an input device, a display device 5 as an output device, an auxiliary storage device 6 used as a working memory, and an analog /
The digital converter 7 is connected.

An audio signal input device 8 including, for example, a microphone is connected to the analog / digital converter 7. The acoustic signal input device 8 captures an acoustic signal such as singing or humming uttered by a user or a musical tone generated from a musical instrument and converts the signal into an electric signal, and converts the electric signal into an analog / digital converter. 7 is output.

When a process is instructed by the keyboard input device 4, the CPU 1 starts the transcription process, executes a program stored in the main storage device 3, and converts the sound converted into a digital signal by the analog / digital converter 7. The signals are temporarily stored in the auxiliary storage device 6, and thereafter, these sound signals are converted into musical score data by executing the above-described program and output to the display device 5 as necessary.

Next, the music transcription process performed by the CPU 1 after capturing the audio signal will be described in detail with reference to the flowchart shown in the functional level of FIG.

First, the CPU 1 performs an autocorrelation analysis of the acoustic signal to extract pitch information of the acoustic signal at each analysis cycle, and also performs a sum-of-squares process to extract power information at each analysis cycle, and then performs noise removal and smoothing processing. And other post-processing (steps SP1, SP
2). Thereafter, for the pitch information, the CPU 1 calculates a shift amount of the pitch axis of the acoustic signal with respect to the absolute pitch axis based on the analysis state, and performs a tuning process of shifting the obtained pitch information according to the shift amount. Execute (step SP3). That is, the pitch information is corrected so that the difference between the pitch axis of the singer or the musical instrument that generated the acoustic signal and the absolute pitch axis becomes smaller.

Next, the CPU 1 obtains a continuous period of pitch information in which the obtained pitch information is considered to indicate the same pitch,
A segmentation is performed to divide the acoustic signal into segments for each sound, and a segmentation is performed based on the obtained change in power information (step SP
4, SP5). On the basis of these two pieces of segment information obtained, the CPU 1 calculates a reference length corresponding to a time length of a quarter note, an eighth note, and the like, and performs more detailed segmentation based on the reference length (step SP6).

The CPU 1 identifies the pitch of the segment as a pitch on the absolute pitch axis that can determine that the pitch information is the closest based on the pitch information of the segment thus segmented, and further identifies the pitch of the identified continuous segment. Segmentation is again performed based on whether or not are the same (steps SP7 and SP8).

Thereafter, the CPU 1 obtains a product sum of the frequency of appearance of the scale obtained by calculating the pitch information after tuning and a predetermined weighting coefficient determined according to the key, and based on the maximum information of the product sum, for example, Determine the key of the music of the input audio signal, such as C major or A minor, and review and correct the pitch of the predetermined pitch on the scale in the determined key in more detail with respect to the pitch information (step SP9, S
P10). Next, the CPU 1 determines whether or not there is the same one for consecutive segments from the finally determined pitch,
The segmentation is reviewed based on whether or not there is a change in power between consecutive segments, and final segmentation is performed (step SP11).

Once the pitch and segment are determined in this way,
The CPU 1 extracts measures from the viewpoint that the music starts from the first beat, the last sound of the phrase does not extend to the next measure, and there is a break in each measure, and determines the time signature from the measure information and the segmentation information. The tempo is determined from the determined time signature information and the length of the bar (step SP1).
2, SP13).

Then, the CPU 1 organizes the information on the determined pitch, pitch, key, beat, and tempo to finally create the musical score data (step SP14).

Segmentation Based on Power Information Next, a segmentation process (step SP5 in FIG. 5) based on the power information of the audio signal in such an automatic transcription system will be described in detail with reference to the flowcharts of FIGS. FIG. 1 is a flowchart showing such processing at a functional level, and FIG. 2 is a flowchart showing FIG. 1 in more detail.

As power information of the acoustic signal, the acoustic signal is squared at each sampling point in the analysis cycle, and
The sum of the power values is used as power information in the analysis cycle.

The CPU 1 compares the power information of each analysis point with a threshold, divides the audio signal into a section larger than the threshold and a section smaller than the threshold, sets a section larger than the threshold as a segment of an effective section, sets a smaller section as a segment of an invalid section, At the beginning of the invalid segment, and the beginning of the invalid segment at the beginning of the invalid section (steps SP15 and SP16). The reason for this is that the pitch of the acoustic signal is often unstable in a range where the power information is small, and the pitch identification is often not performed properly, and also to detect a rest period. .

Next, the CPU 1 calculates a change function of the power information in the divided effective segments, extracts a rising change point of the power information from the change function, and subdivides the effective segment into the extracted rising change points. A mark indicating the start of the effective segment is given at that point (steps SP17 and SP18). The reason for this is that the sound may shift to the next sound while maintaining a certain level of power. In this case, there is a possibility that a segment including two or more sounds may be generated by the above processing alone. This is because such segments are subdivided by focusing on increasing the power at the start of the next sound.

Thereafter, the CPU 1 counts the length of each segment irrespective of validity or invalidity, and connects a segment whose length is shorter than a predetermined length to the previous segment to form one segment (steps SP19 and SP20). This is because the segment may be divided into minute segments due to noise or the like, and the segment is connected to another segment. In addition, although one sound is intended, the sound is divided into a plurality of segments by the subdivision processing based on the rising transition point described above.

Next, such processing will be described in more detail with reference to the flowchart of FIG.

CPU1 first clears the analysis point parameter t to 0,
Confirm that the analysis point data to be processed is not completed, and power information of the acoustic signal at that analysis point
It is determined whether (t) is smaller than the threshold value θp (steps SP21 to SP23).

When the power information power (t) is smaller than the threshold value θp, the CPU 1 increments the analysis point parameter t and returns to step SP22 again to determine the power information of the next analysis point (step SP24).

On the other hand, the CPU 1 determines in step SP23 that the power information power
If the value of (t) is equal to or larger than the threshold value θp, the analysis point is marked as an effective segment start point, and the next step SP
The process moves to the process after step 26 (step SP25).

At this time, the CPU 1 confirms that the processing has not been completed for all analysis points,
It is determined whether the value of (t) is smaller than the threshold value θp,
If it is equal to or more than p, the analysis point parameter t is incremented and the process returns to step SP26 (steps SP26 to SP28). On the other hand, when the value of the power information power (t) becomes smaller than the threshold value θp, the analysis point is marked as an invalid segment start point, and the process returns to step SP22 (step SP29).

The CPU 1 performs the above processing until it detects in step SP22 or SP24 that processing has been completed for all analysis points, and power information power (t) of all analysis points.
Then, the sound signal is divided into an effective segment equal to or more than the threshold value θp and an invalid segment less than the threshold value θp, and the process proceeds to step SP30 and subsequent steps.

After this, the CPU 1 sets the analysis point parameter t to 0.
Clear and start the following processing from the first analysis point (step SP30). After confirming that the analysis point data to be processed has not been completed, the CPU 1 determines whether or not the analysis point has been marked with the start of a valid segment (steps SP31 and SP3).
2). If it is not the analysis point at the start of the effective segment, the CPU
1 increments the analysis point parameter t and returns to step SP29 described above (step SP33).

On the other hand, when the analysis point at the start of the effective segment is detected, it is confirmed again that there is no analysis point data to be processed, and it is further determined whether the analysis point is at the analysis point at the start of the invalid segment (steps SP34 and SP35). . When the analysis point is not the analysis point at the start of the invalid segment, the analysis point is an analysis point in the valid segment. Therefore, the change function of the power information power (t) (since it is used for the extraction of the rising edge of the power information in the subsequent processing, In the following, d (t) is obtained according to equation (1) (step SP36).

d (t) = {power (t + k) -power (t)} / {power (t + k) + power (t)} (1) where k is a natural number indicating an appropriate time for capturing a change in power. .

Thereafter, the CPU 1 determines whether or not the value of the rising extraction function d (t) obtained is smaller than the threshold value θd. If the value is smaller, the analysis point parameter t is incremented and the process returns to step SP34 (steps SP37 and SP38). On the other hand, the rising extraction function d
If (t) is equal to or larger than the threshold value θd, the analysis point is marked as the start of a new segment (step SP).
39). Thus, the effective segment is subdivided.

Thereafter, the CPU 1 confirms that the processing has not been completed for all the analysis point data, and then determines whether or not the analysis point being processed is marked with a start of an invalid segment, and the analysis point is attached. In this case, the process returns to step SP31 described above and returns to the process of detecting the start point of the next valid segment (steps SP40 and SP41).

On the other hand, when the analysis point is not the analysis point at the start of the invalid segment, the rising extraction function d (t) is obtained by the equation (1) based on the power information power (t).
It is determined whether (t) is smaller than the threshold value θp (steps SP42 and SP43). When it becomes smaller, the process returns to step SP34, and proceeds to the process of extracting the rising change point of the power information. On the other hand, in step SP43, the rising extraction function d of the analysis point
If (t) continues to be equal to or greater than the threshold value θp, the analysis point parameter t is incremented, and the process returns to step SP40 to determine whether the rising extraction function d (t) has become smaller than the threshold value θp for the next analysis point.

By repeating the above processing, steps SP31, SP
If it is detected in 34 or SP40 that the processing has been completed for all analysis points, the process proceeds to segment review processing based on the segment length after step SP45.

In this processing, after clearing the analysis point parameter t to 0, the CPU 1 further confirms that the analysis point data has not been completed, and marks the start of the segment regardless of whether the analysis point is valid or invalid. It is determined whether or not there is (steps SP45 to SP47). If it is not the segment start point,
The process returns to step SP46 to increment the analysis point parameter t and move to the next analysis point data (step SP48).
When the segment start point is detected, the segment length parameter L is set to an initial value "1" to count the length of the segment starting from the start point (step SP4).
9).

Thereafter, the CPU 1 increments the analysis point parameter t, further confirms that the analysis point data is not completed, and determines whether or not the analysis point is marked as a start of a segment regardless of validity or invalidity. Judge (step SP
50-52). As a result, if it is not the segment start point, the segment length parameter L is incremented, the analysis point parameter t is also incremented, and the process returns to step SP51 (steps SP53 and SP54).

By repeating the processing of steps SP51 to SP54, the analysis point to which the start of the segment is marked is eventually given, and a positive result is obtained in step SP52. The segment length parameter L at this time corresponds to the distance between the analysis point to be processed marked with and the analysis point marked immediately before it, that is, corresponds to the length of the segment. . If a positive result is obtained in step SP52, the CPU 1 determines whether or not the parameter L (segment length) is shorter than the threshold value θL. Returning to step SP46, if it is smaller than the threshold value θL, the mark of the start of the front segment is removed, that is, this segment is connected to the front segment, and the process returns to step SP46 (steps SP55 and SP56).

When returning to step SP46 from step SP55 or SP56, if the analysis point data is not completed, an affirmative result is obtained immediately in step SP47, the process proceeds to step SP49 and thereafter, and the mark just found is marked. The operation shifts to the operation of searching for the next mark. In the same manner as described above, the next mark is found and the segment length is reviewed.

By repeating such processing, the review of all segment lengths is completed, and a positive result is obtained in step SP46, and the CPU 1 ends the processing program.

FIG. 3 shows an example of the segmentation by such processing. In the case of this example, the valid segments S1 to S8 and the invalid segments S11 to S8 are repeated based on the power information power (t) by repeating the processing up to step SP29.
It is divided into 18. Thereafter, by repeating the processing up to step SP44, the effective segment S4 is determined based on the rising extraction function d (t), and the effective segment S4 is determined by the rising edge of the power at S41.
And S42. Then, after that, step SP4
The processing after 5 is performed and the segment length is reviewed, but in this case, there is no segment shorter than the predetermined length, so that no particular segment is connected.

Therefore, according to the above-described embodiment, the audio signal is divided into an effective segment whose power information is equal to or more than the threshold value and an invalid segment whose power information is equal to or less than the threshold value, and the effective segment is subdivided according to a rising transition point of the power information. Because the length is reviewed according to the length, it is possible to execute highly accurate segmentation without performing erroneous segmentation due to noise or fluctuation of power.

That is, since a section having power information equal to or greater than the threshold value is set as an effective segment, an unstable period of a pitch at which the audio power is low may not be used for a subsequent process such as a pitch identification process. Further, since the rising change point of the power is extracted and subdivided, the segmentation can be favorably performed even when shifting to the next sound while the power is maintained at a predetermined level or more. Further, since the review is performed based on the segment length, it is possible to eliminate the need to divide one sound or rest period into a plurality of segments.

Other Embodiments In the above-described embodiment, the power information is obtained by using the sum of squares of the audio signal. However, other parameters may be used. For example, the square root of the sum of squares may be used. Further, the rising extraction function is obtained as shown in equation (1), but other parameters may be used. For example, the rising of power information is extracted by a function using only the numerator of equation (1). Is also good.

Furthermore, in the above-described embodiment, the segment having an insufficient segment length is connected to the immediately preceding segment. However, the segment may be connected to the immediately following segment. It may be connected to the immediately preceding segment, and if it is a rest section, connected to the immediately following segment.

In the above-described embodiment, the CPU 1 executes all the processing shown in FIG. 5 according to the program stored in the main storage device 3. However, a part or all of the processing is performed by a hardware configuration. May be executed. For example, as shown in FIG. 6 in which the same reference numerals are given to the corresponding parts in FIG. 4, the audio signal from the audio signal input device 8 is amplified through the amplifier circuit 10, and then the pre-filter is added.
The digital signal is supplied to an analog / digital converter 12 via an analog-to-digital converter 11 and converted into a digital signal. The acoustic signal converted to the digital signal is subjected to autocorrelation analysis by a signal processor 13 to extract pitch information, and a square sum processing is performed. And extract power information
It may be given to the software processing system by CPU1. As the signal processor 13 used in such a hardware configuration (10 to 13), a processor capable of processing a signal in a voice band in real time and providing an interface signal with a host CPU 1 (for example, NEC Corporation) Co., Ltd. μPD7720) can be applied.

[Effects of the Invention] As described above, according to the present invention, the power information is divided into sections larger and smaller than the threshold value, and the larger section is subdivided at the rising transition point of the power information, and then based on the segment length. Since the segmentation is re-evaluated, it is possible to reduce the possibility that the segmentation is erroneously performed due to a noise component, etc. it can.

[Brief description of the drawings]

FIG. 1 is a schematic flowchart showing a segmentation process based on power information according to one embodiment of the present invention, FIG. 2 is a flowchart showing the segmentation process in more detail, and FIG. 3 is a characteristic curve showing an example of the segmentation by the process. FIG. 4 is a block diagram showing the configuration of an automatic transcription system to which the present invention is applied, FIG. 5 is a flowchart showing the automatic transcription process, and FIG. 6 is a block diagram showing another configuration of the automatic transcription system. is there. 1 ... CPU, 3 ... main storage device, 6 ... auxiliary storage device,
7 ... A / D converter, 8 ... Acoustic signal input device.

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Masanori Mizuno Examiner, NEC Technical Information System Development Co., Ltd. Shigeo Arai, 5-7-15 Shiba, Minato-ku, Tokyo

Claims (2)

(57) [Claims] An information processing apparatus comprising: at least a process of extracting power information of an input audio signal; and a segmentation process of dividing the audio signal into sections that can be regarded as having the same pitch based on the extracted power information. In the automatic music transcription method for converting into musical score data, the segmentation processing includes a process of dividing the power information into an effective section having a predetermined value or more and an invalid section having a power value of not more than the predetermined value, and a rising change of the power information for the effective section. A process of extracting a point, a process of subdividing the valid section at the extracted rising transition point, and a process of extracting the length of the valid section and the invalid section including the subdivided valid section; An automatic music transcription method, comprising: connecting a section whose extracted length is shorter than a predetermined length to any of the preceding and following sections. 2. A power extraction unit for extracting power information from an input audio signal, and a segmentation unit for dividing the audio signal into sections that can be regarded as having the same pitch based on the extracted power information. In an automatic music transcription apparatus for converting the acoustic signal into musical score data, the segmentation means may be divided into an effective section in which the power information is equal to or more than a predetermined value and an invalid section in which the power information is equal to or less than the predetermined value. A change point extraction unit that extracts a rising change point of the power information; a subdivision unit that subdivides the effective section with the extracted rising change point; and an effective section including the subdivided effective section; A section length extraction unit that extracts the length of the invalid section, and a section correction unit that connects a section whose extracted length is shorter than a predetermined length to one of the preceding and following sections. Automatic transcription apparatus being characterized in that form.