JP3466894B2 - Music score recognition method and apparatus, and computer readable recording medium recording music score recognition program - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a score recognizing method and a device thereof, which can obtain a high recognition rate by performing a character type distinguishing process on a musical symbol consisting of mixed italic characters and standard characters on a score. The present invention relates to a computer-readable recording medium recording a musical score recognition program.

[0002]

2. Description of the Related Art When recognizing notes, rests, or other symbols on a musical score using a personal computer or a scanner, the musical score image is binarized and read, and the staff and paragraph are recognized. After erasing the staff for each paragraph, a reading label having an arbitrary rectangular shape is set for each symbol, and the label data and the label data previously prepared in the dictionary are pattern-matched. , Was being recognized.

[0003]

Among the musical scores to be read, among the characters forming the musical symbols on the printed musical score, there are alphabets and numbers as characters having two fonts, italic and standard. Among them, the alphabet forms musical symbols that represent ideas, strengths and velocities depending on the combination thereof, and these musical symbols are determined in general character fonts in the printed musical score. Also, the numbers can be used alone or in combination to form musical symbols such as tuplet numbers, fingering numbers, time signature numbers, metronome numbers, repeated parenthesis numbers, long rest numbers, ottava numbers, and tension note numbers of chords, and printed in place. In the score, all the remaining numbers are written in standard letters, except tuplet numbers and Otterva numbers are written in italics. As described above, these characters have different meanings as music symbols depending on the fonts of the same characters, and the fonts of the characters used by the music symbols are generally determined. On the other hand, in the above conventional recognition method, no special character font discrimination processing is performed on these fonts, and the difference in the shape of the reference pattern prepared for each font at the time of pattern matching. The method of obtaining the recognition result was taken only by this. Since there is only a slight pattern difference between the two fonts, it is often mistakenly recognized, and as a result, the musical symbols formed by those characters are mistakenly recognized, and the performance data including the misrecognition data is changed. When an automatic performance is performed by a personal computer or the like based on this, it becomes difficult to perform the music faithfully to the score.

The present invention was devised in view of the above problems of the prior art, and provides a configuration for performing character type character-by-character type discrimination processing for music symbols composed of mixed italic characters and standard characters on a musical score, By doing so, a high recognition rate can be obtained for music symbols in which these fonts are mixed.

[0005]

Before explaining the constitution of the present invention, how the above-mentioned recognition method is carried out, what kind of problems occur when it is carried out, and what is the present inventor? Based on this idea, the process of how the present invention was devised to solve the problem will be described below.

First of all, among the existing characters on the score, the numbers and alphabets are italic and the standard 2 as described above.
There are different types of fonts, and as shown in Table 1 below, these characters have different meanings as musical symbols depending on the fonts, and the fonts of the characters used by the musical symbols are generally determined. There is something. Therefore, in order to obtain a high recognition rate for these musical symbols, italic characters and standard characters must be accurately discriminated.

[0007]

[Table 1]

In the character recognition in the conventional musical score recognition procedure, after erasing images unnecessary for character recognition such as staff lines and bar lines, label extraction is performed, and the shape of the character to be recognized is recognized in each label. The pattern matching with the reference pattern that represents the feature is collectively performed. Here, two types of reference patterns for recognizing each letter of the alphabet and numbers are prepared: standard letters and italic letters. By this pattern matching, italicized upper case alphabets, standard upper case alphabets, italic lower case alphabets, standard lower case alphabets, Obtain the recognition results for italic and standard numbers. The characters recognized in the label unit form a music symbol by the label alone or by combining with a plurality of labels as described below.

Characters recognized as italic numbers are divided into a part of the Otterva symbol and a tuplet number symbol according to the positional relationship with the dotted line, and the characters recognized as standard numbers are first the staff, the metronome symbol, and the repeating bracket. Depending on the positional relationship with symbols, long rest symbols, chord symbols, etc., time signatures, metronome symbols, repeated parenthesis symbols, long rest symbols, and numbers that are part of chord symbols are extracted and the respective symbol attributes are determined.
The remaining standard numbers will be used as fingering symbols after confirming the appropriateness of the label size. The letters recognized as the alphabet are
The sequence in the same font is extracted, and it becomes a part of the character string symbol by the character string matching with the character string composed of each font.

As described above, in the conventional recognition method,
There is no special character font discriminating process, and a method of obtaining a recognition result based only on the shape difference of the reference pattern prepared for each font at the time of pattern matching has been adopted.

However, the reference pattern used for the pattern matching generally uses a universal characteristic for the symbol (character) shape to some extent so as to be able to deal with the shape difference for each score and print state. for that reason,
In many cases, a process corresponding to blurring is used at the time of pre-processing and post-processing for obtaining features and pattern matching. On the other hand, the patterns of standard and italic characters in the same character have only a slight pattern difference, and the difference is almost absorbed in the above-mentioned universal characteristics. There are many false recognitions. Due to the misrecognition between the fonts, a high recognition rate cannot be obtained for a music symbol including a character string symbol or a numeral. In particular, since tuplet numbers and fingering symbols have almost the same label size, it is difficult to distinguish by the label size, and in order to form a music symbol by letters alone without being evaluated by the connection with other symbols, The misrecognition between the fonts appears in the music symbol recognition result as it is.

[0012] Therefore, a study has been made on a structure capable of highly accurately discriminating between the fonts, which is necessary for highly accurately recognizing a musical symbol formed of standard characters and italic characters.

At this time, the basis of the idea of the present invention is that, when observing the difference in the shape between the italicized character and the standard, the italicized pattern is rotated rightward with respect to the standard pattern. As it was deformed, it was expanded and contracted in the vertical direction, and it can be considered that these complex deformations were applied to the standard pattern to form an italic pattern. Therefore, this rotational deformation causes a difference in the main axis direction of the pattern as shown in FIG. 1 (however, the standard pattern does not necessarily have no inclination).

In view of the above, the basic structure of the present invention is to determine a straight line that can be regarded as the same as the main axis direction of the pattern or the same straight line, and determine whether or not it is an italic pattern based on the degree of inclination. That is, in the configuration of the invention according to claim 1, in the score recognition method of reading a score image, recognizing the music symbol, and creating data for performance and score display, among the music symbols to be recognized, character symbols Alternatively, for a symbol including characters, the label is extracted and recognition is performed by pattern matching, then the slope of the estimated regression line of these symbols on the label is detected, and the font of the symbol is determined by the difference in the slope. The basic feature is to determine whether or not italic.

The reason why the font can be discriminated by the above processing will be described. As mentioned above, when observing the difference in shape between the italicized character and the standard, it is intuitive that the italicized pattern is rotated and deformed to the right and vertically expanded and contracted with respect to the standard pattern. Therefore, it can be considered that these complex deformations are applied to the standard pattern to form an italic pattern. Therefore, due to this rotational deformation,
As shown in FIG. 1, a difference appears in the main axis direction of the pattern. For an arbitrary standard character pattern, regression that minimizes the sum of squared errors in the horizontal direction with respect to the sample point coordinates on the XY coordinate plane with each black pixel as the sample point, as shown in FIG. Assuming a straight line, this regression line substantially coincides with the direction of the main axis of the pattern, and a difference appears in the inclination from the regression line of the italic pattern of the same character. Generally, when this estimated regression line is set as x = α + βy, it is proved in the literature such as “Primary Statistical Analysis (Revised Board)” (published by Takamitsu Sawa, Shinsyusha), etc.

[0016]

[Equation 1]

In some sample images, the slope of the regression line for each character of italic characters and standard characters is calculated by the above equation 1, and as shown in Table 2 below, a value of a certain slope is obtained for each character. It can be seen that italic and standard can be distinguished.

[0018]

[Table 2]

Therefore, a threshold value βth for discriminating the slope of the estimated regression line for each character is prepared in advance, and the slope β of the estimated regression line is obtained by the above equation 1 in the binary image label of the target character, and β> If it is βth, italic letters, β
In the case of <βth, the standard character determination process is performed.
Standard lower case alphabets, italic lower case alphabets, standard numbers, obtained by conventional pattern matching processing
By performing the above-mentioned discrimination processing on the italicized number character recognition result, the recognition result can be corrected to a more correct one, and a high recognition rate can be obtained for the music symbols formed by these characters. become.

The above-described structure is the same for the musical score recognition apparatus of claim 3 and the computer-readable recording medium recording the musical score recognition program of claim 5. Among them, the structure of claim 3 is, for the music symbol to be recognized, a label extracting means for extracting the label of the character symbol or a symbol including the character, and pattern matching of these symbols on the extracted label. The pattern matching means for recognizing, the regression line inclination angle detecting means for obtaining the inclination of the estimated regression line of these symbols on the label, and the difference in the inclination determine whether the font of the symbol is italic or not. It is characterized by having a character style discriminating means. Further, according to the configuration of claim 5, a label extraction function for extracting a label of a character symbol or a symbol including a character among the music symbols to be recognized, and pattern matching of these symbols on the extracted label. A pattern matching function for recognition, a regression line inclination angle detection function for obtaining the inclination of the estimated regression line of these symbols on the label, and a difference in the inclination determine whether or not the font of the symbol is italic. It has a musical score recognition program for executing the font discriminating function, and the inclination of the estimated regression line of the symbol detected by the regression line inclination angle detection function determines whether the font of the symbol is italic or not. The font type discriminating function automatically discriminates.

Further, the structures of claims 2, 4 and 6 are such that when the score image itself to be recognized, which will be described later, is inclined,
The method according to claim 2 further comprises a configuration for complementing the inclination, wherein the inclination of the musical score image itself is detected, and the inclination of the estimated regression line is complemented by the detected inclination. The apparatus according to claim 4 further comprises a score image inclination detecting means for detecting the inclination of the musical score image itself, and an inclination complementing means for complementing the inclination of the estimated regression line with the detected inclination. It is characterized by Further, in the recording medium according to claim 6, the program further comprises a score image inclination detecting function for detecting the inclination of the score image itself, and a slope complementing function for complementing the slope of the estimated regression line with the detected slope. It is characterized by having.

[0022]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described below with reference to the accompanying drawings. FIG. 3 shows a flexible disk drive FD which will be described later with a computer-readable recording medium in which the musical score recognition program according to the invention of claim 3 is recorded.
It is a block diagram which shows the Example structure of the musical score recognition apparatus read and operated by external storage devices (or CD-ROM drive etc.), such as D5. This device has a scanner and a MIDI interface circuit added to the configuration of an electronic computer such as a personal computer. CPU1 is ROM2
Alternatively, it is a central processing unit that controls the entire score recognition apparatus based on a program stored in the RAM 3. In addition, it has a built-in timer circuit that interrupts the CPU 1 at a predetermined cycle. The RAM 3 is used as an image data buffer, a work area, etc. in addition to the program area. The hard disk device HDD4 and the flexible disk drive FDD5 store programs, image data, performance data, and the like. CRT6 is CPU
Under the control of 1, the video information output from the CRT interface circuit 7 is displayed, and the information input from the keyboard 8 is taken into the CPU 1 via the keyboard interface circuit 9. The printer 10 prints the print information output from the printer interface circuit 11 under the control of the CPU 1.

The scanner 12 optically scans (printed) a musical score, for example, and converts it into binary or grayscale image data, and can be of any type such as flatbed type, handy type, and feeder type. You can use any scanner. The image information read by the scanner 12 is transferred to the RA through the scanner interface circuit 13.
It is taken into M3 or HDD4. The MIDI interface circuit 14 is an external MID such as a sound source module.
It is a circuit that transmits and receives MIDI data to and from the I device. The bus 15 connects each circuit in the musical score recognition device,
It exchanges various data, programs, addresses, etc. In addition to this, a pointing device such as a mouse or a serial interface circuit such as RS232C may be provided.

FIG. 4 is a flow chart showing the main processing of the CPU 1. In step S1, the scanner 1
The image of the musical score is loaded into the RAM 3 according to 2. The image is captured as binary image data. In step S2, the staff is detected in the score image thus binarized and captured. In step S3, inclination correction is performed on each staff based on the amount of shift with respect to the horizontal line. In step S4, images (non-symbol objects) unnecessary for character recognition such as staff lines and bar lines are deleted. Up to this point, the preprocessing for implementing the present configuration has been performed, and if necessary, these functions may be deleted, replaced with other alternative functions, or additional functions may be added. Next, in step S5, label extraction is performed by the label extraction function, and in step S6,
With the pattern matching function, the individual labels are collectively subjected to pattern matching with a reference pattern that represents the characteristics of the shape of the character to be recognized. Thereafter, in step S7, a process for correcting the result of the pattern matching, that is, the character style of the target character label is determined.
Then, in step S8, a music symbol is formed with respect to the character recognized in the label unit, by the label alone or by combining with a plurality of labels.

In the character recognition process, as shown in FIG. 5, in step S60, the input pattern Pi obtained by the process before the recognition process is used as a reference pattern for both lowercase alphabets and numerals for both standard and italic characters. By including the pattern matching, the recognition result including the standard lowercase alphabet, the italic lowercase alphabet, the standard uppercase alphabet, the italic uppercase alphabet, the standard numeral, and the italic numeral which are the fonts to be distinguished is obtained.

In step S61, as a premise of character type discrimination, a binary image label Pi 'is extracted with respect to the image Pi recognized as the character type discrimination target character.

In the character type discrimination in step 7, the inclination of the estimated regression line is obtained by the regression line inclination angle detection function in step S70 and thereafter in FIG. 5, and the symbol is determined based on the inclination by the character type discrimination function. It is carried out by determining whether or not the font of is italic.

That is, in step S70, the binary image label Pi 'is raster-scanned and the coordinate values (x, y) of the pixels that become black pixels are added to obtain xSum, y.
Sum, divided by the total number of black pixels, sample mean (xAv,
yAv) is calculated.

In step S71, the binary image label is raster-scanned again, and at the coordinate value (x, y) of the pixel that becomes the black pixel, the coordinate error xErr with the sample mean is obtained.
(= X−xAv), yErr (= y−yAv),
The product of xErr and yErr xyErr (= xErr × yE
rr), the square error in the y direction yyErr (= yErr × y
Err) is added to all black pixels, and xy is calculated.
ErrSum and yyErrSum.

In step S72, the slope β of the estimated regression line is calculated by (xyErrSum / yyErrSum).

The discrimination value βth for the target character, which is prepared for each character in advance, is obtained. Table 3 below shows an example of the discriminant value βth for numbers.

[0032]

[Table 3]

Then, in step S73, the slope β of the estimated regression line is less than the discriminant value βth (β <βth).
Then, in step S74, the recognition result is replaced with the standard identical character. If the slope β of the estimated regression line exceeds the discriminant value βth in S73 (β> βth), the recognition result is replaced with the same italicized character in step S75.

Next, an embodiment of the musical score recognition method according to the present invention when the musical score image is inclined will be described. If the score image to be recognized is tilted,
In the configuration of the embodiment described above, in the tilt correction of S3 of FIG. 4, since the correction by rotation is not normally performed, the direction of the main axis in the target label image is still tilted, which may cause misjudgment. Therefore, in the present configuration, this adverse effect can be prevented by adding a function of complementing this inclination.

The inclination of the musical score image is represented by the inclination of the staff with respect to the horizontal direction, as shown in FIG.
Let Lh be the horizontal line passing through the center of the staff, and Lh at the left end of the staff.
Error in the Y-axis direction with respect to ErrL, L at the right end of the staff
Assuming that the error in the Y-axis direction with respect to h is ErrR and the staff length is Lm, the inclination rad of the musical score image is expressed by the following equation 2
Can be found at.

[0036]

[Equation 2]

In order to complement this inclination rad, the above-mentioned FIG.
The step S72 of is performed as follows. That is, from the slope β of the estimated regression line, the angle ra formed by the estimated regression line and the Y-axis
d2 is calculated as rad2 = tan ⁻¹ (β), and the complementary angle rad3 = (rad2
-Rad) and the complementary slope β ′ is β ′ = tan (ra
d3).

After the above-described complementary slope β'is obtained, the recognition results are replaced with the same standard or italicized characters by performing the processes of steps S73 to S75.

In the above two embodiments, the operation is performed by the CPU of the computer under software control. However, the present invention is not limited to this, and an equivalent logic circuit capable of performing the same operation may be used. Needless to say, good things.

[0040]

According to the configuration of the present invention described in detail above, it is possible to discriminate the italic / standard character fonts for each music symbol consisting of mixed italic characters and standard characters on a musical score. A high recognition rate can be obtained for music symbols with mixed fonts. Also, when an automatic performance is performed by a personal computer or the like based on the recognized data, it is possible to perform a performance faithful to the score. Further, even when the score image is tilted, this adverse effect can be prevented by adding a configuration / means and function that complements the tilt.

[Brief description of drawings]

FIG. 1 is an explanatory diagram of standard characters and italic characters in a main axis direction.

FIG. 2 is an explanatory diagram of an estimated regression line that minimizes the sum of squared errors in the X direction.

FIG. 3 is a block diagram showing a configuration of an embodiment of a musical score recognition apparatus which operates by loading a computer-readable recording medium recording a musical score recognition program into an external storage device.

FIG. 4 is a flowchart showing a main process of a CPU.

FIG. 5 is a flowchart showing a standard character / italic character discrimination process.

FIG. 6 is an explanatory diagram showing paragraphs for detecting the inclination of the score image.

[Explanation of symbols]

1 CPU 2 ROM 3 RAM 4 hard disk drive 5 Flexible disk drive 6 CRT 7 CRT interface circuit 8 keyboard 9 Keyboard interface circuit 10 Printer 11 Printer interface circuit 12 scanner 13 Scanner interface circuit 14 MIDI interface circuit 15 bus

─────────────────────────────────────────────────── ─── Continuation of front page (58) Fields surveyed (Int.Cl. ⁷ , DB name) G06K 9/00-9/76 G10H 1/00 102

Claims (6)

(57) [Claims] 1. A musical score recognition method for reading a musical score image, recognizing the musical symbols, and creating data for performance and musical score display, wherein among musical symbols to be recognized, a character symbol or a symbol including a character is recognized. , The label is extracted and recognition is performed by pattern matching, then the slope of the estimated regression line of these symbols on the label is detected, and it is determined whether the font of the symbol is italic by the difference in the slope. A musical score recognition method characterized by discriminating. 2. The musical score recognition method according to claim 1, wherein
A score recognizing method, characterized in that a slope of a score image itself is detected, and the slope of the estimated regression line is complemented by the detected slope. 3. A musical score recognition apparatus for reading a musical score image, recognizing the musical symbols, and creating data for performance and musical score display, among musical symbols to be recognized, for a character symbol or a symbol including a character. A label extracting means for extracting the label, a pattern matching means for recognizing these symbols on the extracted label by pattern matching, and a regression line slope for obtaining the slope of the estimated regression line of these symbols on the label A musical score recognition apparatus comprising: angle detection means; and character style determination means for determining whether or not the character style of the symbol is italic based on the difference in inclination. 4. The musical score recognition apparatus according to claim 3,
The musical score recognition apparatus further comprising: a musical score image inclination detecting means for detecting the inclination of the musical score image itself; and a inclination complementing means for complementing the inclination of the estimated regression line with the detected inclination. 5. A computer-readable recording medium in which a musical score recognition program for reading a musical score image and recognizing the musical symbols to create data for performance and musical score display, among the musical symbols to be recognized, A label extraction function for extracting a label from a character symbol or a symbol including a character, a pattern matching function for recognizing these symbols on the extracted label by pattern matching, and estimation of these symbols on the label A computer recording a musical score recognition program for executing a regression line inclination angle detection function for obtaining the inclination of the regression line and a font discriminating function for discriminating whether or not the font of the symbol is italic depending on the inclination. A readable recording medium. 6. A computer-readable recording medium in which the musical score recognition program according to claim 5 is recorded,
A computer recording a musical score recognition program, further comprising a musical score image inclination detecting function for detecting the inclination of the musical score image itself and an inclination complementing function for complementing the inclination of the estimated regression line with the detected inclination. A readable recording medium.