JP2015056149A - Musical score recognition device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a musical score recognition device and a program for improving a recognition rate of a musical score.SOLUTION: Musical score recognition device includes: a prior recognizing processing part which recognizes some of musical symbols based on a musical score image; a correction part which corrects a recognition result by the prior recognizing processing part; and a recognizing processing part which recognizes other musical symbols based on the musical score image using the recognition result corrected by the correction part.

Description

  The present invention relates to a score recognition apparatus and a program.

  2. Description of the Related Art A score recognition device having a measure length calculation unit that calculates a measure length that elapses when each of a plurality of measures in which a staff described in a score representing a music is divided by a plurality of measure lines is known. (For example, refer to Patent Document 1). The rhythm pattern correction unit corrects the misidentified rhythm pattern of the misidentified measure whose measure length does not correspond to the time signature described in the staff so as to correspond to the time signature. The rhythm pattern dictionary database records a plurality of rhythm patterns in a storage device. The rhythm pattern correction unit searches for a similar rhythm pattern similar to the misidentified rhythm pattern from the plurality of rhythm patterns, and changes the misidentified rhythm pattern to a similar rhythm pattern. The rhythm pattern dictionary database associates a plurality of code lateral position patterns with a plurality of rhythm patterns. The rhythm pattern correction unit searches for a similar code horizontal position pattern similar to the horizontal position pattern of the plurality of codes described in the misrecognized measure from the multiple code horizontal position pattern, and generates a similar rhythm pattern corresponding to the similar code horizontal position pattern. Change the misperception rhythm pattern.

  There is also known a musical score recognition apparatus that recognizes a symbol from musical score image data and generates musical score data based on the recognized symbol (see, for example, Patent Document 2). The score recognition device includes handwritten score recognition processing means for recognizing a symbol of a handwritten score. The handwritten score recognition processing means executes a matching process using a recognition dictionary to recognize a symbol having the best matching degree and recognizes one or more symbols having the next highest matching degree as candidate symbols. Based on the function and musical score notation style information, the validity of the symbol with the best matching is judged, and if it is judged to be invalid, the symbol with the best matching is changed to the candidate symbol and the validity is confirmed. It has a recognition result correction function for determining and correcting a symbol having the best matching degree to a candidate symbol determined to be valid. The recognition result correction function compares the bar performance length calculated from the recognized symbols with the actual bar length calculated from the time signature information as the notation style information in the score, and the bar performance length is the actual bar length. If the measure performance length is different from the actual measure length, the rhythm pattern of the measure is compared with the rhythm pattern dictionary and the closest rhythm pattern is obtained. Includes the ability to correct symbols.

  Also, there is known a score recognition device having an image acquisition unit that acquires an image including information on a sheet music score from an image reading unit (see, for example, Patent Document 3). The score recognition means first recognizes the score included in the image acquired by the image acquisition means using the first score recognition method. The error location detection means detects a location that is erroneously recognized from the score recognized by the recognition means. The control means controls the score recognition means so as to perform score recognition using a score recognition method different from the first score recognition method for the erroneous recognition location detected by the mistake location detection means. The score correcting means recognizes the misrecognized portion with the correct recognition result when the misrecognized location is correctly recognized as a result of performing the score recognition using a score recognition method different from the first score recognition method. Correct it. The error location detection means detects an error location from the time signature and the number of beats of the notes in the measure.

Japanese Patent No. 4738135 Japanese Patent No. 5030144 JP 2012-73769 A

  The score includes a printed score and a handwritten score. The recognition of handwritten music is more erroneous and the recognition rate is lower than the recognition of printed music.

  An object of the present invention is to provide a score recognition device and a program capable of improving the score recognition rate.

  The score recognition apparatus of the present invention is corrected by a pre-recognition processing unit for recognizing a part of score symbols based on a score image, a correction unit for correcting a recognition result of the pre-recognition processing unit, and the correction unit. And a main recognition processing unit for recognizing other musical score symbols based on the musical score image using the recognition result.

  Further, the program of the present invention has been modified by the pre-recognition processing step for recognizing some musical notation symbols based on the score image, the correction step for correcting the recognition result of the pre-recognition processing step, and the correction step. This is a program for causing a computer to execute a main recognition processing step of recognizing another musical score symbol based on the musical score image using a recognition result.

  Especially, recognition of handwritten music is often misrecognized, but the recognition result of some music symbols is corrected and the score recognition rate is improved by recognizing other music symbols using the corrected recognition results. be able to.

FIG. 1 is a flowchart showing a processing method of the score recognition apparatus. FIG. 2 is a diagram showing a recognition screen. FIG. 3 shows a score image. FIG. 4 is a diagram illustrating a determination result of a musical score image to be recognized. FIG. 5 is a diagram showing an inquiry screen as to whether to recognize the whole after correcting the prior recognition result. FIG. 6 is a diagram showing an original score image. FIG. 7 is a diagram illustrating a recognition result of the prior recognition processing unit. FIG. 8 is a diagram illustrating a correction result of the correction unit. FIG. 9 is a diagram illustrating a recognition result of the first handwritten score recognition unit. FIG. 10 is a diagram illustrating a recognition result of the second handwritten score recognition unit. FIG. 11 is a diagram illustrating a hardware configuration example of a computer for realizing the score recognition apparatus. FIG. 12 is a diagram illustrating a configuration example of a score recognition apparatus.

  FIG. 12 is a diagram illustrating a configuration example of a score recognition apparatus 1200 according to the embodiment of the present invention. A score recognition apparatus 1200 includes a score image designation unit 1201, a staff recognition processing unit 1202, a recognition target determination unit 1203, a recognition instruction unit 1204, a first handwritten score recognition unit 1205, a second handwritten score recognition unit 1206, a printed score. It has a recognition unit 1207 and a display unit 1208. The first handwritten score recognition unit 1205 includes a prior recognition processing unit 1211, a correction unit 1212, and a main recognition processing unit 1215. The main recognition processing unit 1215 includes a score recognition processing unit 1213 and an estimation processing unit 1214. The second handwritten score recognition unit 1206 includes a score recognition processing unit 1221 and an estimation processing unit 1222. The printed score recognition unit 1207 includes a score recognition processing unit 1231 and an estimation processing unit 1232.

  FIG. 11 is a diagram illustrating a hardware configuration example of a computer for realizing the musical score recognition apparatus 1200 of FIG. The computer is, for example, a personal computer or a tablet computer. A central processing unit (CPU) 1102, ROM 1103, RAM 1104, network interface 1105, input device 1106, output device 1107, and external storage device 1108 are connected to the bus 1101. The CPU 1102 performs data processing or calculation and controls various components connected via the bus 1101. The ROM 1103 stores a control procedure (computer program) of the CPU 1102 in advance, and is activated when the CPU 1102 executes the computer program. A computer program is stored in the external storage device 1108, and the computer program is copied to the RAM 1104 and executed by the CPU 1102. The RAM 1104 is used as a work memory for data input / output, transmission / reception, and temporary storage for control of each component. The external storage device 1108 is, for example, a hard disk storage device or a CD-ROM, and the stored content does not disappear even when the power is turned off. The CPU 1102 executes the computer program in the RAM 1104 to perform processing of the score recognition apparatus 1200. A network interface 1105 is an interface for connecting to a network such as the Internet. The input device 1106 is a keyboard and a mouse in a personal computer, and is a touch panel in a tablet computer, and can perform various designations or inputs. The input device 1106 includes a scanner for generating image data of a score. The output device 1107 is a display device, a speaker, a printer, or the like.

  This embodiment can be realized by a computer executing a program. Further, a computer-readable recording medium in which the above program is recorded and a computer program product such as the above program can also be applied as an embodiment of the present invention. As the recording medium, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  FIG. 1 is a flowchart showing a processing method of the score recognition apparatus 1200. The CPU 1102 in FIG. 11 performs the process of the flowchart in FIG. 1 by executing a score recognition program. Hereinafter, an example in which the input device 1106 is a mouse will be described, but the same applies when the input device 1106 is a touch panel.

  First, in step S <b> 101, the display unit 1208 displays the recognition screen shown in FIG. 2 on the output device (display device) 1107. When the user clicks the “scan” button 201 with a mouse, the scanner scans the score and generates a score image file. Note that a score image file may be prepared without using a scanner.

  Next, in step S102, the score image designation unit 1201 checks whether or not a score image file has been registered. The score image designation unit 1201 registers a file automatically when a score image file is generated using a scanner, and registers the designated file when the user directly designates a file. When registering by direct file designation by the user, for example, the image file 202 is selected and registered. Then, the display unit 1208 displays the registered score image 301 as shown in FIG. If the score image is registered, the process proceeds to step S103, and if not, the process proceeds to step S104.

  In step S103, the staff recognition processing unit 1202 performs staff recognition processing of the score image 301. Thereafter, the process proceeds to step S104.

  In step S104, the CPU 1102 checks whether or not one or more score images are registered. If not registered, the process returns to step S102, and if registered, the process proceeds to step S105.

  In step S <b> 105, the CPU 1102 checks whether or not the “recognition execution” button 302 is clicked with the mouse by the user. When the “recognition execution” button 302 is clicked, the process proceeds to step S106, and when not clicked, the process returns to step S102.

  In step S106, the recognition target determination unit 1203 determines whether the score image 301 is a printed score or a handwritten score. The notes in the score image 301 have note heads. For example, the recognition target determination unit 1203 determines that the score image 301 is a printed score when the variation in the shape of the note heads (tama) of the plurality of notes in the score image 301 is smaller than the threshold, and is larger than the threshold. In this case, it is determined that the score image 301 is a handwritten score.

  Next, in step S107, the display unit 1208 displays the determination result of the recognition target determination unit 1203 on the output device (display device) 1107 as shown in FIG. For example, “Judged to be printed score” is displayed.

  In step S108, the recognition instruction unit 1204 checks which of the “recognize as a printed score” button 401 and the “recognize as a handwritten score” button 402 is clicked by the user with the mouse. If the recognition target determination unit 1203 determines that the score is a printed score, the cursor is set as a default at the position of the “recognize as a printed score” button 401, and the setting of “recognize as a printed score” can be easily performed. Become. If the “recognize as printed score” button 401 is clicked, the process proceeds to step S116, and if the “recognize as handwritten score” button 402 is clicked, the process proceeds to step S109.

  In step S109, the display unit 1208 displays the screen shown in FIG. 5 on the output device (display device) 1107, performs prior recognition, and asks the user whether to recognize the whole after correcting the result. The recognition instruction unit 1204 checks which of the “Yes” button 501 and the “No” button 502 is clicked by the user with the mouse. If the “Yes” button 501 is clicked, the process proceeds to step S110, and if the “No” button 502 is clicked, the process proceeds to step S115.

  The processing in steps S110 to S114 is performed by the first handwritten score recognition unit 1205. The first handwritten score recognition unit 1205 includes a prior recognition processing unit 1211, a correction unit 1212, and a main recognition processing unit 1215.

  In step S110, the pre-recognition processing unit 1211 recognizes some important score symbols among all the score symbols based on the registered score image 301. Important musical notation symbols are time signatures, bar lines, clefs and key signatures. For example, as shown in FIG. 6, the prior recognition processing unit 1211 performs recognition processing of a clef symbol (fave symbol) 601, a time signature symbol 602, and bar lines 603 to 605 in the score image 301.

  Next, in step S111, the display unit 1208 displays the original musical score image 301 of FIG. 6 in black, and displays the musical score symbols 702, 703, and 705 of the recognition result of FIG. Overlapped and displayed in red. Originally, the screen of FIG. 6 and the screen of FIG. 7 are overlapped and displayed, but the screen of FIG. 6 and the screen of FIG. 7 are shown separately to clarify the display contents. The pre-recognition processing unit 1211 cannot recognize the handwritten clef symbol (female symbol) 601 in the musical score image 301 in FIG. 6 and cannot display it in the recognition result in FIG. Further, the pre-recognition processing unit 1211 misrecognizes the handwritten “2/4” time signature 602 in the score image 301 of FIG. 6 as the “2/8” time signature 702 of FIG. Further, the prior recognition processing unit 1211 correctly recognizes the bar line 603 in the musical score image 301 in FIG. 6 as the bar line 703 in FIG. Further, the prior recognition processing unit 1211 cannot recognize the blurred bar line 604 in the musical score image 301 of FIG. 6 and cannot display it in the recognition result of FIG. Further, the prior recognition processing unit 1211 correctly recognizes the bar line 605 in the musical score image 301 in FIG. 6 as the bar line 705 in FIG.

  Next, in step S112, the correction unit 1212 corrects the recognition result of FIG. 7 of the prior recognition processing unit 1211 to the correct recognition result of FIG. 8 according to a user instruction. The user clicks on the “sound symbol” button 812 with the mouse, and designates the position of the clef (symbol symbol) 601 in FIG. 6 with the mouse. Then, the correction unit 1212 adds the clef symbol (female symbol) 801 in FIG. 8 as the recognition result of the clef symbol (fault symbol) 601 in FIG. 6. The display unit 1208 displays the red clef symbol (female symbol) 801 in FIG. 8 on top of the clef symbol (fever symbol) 601 in FIG.

  In addition, the user clicks the “eraser” button 811 with the mouse, and designates the position of the time signature “702” of the erroneous recognition “2/8” in FIG. 7 with the mouse. Then, the correction unit 1212 deletes the “2/8” time signature 702 in the recognition result of FIG. The display unit 1208 deletes the display of the time signature 702 of “2/8” in the recognition result of FIG.

  Next, the user clicks the “2/4” time signature button 813 with the mouse, and designates the position of the time signature 602 in FIG. 6 with the mouse. Then, the correction unit 1212 adds the time signature 802 of “2/4” in FIG. 8 as the recognition result of the time signature 602 in FIG. 6. The display unit 1208 displays the red “2/4” time signature 802 in FIG. 8 on top of the time signature 602 in FIG. 6.

  Next, the user clicks the “bar line” button 814 with the mouse and designates the position of the bar line 604 in FIG. 6 with the mouse. Then, the correction unit 1212 adds the bar line 804 in FIG. 8 as the recognition result of the bar line 604 in FIG. The display unit 1208 displays the red bar line 804 in FIG. 8 so as to overlap the bar line 604 in FIG. 6.

  Next, in step S113, the correction unit 1212 checks whether the correction has been completed. Specifically, when the “OK” button 815 is clicked with the mouse by the user, the correcting unit 1212 determines that the correction has been completed, and proceeds to step S114, where the “OK” button 815 has been clicked. If not, the process returns to step S112.

  In step S114, the main recognition processing unit 1215 uses the recognition result corrected by the correction unit 1212, and based on the score image 301, the important score symbols (time signature, bar line, clef, and key signature). Recognize other non-important musical notation symbols. Non-important score symbols are score symbols other than important score symbols (time signature, bar line, clef, and key signature), and include notes and rests.

  Specifically, the main recognition processing unit 1215 includes a score recognition processing unit 1213 and an estimation processing unit 1214. The score recognition processing unit 1213 recognizes non-important score symbols (notes, rests, etc.) based on the score image 301. Here, the score recognition processing unit 1213 recognizes the note 606 in the score image 301 as two eighth notes.

  The estimation processing unit 1214 corrects the recognition result of the non-important score symbols (notes and / or rests, etc.) of the score recognition processing unit 1213 using the recognition result of FIG. 8 corrected by the correction unit 1212. For example, the estimation processing unit 1213 uses the information of the “2/4” time signature 802 and the bar lines 703, 804, and 705 of the recognition result of FIG. Correct the recognition result. That is, the length of the notes and / or rests so that the sum of the lengths of the notes and / or rests in each measure between the measure lines 703, 804 and 705 matches the time signature 802 of “2/4”. Correct the size. For example, the first measure is recognized by the score recognition processing unit 1213 as having one half note and two eighth notes. However, in this case, the first measure becomes 3 beats and does not match the beat number of the time symbol 802 of “2/4”. In this case, the estimation processing unit 1222 corrects the note of the first measure so as to match the number of beats of the “2/4” time signature 802. Specifically, the estimation processing unit 1222 corrects the note 606 in the score image 301 of FIG. 6 from two eighth notes to two decoration notes 906 (FIG. 9). The decorative note 906 is a short decorative sound and is represented by a small note symbol. As described above, the first handwritten score recognition unit 1205 recognizes the score symbol in the score image 301 and generates score data.

  In step S117, the display unit 1208 displays the score of the recognition result of the first handwritten score recognition unit 1205 on the output device (display device) 1107 as shown in FIG. The musical score symbols recognized by the first handwritten musical score recognition unit 1205 are all correct.

  In step S115, the second handwritten score recognition unit 1206 includes a score recognition processing unit 1221 and an estimation processing unit 1222, and performs score recognition processing of the score image 301. The score recognition processing unit 1221 performs recognition processing of all score symbols (including important score symbols and non-important score symbols) based on the score image 301. The estimation processing unit 1222 corrects the recognition result of the score recognition processing unit 1221.

  In this case, the score recognition processing unit 1221 cannot recognize the handwritten clef symbol (Heisei symbol) 601 in the score image 301 of FIG. 6 (see FIG. 7), as in step S110. In this case, since the clef is lost, the estimation processing unit 1222 adds a default clef (g clef) 1001 (FIG. 10).

  Also, the score recognition processing unit 1221 erroneously converts the handwritten “2/4” time signature 602 in the score image 301 in FIG. 6 to the “2/8” time signature 702 in FIG. recognize. Also, the score recognition processing unit 1221 correctly recognizes the bar line 603 in the score image 301 in FIG. 6 as the bar line 703 in FIG. Further, the score recognition processing unit 1221 cannot recognize the blurred bar line 604 in the score image 301 of FIG. 6 (see FIG. 7). Further, the score recognition processing unit 1221 correctly recognizes the bar line 605 in the score image 301 of FIG. 6 as the bar line 705 of FIG. As a result, as shown in FIG. 10, four quarter notes are recognized between the second bars between the bar lines 703 and 705.

  When the estimation processing unit 1222 estimates the time signature based on the four quarter notes in the second measure in FIG. 10, the time signature symbol 702 of “2/8” in FIG. 7 recognized by the score recognition processing unit 1221 is obtained. The time signature 702 of “2/8” in FIG. 7 is corrected to the time signature 1002 of “4/4” in FIG.

  Also, the score recognition processing unit 1221 recognizes the note 606 in FIG. 6 as the two eighth notes 1006 in FIG.

  Next, in step S117, the display unit 1208 displays the score of the recognition result of the second handwritten score recognition unit 1206 on the output device (display device) 1107 as shown in FIG. The score of FIG. 10 has four misrecognitions with respect to the score image 301 of FIG. First, the treble clef 601 in FIG. 6 is erroneously recognized as the treble clef 1001 in FIG. Second, the “2/4” time signature 602 in FIG. 6 is erroneously recognized as the “4/4” time signature 1002 in FIG. Third, the decorative note 606 in FIG. 6 is erroneously recognized as the two eighth notes 1006 in FIG. Fourth, the bar line 604 in FIG. 6 is not recognized, and the score in FIG. 6 is three bars, but the score in FIG. 10 is erroneously recognized as two bars.

  On the other hand, as shown in FIG. 9, all the musical score symbols recognized by the first handwritten musical score recognition unit 1205 are correct. The recognition process of steps S110 to S114 of the first handwritten score recognition unit 1205 can improve the recognition rate as compared with the recognition process of step S115 of the second handwritten score recognition unit 1206.

  In step S116, the printed score recognition unit 1207 includes a score recognition processing unit 1231 and an estimation processing unit 1232, and performs score recognition processing of the score image 301. Similar to the score recognition processing unit 1221 of the second handwritten score recognition unit 1206, the score recognition processing unit 1231 recognizes all score symbols (including important score symbols and non-important score symbols) based on the score image 301. I do. The estimation processing unit 1232 corrects the recognition result of the score recognition processing unit 1231 similarly to the estimation processing unit 1222 of the second handwritten score recognition unit 1206. Thereafter, in step S117, the display unit 1208 displays the score of the recognition result of the printed score recognition unit 1207 on the output device (display device) 1107.

  As described above, the handwritten score is different from the printed score in terms of variations in the shape and size of the note heads in the score, the position of the stem of the note head, etc. Variation is large. Therefore, the score recognition processing units 1213 and 1221 in the handwritten score recognition units 1205 and 1206 perform recognition by increasing the above-described variation threshold as compared to the score recognition processing unit 1231 in the printed score recognition unit 1207.

  The important musical score symbols are not limited to time signatures, bar lines, clefs, and key signatures. The important musical notation symbols only need to include time signatures or bar lines, and the non-important musical notation symbols only need to include notes or rests. Preferably, important score symbols include time signatures and bar lines, and non-important score symbols include notes or rests. More preferably, the important musical notation symbols include time signatures, bar lines, clefs and key signatures.

  The prior recognition processing unit 1211 recognizes at least a time signature and a bar line. The correction unit 1212 corrects the time signature and / or bar line recognized by the prior recognition processing unit 1211. The score recognition processing unit 1213 recognizes at least notes and rests. The estimation processing unit 1214 corrects the notes and / or rests recognized by the score recognition processing unit 1213 using the time signature and / or bar line corrected by the correction unit 1212.

  In particular, recognition of handwritten music is often misrecognized. The first handwritten score recognizing unit 1205 corrects the recognition result of a part of the musical notation symbols of the pre-recognition processing unit 1211 by the processes of steps S110 to S114, and uses the corrected recognition result to change other score symbols. By recognizing, the score recognition rate can be improved. Note that the recognition rate may be poor even for a printed musical score, and therefore, prior recognition and correction processing may be added to a printed musical score.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

1200 score recognition device 1201 score image designation unit 1202 staff recognition processing unit 1203 recognition target determination unit 1204 recognition instruction unit 1205 first handwritten score recognition unit 1206 second handwritten score recognition unit 1207 printed score recognition unit 1208 display unit 1211 in advance Recognition processing unit 1212 Correction unit 1213 Score recognition processing unit 1214 Estimation processing unit 1215 Main recognition processing unit 1221 Score recognition processing unit 1222 Estimation processing unit 1231 Score recognition processing unit 1232 Estimation processing unit

Claims (7)

A pre-recognition processing unit for recognizing some musical notation symbols based on a score image;
A correction unit for correcting the recognition result of the prior recognition processing unit;
A score recognition apparatus, comprising: a main recognition processing unit that recognizes another score symbol based on the score image using the recognition result corrected by the correction unit. The pre-recognition processing unit recognizes a time signature or a bar line,
The score recognition apparatus according to claim 1, wherein the main recognition processing unit recognizes a note or a rest. The pre-recognition processing unit recognizes time signatures and bar lines,
The musical score recognition apparatus according to claim 1, wherein the main recognition processing unit recognizes a note and a rest. The pre-recognition processing unit recognizes time signatures, bar lines, clefs and key signatures,
The musical score recognition apparatus according to claim 1, wherein the main recognition processing unit recognizes a note and a rest. The main recognition processing unit
A score recognition processing unit for recognizing other score symbols based on the score image;
The score recognition apparatus according to claim 1, further comprising an estimation processing unit that corrects the recognition result of the score recognition processing unit using the recognition result corrected by the correction unit. The pre-recognition processing unit recognizes time signatures and bar lines,
The correction unit corrects the time signature and / or bar line recognized by the prior recognition processing unit,
The score recognition processing unit recognizes notes and rests,
6. The estimation processing unit corrects the notes and / or rests recognized by the score recognition processing unit using the time signature and / or bar line corrected by the correction unit. The musical score recognition apparatus described. A pre-recognition processing step for recognizing some musical notation symbols based on a score image;
A correction step of correcting the recognition result of the pre-recognition processing step;
A program for causing a computer to execute a main recognition processing step of recognizing another musical score symbol based on the musical score image using the recognition result corrected in the correcting step.

Images