JP2008123181A - Musical score recognition device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a musical score recognition device and program for appropriately recognizing musical score marks from musical score image data provided by reading a handwritten musical score or a printed musical score. <P>SOLUTION: The printed or handwritten musical score image data sent out from a scanner 12 is stored in RAM 3 or an HDD 4. CPU1 recognizes the marks from the musical score image data stored in the RAM 3 or the HDD 4 and creates musical score data based on the recognized marks. The CPU 1 first enters a printed musical score recognition mode, accumulates information for assuming that the musical score is handwritten while recognizing the musical score as the printed musical score, and computes handwritten musical score determination information based on the accumulated value. The device determines whether the musical score is assumed to be a handwritten musical score in fact based on the handwritten musical score determination information, and when it is determined to be a handwritten musical score, mode is automatically switched to a handwritten musical score recognition mode and re-recognizes the musical score as a handwritten musical score. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a score recognition apparatus and a program for recognizing a score symbol from score image data and generating score data based on the recognized score symbol, and in particular, a score obtained by reading it from a handwritten score or a printed score. The present invention relates to a musical score recognition apparatus and program capable of appropriately recognizing musical score symbols from image data.

  The score includes a score created by printing or cleaning, a score (printed score) created equivalent to printing by various notation software, a normal pencil, pen, or calligraphy pen. There is a musical score (handwritten musical score) handwritten with a pen that can represent different lines.

  Since normal pencils and pens have the same thickness regardless of the direction, when they are used as they are, the line thickness is the same regardless of the direction as shown in FIG. By tracing and thickening, like the printed musical score, the connection line thickness and the note head and note tails can be made thick. Since the calligraphic pen can express lines having different thicknesses in the direction perpendicular to each other, as shown in FIG. 10, it is possible to express a musical score having different line thicknesses in the vertical direction and the horizontal direction without tracing, as shown in FIG. Some notation software has handwritten fonts that can display symbols with different line thicknesses in the vertical and horizontal directions, and this also allows the display of musical scores with different line thicknesses in the vertical and horizontal directions. Handwritten music with different line thickness in vertical and horizontal directions is often found in jazz music.

  Since the handwritten score is handwritten to the last, there is a blurring between the notifiers, and there is a blurring in the score by the same notifier. In particular, handwritten musical scores written with calligraphic pens that have different line thicknesses in the vertical and horizontal directions have a certain level of writing style. The blurring of the symbols is larger than that of ordinary handwritten music (handwritten music with the same line thickness in the vertical and horizontal directions).

  2. Description of the Related Art Conventionally, there are musical score recognition apparatuses that recognize musical score symbols from musical score image data obtained by reading a musical score and generate performance data and musical score data for displaying musical scores based on the recognized musical score symbols.

  Patent Document 1 describes a score recognition device that detects staffs and paragraphs from score image data, determines a processing rectangle based on the staff and paragraph, and recognizes various score symbols within the processing rectangle.

  Patent Document 2 describes a score recognition device that detects a fine line symbol from a fine line image portion of musical score image data, detects a thick symbol from other image portions, and recognizes a note using the detection result. Has been.

  In Patent Document 3, a thin line symbol is detected from the thin line image portion of the musical score image data, and a black ball head, a continuous symbol, and a thick symbol of a flag are detected from other image portions, and based on these detection results. A musical score recognition device for detecting musical notes is described.

Patent Document 4 recognizes a constituent symbol of a score inputted by handwriting and replaces the display by the read graphic data with the display by the display data of a predetermined score constituent symbol, thereby enabling handwriting input of the score. In addition, there is described a musical score display device that can display a musical score by a predetermined musical score component symbol.
Japanese Patent Laid-Open No. 9-97058 (Japanese Patent No. 3613356) JP-A-9-97060 (Patent No. 3445039) JP 2003-242439 A No. 7-15024

  As described above, since the score includes not only the printed score but also the handwritten score, it is desirable that the score recognition apparatus can recognize the score symbol in the handwritten score together with the printed score. For this reason, it is conceivable to configure the recognition means so as to cover all the inconsistencies in the score and recognize the symbols. However, in order to do this, it is possible to relax the threshold for dictionary matching, the symbol parts constituting symbols and the threshold for connection between symbols, or notation that exists only in handwritten scores, not in printed scores. In this case, there is a problem that the recognition rate in the recognition of the printed musical score is lowered.

  In printed music, it is required to reduce the number of erroneous symbol recognition as much as possible, while in handwritten music, it is required to recognize symbols in various styles (handwriting) as widely as possible. In order to satisfy this requirement, it is possible to provide a printed score recognition mode for recognizing a printed score and a handwritten score recognition mode for recognizing a handwritten score, and it is possible to selectively switch between these modes. In this case, the user needs to identify whether the score is a printed score or a handwritten score and select a mode accordingly. However, there are cases where it is difficult for the user to distinguish between a printed score and a handwritten score. It is also possible to actually perform the recognition processing in each mode, compare the recognition rates of the two, and select the mode with the higher recognition rate. However, requesting the user for these operations has a problem of increasing the burden on the user.

  It is also possible to recognize the score as a printed score and a handwritten score, compare the recognition results of both, and automatically determine whether to recognize the score as a printed score or as a handwritten score according to the comparison result. However, in this case, since it is necessary to recognize the same score as a printed score and a handwritten score, there is a problem that the processing cost increases.

  In the score recognition devices of Patent Documents 1 to 3 and the score display device of Patent Document 4, it is considered to efficiently automatically determine whether a score is a handwritten score or a printed score, and to perform score recognition in an optimal mode accordingly. Not.

  An object of the present invention is to provide a score recognition apparatus and a program capable of appropriately recognizing a score symbol from score image data obtained by reading a handwritten score or a printed score.

  In order to solve the above-described problems, the present invention provides a musical score recognition apparatus for recognizing a symbol from musical score image data and generating musical score data based on the recognized symbol. A paragraph recognizing unit and an in-paragraph part re-recognizing unit, wherein the in-paragraph part re-recognizing unit recognizes the in-paragraph part recognized by the staff / paragraph recognizing unit as a printed score; A handwritten score recognition mode for recognizing as a score is provided. First, handwritten score determination information indicating whether the score is estimated as a handwritten score is calculated while recognizing a score as a printed score recognition mode, and based on the handwritten score determination information The score is estimated as a handwritten score, and if it is determined as a handwritten score, the score is automatically switched to the handwritten score recognition mode. There is a first feature in that recognition.

  In the present invention, the re-recognizing means in the paragraph further comprises a re-printed score recognition mode for re-recognizing the in-paragraph part as a printed score, and indicates whether the score is estimated as a printed score at the time of re-recognition. The determination information is calculated, and it is determined whether the score is estimated as a printed score based on the printed score determination information. When it is determined that the score is estimated as the printed score, the print score recognition mode is automatically switched again. The second feature is that the score is recognized again.

  In the present invention, the part re-recognition means in the paragraph determines that the score is estimated as a handwritten score when the calculated handwritten score determination information is equal to or greater than a predetermined threshold, and automatically enters the handwritten score recognition mode at this time. There is a third feature in that switching is performed automatically.

  Further, according to the present invention, the re-recognizing means in the paragraph part recognizes the number of symbols unique to the handwritten score, the number of symbols that have failed to be recognized in the matching process with the symbol recognition dictionary, and the measure calculated based on the recognized score. Handwritten score determination information based on at least one of the number of measures having different actual bar lengths calculated with reference to the performance length and time signature information, and the number of failed connection between the ellipse of the note head candidate and the straight line of the tail candidate There is a fourth feature in the point of calculation.

  The fifth aspect of the present invention is that the in-paragraph part re-recognizing means uses a recognition result obtained at the time of previous recognition at the time of re-recognition.

  In the present invention, the part re-recognizing means in the paragraph uses the recognition result of the score part obtained in the previous printed score recognition mode as it is when re-recognizing, and the re-recognition is performed from the score part after that. There is a sixth feature.

  The present invention further comprises a recognition result correcting means for correcting a recognition result by the in-paragraph part re-recognizing means, wherein the in-paragraph part re-recognizing means uses a recognition dictionary in the score recognition mode being executed. A recognition function for recognizing a symbol having the best matching degree as a candidate symbol and recognizing one or more symbols having the next highest matching degree as a candidate symbol. Based on the music notation style information, determine the validity of the symbol with the best matching, and if it is determined not to be valid, determine the validity while changing the symbol with the best matching to a candidate symbol, The seventh feature is that a symbol having the best matching degree is corrected for a candidate symbol determined to be appropriate.

  Furthermore, the present invention has an eighth feature in that the correction result obtained by the recognition result estimating means is used for learning a symbol recognition dictionary.

  The present invention can be realized not only as hardware but also as a program for causing a computer to realize each function.

  According to the first to seventh features of the present invention, a printed score or a handwritten score is automatically discriminated, and a score is recognized by automatically switching to a printed score recognition mode or a handwritten score recognition mode accordingly. Recognize musical notation symbols without imposing a burden.

  Further, according to the second feature, when it is estimated as a printed score at the time of recognition in the handwritten score printing mode, it is recognized again as a printed score, so that recognition can be performed more appropriately.

  Further, according to the third feature, it is possible to automatically determine whether the score is a printed score or a handwritten score without recognizing the end of the score, so that the print score recognition mode or the handwritten score recognition mode is automatically switched while preventing an increase in processing cost. Thus, the musical score symbol can be properly recognized.

  Further, according to the fourth feature, it is possible to obtain information effective for discriminating between a printed score and a handwritten score.

  Further, according to the fifth and sixth features, it is possible to recognize a musical score symbol efficiently and appropriately by using a recognition result at the time of previous recognition.

  Further, according to the seventh feature, the symbol recognized by the in-paragraph part re-recognition means can be corrected to a valid symbol.

  Furthermore, according to the eighth feature, the recognition rate in the subsequent recognition can be increased by using the correction result in the recognition result correction process for dictionary learning.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an embodiment of a score recognition apparatus according to the present invention. This apparatus is obtained by adding a scanner and a MIDI interface circuit to a general computer system such as a personal computer. The CPU 1 is a central processing unit that controls the entire score recognition apparatus based on a program stored in the ROM 2 or RAM 3. In addition, a timer circuit for interrupting the CPU 1 at a predetermined cycle set in advance is incorporated. The RAM 3 is used as an image data buffer, a work area, etc. in addition to a program area. A hard disk device (HDD) 4 and a floppy disk device (FDD) 5 store programs, image data, performance data, and the like. The CRT 6 displays video information output from the CRT interface circuit 7 based on the control of the CPU 1. Information input from the keyboard 8 is taken into the CPU 1 via the keyboard interface circuit 9. The printer 10 prints the print information output from the printer interface circuit 11 based on the control of the CPU 1.

  The scanner 12 optically scans a handwritten score or a printed score and converts it into binary or gray scale image data. As the scanner 12, an arbitrary type such as a flat bed type, a handy type, or a feeder type can be used. The musical score image data obtained from the scanner 12 is taken into the RAM 3 or the HDD 4 via the scanner interface circuit 13. The MIDI interface circuit 14 is a circuit that transmits and receives MIDI data to and from an external MIDI device such as a sound module. The bus 15 connects each circuit in the score recognition apparatus. In addition, a pointing device such as a mouse, a serial interface circuit such as RS232C, and the like may be provided.

  The CPU 1 also functions as a score recognition means for recognizing a score symbol from the score image data taken into the RAM 3 or the HDD 4 and generating score data based on the recognized score symbol. This score recognizing means has an intra-paragraph part re-recognition function. The intra-paragraph part re-recognition function first sets the mode to the print score recognition mode and estimates whether the score is a handwritten score while recognizing the score as a print score.

  For this purpose, information for estimating that the score is a handwritten score is accumulated while recognizing the score in the printed score recognition mode, and handwritten score determination information is calculated based on the accumulated value. Then, based on the calculated handwritten score determination information, it is determined whether the score is estimated as a handwritten score. If it is determined as a handwritten score, the mode is automatically switched to the handwritten score recognition mode, and the score is re-recognized as a handwritten score. .

  In this case, the musical score portion to be subjected to the estimation process for estimating whether the musical score is a handwritten musical score can be limited to a certain portion of the musical score. For example, a multi-page score is read as a plurality of images each including a plurality of parts, and handwritten score determination information can be calculated based on information accumulated from a plurality of parts from the head or one image. It is also possible to stop the estimation process when the handwritten score determination information calculated while recognizing the score exceeds a predetermined threshold, and not to perform the estimation process thereafter. Since the portion can be reduced, the processing cost can be reduced. As described above, the present invention can be realized as software (program), but can also be realized as hardware.

  FIG. 2 is a flowchart showing main processing of the CPU 1. When the score is recognized, the score is optically scanned by the scanner 12, and the score image data is taken into the RAM 3 (S201). The image is captured as a binary image, and an image quality smoothing process is performed as necessary to reduce blurring and dot noise.

  Next, staff recognition (S202) and paragraph recognition are performed (S203). In staff recognition, staff scanning start position detection and staff shift amount detection are performed, and in paragraph recognition, the staff in each image is searched for a pair of staffs whose left ends are substantially the same, and the staffs are black pixels. The paragraph is recognized by checking whether it is connected with. In subsequent S204, the recognition result of the paragraph is displayed and the user is checked whether or not the paragraph recognition result is correct, and it is determined whether or not it is OK. If this determination result is OK, the process proceeds directly to S26 and OK. If not, the process proceeds to S206 after correcting the paragraph recognition result (S205). Since the processing of S201 to S205 is described in detail in Patent Document 1, description thereof is omitted here.

  In S206, the in-paragraph part re-recognition process is executed. The process for re-recognizing the part in a paragraph will be described in detail later with reference to FIG. 3. The outline thereof is as follows. First, the mode is set as a print score recognition mode, and the score is recognized as a handwritten score while recognizing the score as a print score. Determine whether it is estimated. If it is determined that the score is a handwritten score, the mode is automatically set to the handwritten score recognition mode, and the score is re-recognized as a handwritten score.

  In the in-paragraph part re-recognition process (S206), a musical score symbol is recognized by performing a matching process using a symbol recognition dictionary and further performing symbol parts and symbol linking. In addition to the symbols of the printed music score, a symbol specific to the handwritten music score is added to the symbol recognition dictionary in addition to a flag indicating that it is a sample from the handwritten music score. In the matching process in the printed score recognition mode, the number of matching with a symbol unique to the handwritten score is calculated while obtaining the recognition result by performing the matching process with the symbol of the printed score, and the score is converted into the handwritten score based on this. Is estimated. In the matching process in the handwritten score recognition mode, symbols unique to the handwritten score are also used for recognition. In addition, a musical score symbol is recognized by relaxing a matching threshold value, a threshold value such as a symbol component or a symbol connection so that various handwriting can be recognized widely.

  In S207, the musical score image data is synthesized and displayed based on the result of the part re-recognition process in the paragraph, and the user is checked whether it is correct or not, and it is determined whether or not it is OK. If this determination result is OK, the process proceeds to S209 as it is, and if not, the user manually corrects the recognition result using a mouse, keyboard, etc. (S208). Thereafter, the process proceeds to S209. In S209, performance data is created based on the re-recognition process within the paragraph or the corrected recognition result. In this process, based on the recognized various symbols and note information, for example, MIDI file data in a known performance data format is generated.

  FIG. 3 is a flowchart showing the intra-paragraph part re-recognition process (S206). First, nRecogMode ← “0”, nImage ← “0”, and nPart ← “0” are set as initialization (S301), and part score recognition processing is executed (S302).

  Here, nRecogMode is a flag for instructing the print score recognition mode, the handwritten score recognition mode, and the print score recognition mode again. NRecogMode “0”, “1”, and “2” indicate the print score recognition mode, that is, the score. A mode for recognizing as a printed score, a mode for recognizing a handwritten score, that is, a mode for recognizing a score as a handwritten score, and a mode for recognizing a printed score again, that is, a mode for recognizing a score as a printed score are designated. Further, nImage and nPart respectively designate an image (0 to all images) and an in-image part (0 to all parts) to be recognized. FIG. 4 shows the relationship between a paragraph in an image and a part in a paragraph. Of course, a part may exist alone.

  In S301 and S302, first, the score of the first part of the first image is recognized in the printed score recognition mode. The recognition processing in S302 will be described later with reference to FIG. 5, but the technique described in Patent Document 1 and other techniques can be used. As described above, the symbol recognition dictionary used for recognition is added with a symbol indicating that it is a sample from a handwritten score and a symbol specific to the handwritten score in addition to the symbol of the printed score. . In the printed score recognition mode, information indicating whether the score is estimated as a handwritten score, such as the number of matching with a symbol unique to the handwritten score, is calculated while performing the matching process with the symbol of the printed score and obtaining the recognition result. And accumulate. However, information indicating whether or not the score is estimated as a handwritten score and the accumulated value thereof do not affect the recognition result in the printed score recognition mode.

  Next, nRecogMode is determined (S303). Initially, nRecogMode is “0”, and since it is determined to be the print score recognition mode, the process proceeds to S304, and nImageEndPrint ← nImage and nPartEndPrint ← Part. Each of nImageEndPrint and nPartEndPrint represents a musical score portion that has already been recognized in S302, and nImageEndPrint “0” and nPartEndPrint “0” indicate that the first part of the first image has been recognized.

  Next, handwritten score determination information is calculated based on a cumulative value of information indicating whether the score can be estimated as a handwritten score (S305).

  Information indicating whether it can be estimated as a handwritten score includes the number of recognized symbols unique to the handwritten score with a flag set, the number of symbols that failed to be recognized in the matching process with the symbol recognition dictionary (recognition failure symbols), Number of measures (measure length failure measure) with different measure length (measured length) and different actual measure length (measured actual measure length) with reference to time signature The number of unsuccessful connections between the ellipse of the head candidate and the straight line (vertical line) of the tail candidate, and the total number of symbols recognized as score symbols, such as when the handwritten score is recognized as a printed score and when the printed score is recognized as a printed score You can use information that makes a difference. The time signature information can be detected by a technique described in Patent Document 2, for example.

  If a symbol specific to a handwritten score is added to the symbol recognition dictionary in addition to a printed score, a handwritten score can be estimated by increasing the recognition count of the symbol. A typical handwriting style for a quarter rest is shown in FIG. Such symbols are given the same or individual handwritten symbol IDs.

  In addition, when a handwritten score is recognized in the printed score recognition mode, the number of recognition failure symbols is likely to increase. Therefore, it can be estimated as a handwritten score by increasing the number of recognition failure symbols. In this case, if the positions of the recognition failure symbols to be counted are limited, the influence of symbols not related to the score can be eliminated.

  In addition, when handwritten scores are recognized in the printed score recognition mode, the number of recognition failure symbols increases, and as a result, the measure length calculated from the notes, dotted information, and tuplet information recognized within the measure is time signature information. There is a high possibility that the actual bar length calculated by reference will be shorter. Therefore, it can be estimated as a handwritten score by counting the number of measures whose measure performance length is shorter than the actual measure length or by increasing the proportion of measures whose measure performance length is shorter than the actual measure length relative to the number of measures in the part. If the first or last measure is an incomplete measure, it is desirable to exclude that measure.

  In addition, when a handwritten score is recognized in the printed score recognition mode, there is a high possibility that the process of connecting the ellipse information of the note head candidate detected for note recognition and the vertical line information of the tail candidate to make a note will fail. . Therefore, it can be estimated that the score is a handwritten score by increasing the number of ellipses or vertical lines that failed to be connected to the note, or both. In this case, the number of failed connections may be detected after limiting the bars and positions or narrowing down the shape of the ellipse or vertical line to some extent.

  Each piece of information as described above can be used as handwritten score determination information, but a plurality of pieces of information can be combined to be used as handwritten score determination information. For example, a recognition failure symbol and a measure length failure measure are linked, and the number of recognition failure symbols in the measure length failure measure can be used as handwritten score determination information. Moreover, handwriting determination information can be calculated by integrating individual information. For example, handwritten score determination information can be calculated by multiplying each information by a weight (weight) according to its effectiveness and adding it, and further dividing by the number of parts provided for obtaining the individual information.

  Next, based on the handwritten score determination information calculated in S305, it is determined whether the score is estimated as a handwritten score (S306). The determination of whether or not the handwritten score is estimated is, for example, whether or not certain information is greater than or equal to a predetermined threshold, whether or not the total information obtained by weighting individual information is greater than or equal to a predetermined threshold, Whether or not a predetermined number of parts whose total information obtained by weighting information is equal to or greater than a predetermined threshold continues, and whether or not a predetermined number of parts whose total information obtained by weighting individual information is equal to or greater than a predetermined threshold are counted Or in their synthesis.

  If it is not determined as a handwritten score in S306, it is determined whether processing for all parts in the image has been completed (S307). If it is determined that the processing has not been completed, nPart is sequentially incremented for the same image (nImage "0") (S308) and return to S302. By repeating this process, parts in the same image are recognized one after another in the printed musical score recognition mode. If it is determined in S307 that processing for all parts in the image has been completed, it is determined whether processing for all images has been completed (S309). If it is determined that processing has not been completed, nImage ← nImage Set +1, nPart ← 0 (S310), and return to S302. By repeating this process, the following parts in the image are recognized one after another in the printed musical score recognition mode.

  Note that it is not necessary to determine whether the score is estimated as a handwritten score after recognition until the end of the score, and is limited to a certain part such as multiple parts from the first part of the first image while considering that it can be estimated without error. Or when it is estimated that the score is a handwritten score. If the determination is completed, the process of S305 is not performed after that, and it is simply recognized sequentially.

  When the recognition of all images is completed in the printed music score recognition mode, the recognition result up to that point is set as the final recognition result (S318).

  In the process of determining whether or not the score is a handwritten score, if it is determined that the score is a handwritten score in S306, nRecogMode ← “1” is set (S311), and nImage ← “0” and nPart ← “0” are set (S312). Return to S302.

  Since nRecogMode is set to “1” in S311 and nImage ← “0” and nPart ← “0” in S312, the part score recognition processing is re-recognized from the first part of the first image in the handwritten score recognition mode. .

  When recognizing several pages of a score as multiple images, there is no possibility that they are mixed in print and handwriting, but there are few such cases in practice, so here the processing cost due to the complexity of the processing In order to prevent this increase, all of the first part of the first image is re-recognized in the handwritten score recognition mode.

  Next, nRecogMode is determined in S303. Since nRecogMode is set to “1” in S311, the handwritten score recognition mode is determined here. In the handwritten score recognition mode, depending on the symbol, the recognition result at the previous recognition can be used as it is as the current recognition result. For example, information such as staff and bar lines, and information such as beam thickness and tuplet inclination are considered to be the same as in the previous recognition, so that the recognition result in the previous recognition can be used as it is in the re-recognition. When the recognition of the part is completed, the current recognition result can be corrected using the recognition result recognized at the previous recognition. For example, when the bar performance length is worse than the recognition at the time of the previous recognition, the symbols in the bar can be corrected according to the recognition result at the time of the previous recognition.

  Although the recognition result obtained as described above may be used as the final score recognition result, there is still a possibility that it is erroneously determined as a handwritten score. Therefore, in the present embodiment, the recognition result is further recognized in the handwritten score recognition mode. Print score determination information is calculated from the recognition result, and it is determined whether the score is estimated as a print score based on the calculated print score determination information. If it is determined that the score is a printed score, the score is again switched to the printed score recognition mode so that the score is recognized again as a printed score.

  Therefore, as in S305, information indicating whether the score can be estimated as a printed score from the recognition result in the handwriting recognition mode is calculated and accumulated, and the printed score determination information is calculated based on the accumulated value (S313). . Information indicating whether the score can be estimated as a printed score may be the same as in S305. However, the printed score determination information is appropriately weighted differently from the handwritten score determination information in accordance with the effectiveness of each piece of information, and information that is not valid in determining whether the score is a subsequent handwritten score is removed. In addition, information that is valid in the subsequent determination may be newly added.

  Next, it is determined whether the score is estimated as a printed score based on the calculated printed score determination information (S314). This determination is the same as the determination of whether or not the score is a printed score at the time of reprocessing, but the determination of whether or not the score is a printed score here is whether it has increased compared to the determination information calculated at the time of previous recognition. That is, whether the score recognition rate has deteriorated.

  Whether or not it is a printed score, for example, whether or not certain information has risen above a predetermined threshold, whether or not the total information obtained by weighting individual information has risen above a predetermined threshold, Whether the total information obtained exceeds a predetermined threshold continues for a predetermined number of times, whether the total information obtained by weighting individual information exceeds a predetermined threshold is counted a predetermined number, or the total It can be carried out.

  If it is not determined as a printed score in S314, it is determined whether processing for all parts in the image has been completed (S307). If it is determined that the processing has not been completed, nPart is sequentially incremented for the same image (nImage "0") (S308) and return to S302. By repeating this process, parts in the same image are recognized one after another in the handwritten score recognition mode. If it is determined in S307 that processing for all parts in the image has been completed, it is determined whether processing for all images has been completed (S309). If it is determined that processing has not been completed, nImage ← nImage Set +1, nPart ← 0 (S310), and return to S302. By repeating this process, the subsequent parts in the image are recognized one after another in the handwritten score recognition mode.

  When the recognition of all images is completed in the printed score recognition mode as described above, the recognition result so far in the handwriting recognition mode is set as the final recognition result (S318).

  If it is determined in step S314 that the score is a printed score in the process of determining whether it is a printed score, nRecogMode ← “2” is set (S315), and nImage ← nImageEndPrint + 1 and nPart ← nPartEndPrint + 1 are set (S316). . From nImageEndPrint and nPartEndPrint, you can see the parts that have already been recognized in the print score recognition mode. In S317, the recognized part is restored based on the recognition result that has already been recognized and saved.

  Thereafter, the process returns to S302. Since nRecogMode ← 2 in S315 and nImage ← nImageEndPrint + 1 and nPart ← nPartEndPrint + 1 in S316, the part score recognition process starts again from the part part that follows the already recognized part. Re-recognize in recognition mode. At the time of re-recognition, as in the case of re-recognition, the recognition result at the previous recognition can be used as it is, or the recognition result at this time can be corrected by using it.

  In the printed score recognition mode again, it is only necessary to recognize the score as a printed score without determining whether the score is a printed score or a handwritten score. The recognition result recognized in this way is used as the final recognition result (S318).

  FIG. 6 is a flowchart showing the part score recognition process (S302). First, a processing rectangle is determined in S601. Here, a rectangle that is somewhat wide including the staff obtained in S202 (the staff in the staff in the case of a grand staff) is taken, and this is used as a recognition rectangle. In this case, the rectangle was changed from the calculated staff to a rectangle having a certain width in consideration of the shift amount of the staff, and a space with a certain width in the vertical direction was seen from the staff side. In this case, the rectangle is reduced.

  In S602, the staff inclination is corrected. The outline of this process will be described. Based on the staff shift amount obtained in S202, the vertical pixel row of the rectangular image is shifted up and down. In the case of a grand staff, there are multiple staffs, so the shift amount of the top staff or the average of the shift amounts is adopted. After that, if the rectangle overlaps the upper and lower staffs, delete that part, and if there is a label that touches the upper and lower ends of the rectangle and does not reach the recognized staff part, Erase as a staff label. After performing this processing, the rectangle may be further reduced from the end of the rectangle to the portion where the black pixel exists. Here, the label means a set of pixels that can be regarded as one lump in the image. In the example shown in FIG. 8, “m” and “f” are labels.

  In S603 to S608, various symbols are recognized. The musical score symbol includes a clef, a time signature, a note, a fixed symbol, a character string, a slur, a tie, and other symbols, which are sequentially recognized in S603 to S608. Since the recognition processing of various musical score symbols is described in detail in Patent Documents 1 and 2, description thereof is omitted here. In the printed score recognition mode, the recognition result is obtained by performing matching processing with the symbols of the printed score, while recognizing the information that is the basis of the handwritten score determination information within a range that does not affect the recognition result. As described above, when re-recognizing in the recognition mode, the information that is the basis of the printed musical score determination information is recognized.

  After recognizing various musical notation symbols in S603 to S608, symbol parent-child relationship / musical score layout processing (S609) is performed and a recognition result is obtained (S610).

  Although the embodiment has been described above, the present invention is not limited to the above embodiment, and various modifications are possible. For example, it is possible to add a recognition result estimation process for correcting the recognition result obtained by the intra-paragraph part re-recognition process. This recognition result estimation process is particularly effective when applied to the recognition result in the handwritten score recognition mode. However, the recognition result estimation process uses the recognition result that is finally transmitted from the intra-paragraph part re-recognition process as a processing target so that the correction does not affect the handwritten score determination information and the printed score determination information.

  FIG. 7 is a flowchart showing recognition result estimation processing. When performing this estimation process, in the part score recognition process (S302), not only the symbol with the best matching degree, but also the one or more symbols with the next highest matching degree, and the matching degree even when the recognition fails. Recognize relatively good symbols as candidate symbols.

  The recognition result estimation process includes bar length validity detection (S701), rest estimation (S702), note estimation (S703), rhythm estimation (S704), and bar length unrelated FAIL symbol estimation (S705).

  In bar length validity detection (S701), in order to estimate bar length related FAIL symbols such as rests, notes, rhythms, etc., the bar length calculated based on the recognized symbols is valid. To detect.

  For example, using a tool that calculates the total playing time (measure performance length) of notes and rests in a measure, add the note and rest performance lengths (gate time) and add the notes in the measure. And the rest playing length is processed appropriately using information such as dotted and tuplet, and all rests and long rests are processed appropriately. Calculate Also, the actual length of the bar (actual bar length) is obtained from the time signature information in the score. When the measure performance length is shorter than the actual measure length, it is assumed that the measure performance length is not appropriate.

  Note that the time signature information is recognized by the sound part / time signature recognition (S603 in FIG. 6) of the in-part score recognition process. If a time signature exists in a measure other than the beginning of the song, the time signature is also stored in correspondence with the measure.

  In rest estimation (S702), if the measure performance length is not valid in S701, the rest of the score is estimated to have failed dictionary matching, and rest estimation is performed. The symbol that is appropriate for changing to a rest, which is in a valid position as a rest, is temporarily regarded as a recognition failure symbol (FAIL symbol), which is regarded as a rest, and the measure playing length is calculated. A reasonable position as a rest is, for example, limited between the first line and the fifth line of the staff. If the calculated measure performance length is the same as or close to the actual measure length, the FAIL symbol is corrected to rest as it is appropriate to change the FAIL symbol to rest.

  Symbols that are reasonable to change to rest are, for example, (1) a symbol where the candidate symbol after the next point is a rest, (2) a symbol where the candidate symbol after the next point is a rest with a FAIL symbol, (3) A simple symbol whose shape is close to a rest, (3) A FAIL symbol and a candidate symbol whose shape after the next point is close to a rest, (4) A simple label (pixels that can be regarded as a lump in the image) The symbol that has the highest recognition rate or the order of the symbols to be changed is selected from these symbols. Set it appropriately.

  An accent or the like exists as a symbol whose shape is close to a rest, but the accent can be distinguished on the condition that a parent note exists.

  If the bar performance length is the same as the actual bar length as a result of calculating the bar performance length by regarding the FAIL symbol as a rest, it is corrected to a rest that estimates the FAIL symbol as the rest estimation is valid. When the measure performance length is larger than the actual measure length, the rest estimation is invalid and the estimation is returned to the state before estimation. Further, when the measure performance length is still smaller than the actual measure length, it may be invalid because the rest estimation is not valid, but it may be effective if it is as close as possible to the actual measure length. In rest estimation, the judgment conditions can be changed in consideration of the importance of FAIL symbols, the probability of failure, and the effects of incorrect estimation.

  The note estimation (S703) is an estimation when a note is recognized by matching processing with a symbol recognition dictionary, and estimates a note-related symbol in the same manner as rest estimation.

  Note that the note matching process can be performed as follows, for example. In the handwritten score, the note shape changes greatly. In note recognition by dictionary matching, it is possible to cope with this by changing the symbol component shape threshold value and the symbol component connection threshold value, which is effective in improving the recognition rate. In this case, matching symbols are limited to notes having no consecutive digits or chords, and a dictionary is constructed and recognized as separate symbol IDs depending on the direction of the tail, the number of lines, and the number of flags. The recognized note symbol calculates the ball position, the number of lines and the number of flags from the symbol ID, and changes it to an actual note. This makes it possible to recognize a note even when the note cannot be recognized based on detection of an ellipse or a straight line. However, with such musical note symbols, the number of additional lines may be misrecognized. Therefore, a process for correcting this is added.

  In this correction processing, it is possible to use the horizontal line obtained for estimation of the added line position by the pixel histogram in the horizontal direction or for the note detection. In addition, although the pitch of other recognized notes and estimation from the staff position are also added, in the case of handwritten music, the width of the staff is often different from the width of the staff, and the heights of the notes are also different. Since there may be differences, it is preferable to lower the priority of these modifications.

  In rhythm estimation (S704), if the measure length is the same as the actual measure length, the rhythm pattern of the measure is saved as a rhythm pattern dictionary, and if the measure performance length is different from the actual measure length, The pattern is compared with the rhythm pattern dictionary, and the FAIL symbol is corrected to another symbol so that the closest rhythm pattern is obtained. Here, the technique proposed in Japanese Patent Application No. 2005-311788 by the present applicant can be used. Although the detailed explanation of this technique is omitted, the outline of the technique is to calculate the length of the bar that passes when it is played, and this means that a bar whose length does not correspond to the time signature on the staff is misidentified. The pattern is set as a misidentified rhythm pattern, a similar rhythm pattern similar to the misidentified rhythm pattern is searched from a plurality of rhythm patterns in the dictionary database, and the misidentified rhythm pattern is changed to a similar rhythm pattern. In addition to this technique, a rhythm pattern that is the same when the FAIL symbol is changed to another symbol is selected and corrected to this preferentially.

  For example, bar data when the FAIL symbol is changed to a candidate symbol for the next point is created, and further, pre-processing for performance data creation, such as processing for detecting dotted notes and tuplet parent notes, is performed on the data within the bar. Next, a rhythm pattern is detected from the preprocessed measure data. If the same rhythm pattern as this rhythm pattern exists in the rhythm pattern dictionary, the FAIL symbol is corrected to the next candidate symbol and becomes a formal symbol.

  FAIL symbols related to measure lengths other than rests, notes, and rhythms, such as tuplet numbers, can also be estimated. Of the FAIL symbols, if the candidate symbol for the next matching point is a tuplet number, convert it to a normal symbol, search for a tuplet number parent note or rest, then calculate the measure performance length, The calculated measure performance length is compared with the actual measure length, and the estimation is enabled or disabled according to the result. In the estimation of symbols related to measure lengths other than rests, notes, and rhythms, as with rest estimation, the importance of symbols to be estimated, the probability of failure, and the effects of incorrect estimation are considered. You can change the conditions.

  In FAIL symbol estimation not related to measure length (S705), a FAIL symbol not related to measure length is estimated. FAIL symbols that are appropriate to change to normal symbols are, for example, (1) FAIL symbols that exist next to notes without a temporary symbol, and the next candidate symbol is a temporary symbol, and (2) below the staff. The next candidate symbol on the side is the FAIL symbol of the pedal system, and (3) the next candidate symbol that is above or below the note without accent, tenuto, fermata, etc. is the accent, tenuto, fermata, etc. FAIL symbols, (4) FAIL symbols in the staff at the left end of the staff, where the next candidate symbol is a clef or time signature. Even in this estimation, as in the case of rest estimation, the determination condition can be changed in consideration of the importance of the symbol to be estimated, the probability of failure, the influence of incorrect estimation, and the like.

  Further, by using the result of the recognition result estimation process for dictionary learning, the recognition rate in the subsequent recognition of handwritten musical scores can be increased. When the rest estimation result is used for dictionary learning, the feature amount of the FAIL symbol (recognition source symbol) estimated to be a rest is calculated, and this feature amount is added to the symbol recognition dictionary as data for the rest. . This makes it possible to recognize symbols that have been regarded as FAIL symbols and similar symbols from the beginning as rests. It is also possible to use the results of note estimation, rhythm estimation, and measure length unrelated FAIL symbol estimation for dictionary learning.

  However, it is desirable that the conditions for adding dictionary data by dictionary learning are strict. For example, in the case of a rest, if the dictionary learning is performed using a symbol that is not actually a rest as a rest, the recognition rate in the subsequent recognition deteriorates. In consideration of this point, the determination as to whether or not to learn the dictionary is made strictly under strict conditions. For example, the conditions include (1) rectangular shape of recognition source label, (2) position of recognition source label, (3) probability of dot presence of recognition source label, (4) matching degree as rest, (5) Only when the measure length is equal to the actual measure length.

  Also, depending on the type of symbol, there are things that should not be learned and those that should be learned a lot, so it is also preferable to limit the number of dictionary learning for one symbol. In addition, the learning result may be valid at the next musical score recognition, or may be discarded for each musical score recognition so that an unexpected learning error result is not carried over.

  Also, image components such as ellipses, vertical thin lines, horizontal thin lines, horizontal thick lines, and holes (holes) in images that are detected during note recognition, depending on whether estimation is enabled and dictionary learning is performed. You can also use this information. For example, information on vertical lines and holes can be effectively used for temporary symbols.

  Furthermore, the recognition result estimation is particularly effective for recognizing a handwritten score that has a relatively large written variation. However, the recognition result may be estimated by recognition in a printed score recognition mode.

  Also, for example, depending on the score, it may not be possible to properly determine whether the score is a handwritten score or a printed score, and the recognition mode may not be automatically switched appropriately. An interface that can forcibly set the recognition mode can be provided to cope with such a case.

It is a block diagram which shows the structure of one Example of the score recognition apparatus which concerns on this invention. It is a flowchart which shows the main process of CPU. It is a flowchart which shows the part re-recognition process in a paragraph. It is a figure which shows the relationship between the paragraph in an image, and the part in a paragraph. It is a figure which shows the typical handwriting style of a quarter rest. It is a flowchart which shows a part score recognition process. It is a flowchart which shows a recognition result estimation process. It is a figure which shows the example of the label in a score. It is a figure which shows the example of a handwritten score. It is a figure which shows the other example of a handwritten score.

Explanation of symbols

1 ... CPU, 2 ... ROM, 3 ... RAM, 4 ... Hard disk drive (HDD), 5 ... Floppy disk drive (FDD), 6 ... CRT, 7 ... CRT Interface circuit 8 ... Keyboard 9 ... Keyboard interface circuit 10 ... Printer 11 ... Printer interface circuit 12 ... Scanner 13 ... Scanner interface circuit 14 ... MIDI Interface circuit, 15 ... Bus

Claims (16)

In a score recognition device that recognizes a symbol from score image data and generates score data based on the recognized symbol,
Staff / paragraph recognition means for recognizing staff and paragraph from musical score image data;
It has means for re-recognizing the part in the paragraph,
The in-paragraph part re-recognition means includes a printed score recognition mode for recognizing a part in a paragraph recognized by the staff / paragraph recognition means as a printed score and a handwritten score recognition mode for recognizing as a handwritten score. Handwritten score determination information indicating whether the score is estimated as a handwritten score while recognizing a score as a recognition mode is calculated, and it is determined whether the score is estimated as a handwritten score based on the handwritten score determination information. A score recognition apparatus, wherein when it is determined that a score is estimated, the score is re-recognized by automatically switching to a handwritten score recognition mode.   The in-paragraph part re-recognizing means further comprises a re-printed score recognition mode for re-recognizing the in-paragraph part as a printed score, and calculates printed score determination information indicating whether the score is estimated as a printed score at the time of re-recognition, It is determined whether or not the score is estimated as a printed score based on the printed score determination information, and when it is determined that the score is estimated as a printed score, the score is automatically switched again to the printed score recognition mode and the score is recognized again. The musical score recognition apparatus according to claim 1.   The in-paragraph part re-recognition means determines that a score is estimated as a handwritten score when the calculated handwritten score determination information is equal to or greater than a predetermined threshold, and automatically switches to a handwritten score recognition mode at this point. The musical score recognition apparatus according to claim 1 or 2.   The intra-paragraph part re-recognition means includes the number of recognized symbols unique to the handwritten score, the number of symbols that failed to be recognized in the matching process with the dictionary for symbol recognition, and the bar performance length and time signature information calculated based on the recognized score. The handwritten score determination information is calculated based on at least one of the number of measures having different reference actual lengths calculated by reference and the number of connection failures between the ellipse of the note head candidate and the straight line of the tail candidate. The musical score recognition apparatus according to any one of claims 1 to 3.   5. The score recognition apparatus according to claim 1, wherein the re-recognizing means in the paragraph uses a recognition result obtained at the previous recognition at the time of re-recognition.   3. The intra-paragraph part re-recognition means uses the result of recognizing a musical score portion obtained in the previous printed musical score recognition mode at the time of re-recognition, and re-recognition is performed from the subsequent musical score portion. The musical score recognition apparatus described in 1.   And a recognition result estimating means for correcting a recognition result by the intra-paragraph part re-recognizing means, wherein the intra-paragraph part re-recognizing means executes a matching process using a recognition dictionary in the musical score recognition mode being executed. And a recognition function for recognizing one or a plurality of symbols having the best matching degree as a candidate symbol, and the recognition result estimating means is based on score notation style information. In addition, the validity of the symbol with the best matching degree is judged, and when it is judged to be invalid, the validity is judged while changing the symbol with the best matching degree to the candidate symbol, and the matching degree is the best. 7. A musical score recognition apparatus according to claim 1, wherein the symbol is corrected to a candidate symbol determined to be valid.   8. The score recognition apparatus according to claim 7, wherein the correction result obtained by the recognition result estimation means is used for learning a symbol recognition dictionary. In a program for causing a computer to realize a function of recognizing a symbol from musical score image data and generating musical score data based on the recognized symbol,
Recognize staff and paragraph from score image data and re-recognize part in paragraph
The in-paragraph part re-recognition function includes a printed score recognition procedure for recognizing a part in a paragraph recognized by the recognition function as a printed score, and a handwritten score recognition procedure for recognizing as a handwritten score. Handwritten score determination information indicating whether the score is estimated as a handwritten score while recognizing the score is calculated, and whether the score is estimated as a handwritten score is determined based on the handwritten score determination information, and is estimated as a handwritten score A program characterized by automatically switching to a handwritten score recognition procedure and re-recognizing a score when it is determined that the score is to be played.   The recognition function further includes a reprint score recognition procedure for re-recognizing a part in a paragraph as a print score, calculates print score determination information indicating whether a score is estimated as a print score at the time of re-recognition, and the print score It is determined whether the score is estimated as a printed score based on the determination information, and when it is determined that the score is estimated as a printed score, the score is recognized again by automatically switching to the printed score recognition procedure again. The program according to claim 9.   The in-paragraph part re-recognition function determines that a score is estimated as a handwritten score when the calculated handwritten score determination information exceeds a predetermined threshold, and automatically switches to a handwritten score recognition mode at this point. The program according to claim 9 or 10.   The part re-recognition function in the paragraph includes the number of recognitions of symbols peculiar to handwritten music scores, the number of symbols that have failed to be recognized by the matching process with the symbol recognition dictionary, and the bar performance length and time signature information calculated based on the recognized music scores. The handwritten score determination information is calculated based on at least one of the number of bars having different actual lengths calculated by reference and the number of failed connection between the ellipse of the note head candidate and the straight line of the tail candidate. The program according to any one of claims 9 to 11.   13. The program according to claim 9, wherein the intra-paragraph part re-recognition function uses a recognition result obtained at the time of previous recognition at the time of re-recognition.   11. The intra-paragraph part re-recognition function uses a result of recognizing a musical score portion obtained in the previous printed musical score recognition mode at the time of re-recognition, and re-recognition is performed from the subsequent musical score portion. The program described in.   And a recognition result estimation procedure for correcting a recognition result obtained by the intra-paragraph part re-recognition procedure, wherein the intra-paragraph part re-recognition procedure performs a matching process using a recognition dictionary in the ongoing score recognition procedure. Then, the symbol having the best matching degree is recognized and one or more symbols having the next highest matching degree are recognized as candidate symbols. The recognition result estimation procedure is based on the score notation style information. Determines the validity of the best symbol, and if it is not valid, changes the symbol with the best matching score to a candidate symbol and judges its validity, and determines that the symbol with the best matching score is valid. The program according to claim 9, wherein the program is corrected to the determined candidate symbol.   16. The program according to claim 15, wherein the correction result obtained by the recognition result estimation procedure is used for learning a symbol recognition dictionary.

Images