JP2682383B2 - Music score recognition device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention recognizes staffs, notes, symbols and their positions in a musical score from image data obtained by reading the musical score with an image scanner or the like, and based on the recognition result, generates musical tones. The present invention relates to a musical score recognizing device that generates and outputs information such as pitch, sounding timing, and sounding time.

[0002]

2. Description of the Related Art Score information is recognized from image data obtained by reading a score with an image scanner or the like, and a MIDI (Musical Instrument Digital Interface) is recognized from the recognition result.
ce) There is an attempt to create data and supply the created MIDI data to a MIDI tone generator, such as an electronic musical instrument, so that the processes from reading the score to performing automatically can be performed consistently.

The general processing flow of this type of system conventionally performed is as follows. (1) Import music score data. (2) Staff detection and recognition. (3) Clear staff. (4) Bar line detection and recognition. (5) Measure line deletion. (6) Note detection. (7) Note recognition. (8) Delete note. (9) Symbol detection. (10) Symbol recognition. (11) Symbol erasure. (12) Performance data creation. (13) Automatic performance (MIDI data creation / output).

Here, the staff detection / recognition is performed by detecting a line segment having a long and continuous interval in the horizontal direction based on a histogram of the number of pixels in the horizontal direction projected on the vertical axis. The vertical position of the detected staff is used as information for determining the pitch of the detected note. The detected staff is deleted so as not to cause any inconvenience in the note and symbol recognition process. The detected bar lines are used for recognizing a set of a plurality of staffs that play simultaneously. The note detection / recognition and the symbol detection / recognition processing are performed using a known pattern matching technique. Since the MIDI code can be created up to the symbol recognition in (10), the symbol elimination in (11) is not always necessary, but the symbol may be erased for convenience of processing.

[0005]

However, since the above-described conventional score recognition processing is premised on processing by a large computer, it involves complicated image processing, and if similar processing is performed by a general personal computer. ,
There is a problem that the processing time becomes long, and it is not practical. In addition, there are chords that generate multiple notes at the same time in the score, and the notes that make up the chords do not all have the same note length. There is also a problem that it is not easy to determine the actual pronunciation timing from the recognition result of.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and a music score that can efficiently recognize a music score with a small amount of calculation and that can accurately recognize an event on a music score and output a musical sound. It is an object to provide a recognition device.

[0007]

A musical score recognition apparatus according to the present invention recognizes staffs, notes, symbols and their positions in the musical score from image data obtained by reading an image of the musical score, In a musical score recognition device that generates information such as pitch of a musical tone, sounding timing and sounding time based on a recognition result, based on the recognition result of the staff, notes, symbols and their positions in the score, the musical notes Based on the pitch, the event occurrence order and the note length of the event consisting of the notes and symbols, and the coincidence information for associating the plurality of events to be simultaneously occurred, and the event occurrence order and the note length. The note length is the shortest among the plurality of events when the tone generation interval of the musical tone is determined and a plurality of events associated with the coincidence information are present. Based on the note length of the event, characterized in that a sound interval determination means for determining the sound interval.

When the event is a rest, the sounding interval determining means is to add the note length to the sounding interval.

[0009]

According to the present invention, the order of occurrence of events and the note length are obtained based on the recognition result of the staff, notes, symbols and their positions in the musical score, and the simultaneous occurrence of a plurality of events occurs simultaneously. If there are multiple events associated with the coincidence information when determining the interval of musical tone generation, the note length of the event with the shortest note length will be used as the note length. Since the tone generation interval is determined based on this, it is possible to determine the correct tone generation interval by a simple determination process. When the event is a rest, the tone generation timing can be easily matched by a very simple method of adding the note length to the tone generation interval.

[0010]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of a musical score recognition / automatic performance system according to an embodiment of the present invention. This musical score recognition / automatic performance system reads printed musical scores and the like to recognize staves, notes, symbols and their positions in the musical score, and based on the recognition result, determines the pitch, generation timing and sounding time of musical sounds. A music score recognition device 1 that generates performance data and generates and outputs MIDI data in accordance with the performance data, and M output from the music score recognition device 1.
An external MIDI sound source device 2 such as an electronic musical instrument that executes an automatic performance process according to the IDI data, and an output device 3 such as a speaker and an audio device driven by the external MIDI sound source device 2. Music score recognition device 1
Is composed of a computer system such as a personal computer or a workstation, and is connected to an image scanner 12 and a CPU 1 via a system bus 11.
3, ROM 14, RAM 15, timer 16, switch 1
7, display 18 and MIDI interface 19
It is composed of

The image scanner 12 is an image input device that optically reads an image of a musical score and takes binary image data composed of dot data into the musical score recognition device 1 as original image data. The CPU 13 executes a score recognition process according to a score recognition program stored in the ROM 14. The RAM 15 stores the binary image data captured by the image scanner 12 and provides a work area when executing the score recognition processing. Timer 16
Is based on the performance data obtained in the score recognition process.
The output timing of DI data is determined, and M
An interrupt for outputting IDI data is issued. The switch 17 and the display 18 are a man-machine interface between the musical score recognition device 1 and an operator. The musical score recognition device 1 outputs the generated MIDI data to the external MIDI sound source device 2 via the MIDI interface 19.

Next, the operation of the musical score recognition / automatic performance system configured as described above will be described. Figure 2 shows the score recognition and
It is a flowchart of an automatic performance process. The musical score recognition / automatic performance processing in this embodiment is roughly divided into the following four parts. (1) Preprocessing (staff / bar recognition, inclination correction, staff elimination and beam elimination). (2) Object recognition (circumscribed rectangle search and matching processing). (3) Event recognition processing (pitch recognition and pitch recognition processing) and performance data creation. (4) Automatic performance (MIDI data creation and output).

First, a musical score is read by the image scanner 12, and original image data of the musical score is stored in the RAM 15 (step S1). Now, when a musical score as shown in FIG. 3 is read by the image scanner 12, the musical score may be read while being inclined with respect to the X axis and the Y axis. In this case, FIG.
As shown in (1), the obtained original image data is inclined by a predetermined angle with respect to the X axis and the Y axis. The presence of such an inclination in the original image data has a great effect on the recognition processing. That is, regarding the inclination θ1 with respect to the X-axis, the profile of the projection (Y-axis projection of the number of dots) P is considerably reduced even with a slight inclination, as shown in FIG. 4, because the staff is long in the X-axis direction. As a result, the staff position cannot be detected. Further, the inclination θ2 with respect to the Y axis makes it difficult to perform matching processing in object recognition and recognition of chords to be generated simultaneously.

Therefore, in order to correct such an inclination with respect to the X axis and the Y axis, a staff recognition inclination correction processing is executed as a first stage of the preprocessing (step S2). FIG.
It is a flowchart which shows the detail of this staff recognition inclination correction processing. First, in order to remove an object that hinders recognition of a staff, a horizontal run in which dots are continuously arranged by a predetermined number in the horizontal direction (X-axis direction) is extracted from the original image data (step S11). . In order to effectively remove an object, the number of horizontal run dots to be detected needs to be set to a value longer than the general horizontal length of the object. FIG. 6 shows the result of performing the horizontal run extraction process on the original image data of FIG. Some image data other than the staff, such as a beam including a relatively long component in the horizontal direction, may remain, but since most of the objects are removed, there is no major problem.

Next, a staff is determined from the extracted horizontal component image data (step S12). That is, as shown in FIG. 6, the horizontal component image data is sequentially scanned in the Y-axis direction from the left end, and five horizontal lines with uniform intervals are detected as five lines. More specifically, as shown in FIG. 6, the intervals between horizontal runs detected by scanning in the Y-axis direction are d1, d2,
If d3 and d4, then Dm
When Ds (the sum of the deviation between the average interval and each interval) shown in the following Expression 2 satisfies less than 3 dots, the five horizontal runs are determined to be a staff. That is, if the variation of the interval between the horizontal runs is less than a predetermined value, it is regarded as a staff.

[0016]

Dm = (d1 + d2 + d3 + d4) / 4

[0017]

Ds = | Dm−d1 | + | Dm−d2 | + | Dm−d3 | + | Dm−d4 | <3 dots

At this time, a position indicated by a black dot in the figure that first satisfies the above conditions is regarded as a staff position, and the Y coordinate of each staff and the width Da from the top line to the bottom line of the staff are determined. It is stored in the RAM 15. When the resolution of the image scanner 12 is 360 dpi, the thickness of the staff is about 5 to 6 dots. In this case, the Y coordinate of the center of each staff is stored.

Next, as shown in FIG. 6, scanning in the Y-axis direction is carried out while being shifted little by little in the X-axis direction, and the inclination θ of the staff is calculated by the following equation 3 (step S13).

[0020]

３θ = tan-1 (x / y)

Here, x is the number of dots in the X-axis direction, and y is the number of dots in the Y-axis direction. Θ may be data of x dots in the X-axis direction and y dots in the Y-axis direction. Also,
In order to speed up the Y-axis scanning, as shown in FIG. 6, in the scanning after the position of the staff is detected, the range of the staff width Da is skipped. Scanning does not necessarily need to be performed for the entire range in the X-axis direction. For example, it is possible to sufficiently determine the inclination by performing, for example, only the range of 1/3 from the left. In this case, the processing speed can be further increased.

Next, the original image data is sequentially scanned, for example, in the X-axis direction in the same manner as the staff detection, and a line extending long in the Y-axis is detected as a bar line (step S14). For example, in order to play a treble portion (with treble clef) and a bass portion (with treble clef) in parallel, such as a piano score, if a plurality of staffs are connected to a common bar, 3 of line width Da
A Y-axis direction line having a length twice or more is regarded as a bar line. Such a condition may be specified by the switch 17 according to the type of the musical score. The X-coordinate and its length are stored in the RAM 15 as bar line information. The information on the position of the bar line can be used for checking when the time signature does not match in the determination of the sound length described later. The information on the bar length is used to determine a set of staffs to be played in parallel.

Subsequently, the inclination correction processing of the original image data is executed. The tilt correction process includes a Y-axis direction tilt correction process (step S15) and an X-axis direction tilt correction process (step S16). In the Y-axis direction inclination correction processing (step S15), based on the inclination angle θ obtained previously, as shown in FIG. 7A, each staff and its vicinity (the Y-axis direction involved in the performance). Range), the dots on the original image are shifted in the Y-axis direction for each block in the X-axis direction, and tilt correction in the Y-axis direction is performed for each staff and its periphery. The reason why the inclination correction is performed for each staff and its periphery is that, depending on the musical score, there are often cases where the inclination is different for each staff. In the X-axis direction tilt correction process (step S16), similarly, based on the tilt angle θ, as shown in FIG. Is shifted in units of dots in the direction to correct the inclination with respect to the Y axis. However, this X-axis direction inclination correction processing is performed by the inclination angle θ.
May be executed only when the angle is equal to or larger than a predetermined angle. The predetermined angle is set to a limit value that affects object recognition and chord detection in consideration of the resolution of the image scanner. This value may be changed according to the length of the bar line. Accordingly, in the case of a small inclination that does not affect recognition, the processing time can be reduced by omitting the inclination correction processing for the Y axis.
FIG. 8 shows an example of image data after the Y-axis direction inclination correction processing, and FIG. 9 shows an example of image data after the X-axis direction inclination correction processing. As shown in FIG. 9, the projection profile P 'in the X-axis direction after the inclination correction has a high peak value at the position of the staff.

Then, the X of the image data after the inclination correction is
An axial projection is obtained (step S17),
The Y coordinate of the center position in the Y-axis direction of each peak portion of the obtained projection is set as a formal staff position (step S18). At this time, the official staff interval D is also obtained at the same time. In addition to the method of obtaining the projection, the X-axis intermediate position is sampled so as to scan the intermediate position of the peak value indicated by the staff, and the Y coordinate satisfying the condition of Ds <3 dots is set to the official position of the staff. You may make it. Note that the image data in FIG. 9 also performs tilt correction in the X-axis direction. Therefore, when a projection in the Y-axis direction is obtained, an accurate X-axis position of the object can be determined at the time of object recognition described later. it can.

The above-described staff recognition inclination correction processing (step S)
After 2), only the staff portion is deleted from the image data after the inclination correction (step S3). The reason why the staff is erased in this step S3 is that inconvenience will occur if the staff remains in the object recognition described later. It should be noted that when the staff is deleted, as shown in FIG. 13A, processing must be performed so as not to delete the objects constituting the musical score.

The details of the staff clearing process will be described with reference to the flowchart shown in FIG. First, from the image data after the inclination correction, a series of pixel dots having a predetermined thickness corresponding to the average staff thickness and along the recognized staff position is discriminated (step S21). Next, while checking the thickness, when the predetermined thickness is not satisfied, the length of the series is discriminated (step S22).
If the sequence is determined to be, for example, a sequence of twice or more the staff interval D, the sequence is deleted (step S2).
3). That is, in the example of FIG. 11 in which A of FIG. 9 is enlarged and partly omitted, the thickness of the staff is about 5 to 6 dots. The sequence in the X-axis direction that falls within Dl is tracked, and at the time t when the thickness condition is no longer satisfied, it is determined whether or not the length Ll of the sequence is 2D or more. If you meet that condition,
With a slight margin M at both ends, the line segment (Ai area in the figure) between them is erased. As a result, as shown in FIG. 12, most of the staff is erased, and as shown in FIG. 13B in which the portion of FIG. The staff can be erased in a state where is completely left. As for the average staff thickness as a criterion for discrimination, an average range of staff thickness corresponding to the staff interval is calculated and stored in advance from a general musical score. Alternatively, the result of the staff recognition described above may be used.

When the staff erasing process (step S3) is completed, a beam recognition erasing process (step S4) is executed. Similarly to the case of the staff, the beam may be erroneously recognized at the time of object recognition as, for example, a chord of black circles (note heads) of chords as shown in FIG. Until the operation is performed, the image data is deleted. However, in the case of a beam, there are many forms that do not meet certain conditions, and it is impossible to completely recognize and determine the beam. Try to erase it. Even if the extreme form of the beam remains without being erased, it is preferable to the object being erased. As a condition of a beam in a typical form, as shown in FIG. 14, the thickness Db, the length Lb, and the angle θb satisfy the following conditions.

Thickness D / 3 <Db <D Length Lb ≧ 2D Angle −45 ° ≦ θb ≦ 45 °

FIG. 15 is a flowchart of the beam recognition erasing process. First, a series of dots that satisfies the condition of the thickness Db is discriminated from the image data from which the staff is deleted (step S31), and whether or not the series satisfies the condition of the length Lb is discriminated (step S32). If the sequence satisfies the two conditions, it is finally discriminated whether the sequence satisfies the condition of the angle θb (step S33). Then, a series of dots satisfying all the conditions is deleted (step S34).

When the beam recognition erasing process (Step S4) is completed, the object recognition process (Step S5) is performed.
Execute The details of this object recognition processing are shown in FIG.
This will be described with reference to the flowchart shown in FIG. First, since it is inefficient to perform the matching process on the entire surface of the image, in order to limit the range of performing the matching process, scan the image data from which the staff and the beam have been erased, and there is a lump of dots. A circumscribed rectangle is set for each range (step S41). For example, as shown in FIG. 17, the image from which the staff and the beam have been erased is sequentially scanned horizontally from the upper left, and if the image hits a dot at the indicated * point, the outline of the dot cluster (indicated by 塊) is obtained. Tracking in the direction of the arrow shown in the figure, the maximum value Xmax, Ymax and the minimum value Xmin, Ym of XY are obtained.
in is obtained, and a rectangle surrounded by these coordinate values is set as a circumscribed rectangle. Specifically, with the * point shown as a starting point, "up, down, left, and right, and if there is a dot, register that point,
The operation of "erasing the dot once" is repeated until the dot is not found by examining the top, bottom, left and right. In addition, in order to be able to restore the image from which the staff and the beam are erased at the time when the circumscribed rectangle of all the dot clusters is obtained, the image from which the staff and the beam are erased is obtained by circumscribing the rectangle. Either save it separately beforehand, or register all the positions of the dots once erased.

Next, the inside of the determined circumscribed rectangle is scanned with a reference pattern to evaluate the degree of matching (step S42). As shown in FIG. 18, the reference pattern as the matching template used in step S42 is a feature point that effectively captures the pattern features of various objects, that is, a point where a pixel dot should exist (black circle in the figure). And a distribution pattern of points (white circles in the figure) where pixel dots must not exist, and are registered in the ROM 14 in advance.
FIG. 18 shows reference patterns of sharp symbols (a) and (b), natural symbols (c), whole notes (d), half notes (e) and quarter notes (f). The sharp symbols (a) and (b) and the natural symbol (c) are quite similar, but the upper right and lower left portions are the points where pixel dots should exist in the former and the pixel dots exist in the latter. The difference is set so that they can be distinguished from each other. Although the whole note (d) and the half note (e) are quite similar, they can be identified by using a pattern as shown in FIG. In addition, the reference pattern is set to a pattern that does not determine the portion of the staff remaining unerased (the margin M left when the staff is erased). Also, as shown in FIG.
In contrast to the case (a), when the object is shifted by a semitone, the position of the staff disappearing is different, and no feature point is attached to the position where the staff may possibly disappear.
Even if part of the staff remains, it is not affected.

The reference pattern read from the ROM 14 is reduced / enlarged based on the interval between the staffs and subjected to pattern matching processing. By doing this, it is possible to handle a score image that is input at any resolution, and it is also possible to use an image whose object size is significantly different from normal, such as a score for children. Can respond. In this way, using the reference pattern corresponding to the size of the object, the inside of the circumscribed rectangle is scanned, and at each point, if they match, the evaluation value is set to “+1”. As shown in the figure, the evaluation value is set to “+0.9” if the eight directions are further checked and matched. This is to make it possible to cope with a slight deformation of the object shape. As a result of such evaluation, if the finally obtained evaluation value is 95% or more of the perfect matching, it is determined that the object corresponds to the reference pattern (step S43). For example, assuming that the reference pattern includes eight feature points, if the evaluation value is 7.6 (= 8 × 0.95) or more, it is determined that the object corresponds to the reference pattern. .

In the specific processing, step S4
In step S42, the object determination of step 3 and the matching evaluation of step S42 are linked, and “the pattern that has already been recognized is deleted from the image”. Is terminated, and the degree of matching is evaluated by applying the next reference pattern. " Also, in this method, it is difficult to recognize the dots and all rests, and therefore, they are recognized by capturing their geometric features. In other words, all rests are calculated by calculating the area that all rests of the score should have from the staff interval, and are applied to the rest, and are recognized by searching for a pattern with dense dots. Thus, recognition is performed based on the feature that the area is extremely small and the dispersion of dots is small. It should be noted that the information of each reference pattern also includes the information of the center coordinates thereof. At the time when the object is recognized, the coordinates in the image data of the center coordinates of the reference pattern are paired with the information of the recognized object in the RAM 15. be registered.

When the object recognition processing (step S5) is completed, an event recognition processing (step S6) is executed. FIG. 20 is a flowchart showing the event recognition process. First, the pitch of a musical note is recognized based on the staff information obtained by the staff recognition processing and the coordinate information of the object obtained by the object recognition processing (step S51). In step S51, music symbols for controlling pitches such as sharp, flat, and natural obtained in step S43, and white and black circles of notes (and bar lines if necessary) are collected in the X-axis direction. And sort them in chronological order, and apply each note to the staff one by one to determine the pitch. In step S51, the pitch is determined at a position obtained by multiplying the staff interval D by a factor equal to the staff line in principle. In consideration of this, the pitch is determined assuming that the staff interval D is spread by a coefficient such as 1.2 times.

Next, the degree of proximity of the X-coordinate of the note head center position of the note obtained by the object discrimination in step S43 is evaluated, and sounds to be sounded at the same time are grouped as chords (step S52). That is, as a form of a chord, FIG.
Various forms are conceivable, such as h1, h2 and h3 of 1. In the case of h2, X at the center of the note head of the note to be simultaneously sounded
The coordinates will be shifted by Dh. With this in mind,
When the interval Dh between the X-coordinates of the noteheads is within the interval D between the staffs, it is regarded as a chord. When the chord detection is performed more accurately, the intervals between the stems, which will be described later, may be evaluated.

Next, the position of the end of the vertical bar (stem) connected to the note head of the note is evaluated (step S53). This processing is necessary for determining the point at which the search is started when searching and counting the number of beams or flags.
That is, as shown in FIG. 22, in step S53,
The upper right and lower left ends of the note are searched, and the distance between these ends and the center position of the black or white circle of the note head (indicated by * in the figure) is calculated. Coordinates. In order to cope with the distortion of the stem, the search recursively performs dots in two directions from the notehead center position: upper left, upper, upper right, and right, and left, lower left, lower, and lower right. FIG.
If the note head is a white circle as in (b), tracking up, down, left, and right from the center point “*” and recursively starting from the point where the black dot is hit will make the printing of the score poor. Even if there is a search, the search can be performed reliably. If the note recognized as an object has a white circle and the difference between the distance from the upper right end to the center point and the distance from the upper left end to the center point is small, the entire note Treated as a note.

When the position of the stem end is detected, the beam or the flag is counted based on the stem end (step S54). That is, as shown in FIG. 23, when the position detected as the stem end is located at the upper right corner of the note head, it is located along the Y direction from the point swung left and right by D / 2 around the stem end “△”. Start tracking down. The number of pixel dots that are continuous in the tracking direction is counted, and this value
If the beam is within the margin of the beam width described in 31, the counting is performed, and in order to prevent the black circle of the chord from being counted as the beam, the tracking is stopped after a certain amount of blank space. On the other hand, when the detected end of the stem is the lower left point of the notehead, tracking starts in the Y direction and upward from a point swung left and right by D / 2 around the stem end. Next, the number of left and right beams counted as described above is compared, and the shorter sound length, that is, the larger number of beams is recognized as the sound length of the event. Therefore, FIG.
In the case of 3, the middle note to be searched is recognized as a sixteenth note with priority given to the two right beams.

Next, a dot detection process (step S55) is executed. For example, as shown in FIG. 22, if there is a dot group having a sufficiently small area and a small dispersion of pixel dots, a search is made in a rectangular range of a staff interval D horizontally and D / 3 vertically above and below the central point of the note head. It recognizes it as a dot attached to the note. Finally, the tone length is determined based on the information on the note recognized in step S43 and the information on the number of beams or flags obtained in steps S52 and S54. If there is a detected dot, a half note is added to the note length of the corresponding note to obtain a note length (step S5).
6). At this time, by using the previously obtained bar line position information, a final check of the note length is performed for each measure. An error such as non-existence can be output.

Thus, the event recognition result is:
The result of the music score recognition is stored in the RAM 15 as appropriate. For example, the recognition result of the musical score of FIG. 24 is as shown in FIG.
The recognition result includes an event number, an event type, a note length (the number of clocks when a quarter note length is 24 clocks), and a simultaneous (chord) flag. The event number is
A number identifying notes a to e, h and pause symbols f, g, etc.
The event type is information indicating a note or a rest, and in the case of a note, pitch information such as C4 and F3. When the coincidence flag is “1”, a chord is formed with the immediately preceding event information.

When the event recognition (step S6) is completed, performance data as intermediate data for creating MIDI data is created from the obtained recognition result (step S7). That is, as shown in FIG.
In order to control the IDI sound source device 2, a gate time (shorter than the note length at the output interval of the actual MIDI note-on data and note-off data) is calculated from the determined note length, and the note-on and note-off are calculated. It is necessary to determine the timing. It is also necessary to determine the interval from the note-on timing of the preceding event to the note-on timing of the next event, that is, the duration. Therefore, performance data including duration, note number, and gate time is generated from the recognition result shown in FIG.

FIG. 27 is a flowchart of performance data creation, and FIG. 28 is a diagram showing an example of performance data created by this processing. First, “0” is written to the duration as the generation timing of the first note event data, and the duration register (DUR) is set to “0”.
(Step S61). Next, as shown in FIG. 25, the event information pre-sorted in the X-axis direction is extracted in ascending order of the event number, and whether or not the simultaneous flag of the event following the event is “1”. To determine whether there is a simultaneous event (steps S62 and S63). If there is no simultaneous event, the note length is stored in the DUR as the duration up to the next event (step S65). If there is a simultaneous event, the shortest note length is stored in the DUR until the next event. Is stored in the DUR as the duration (step S65).

Subsequently, it is determined whether or not the event is a note (step S66). Only when the event is a note, a note number corresponding to the pitch is written and a note length ×
Write 0.8 as the gate time (step S6)
7, 68). This is because, in actual musical instrument performance, the actual sounding period of a note is often about 0.8 times the note length, except for slurs and staccato sounds. If there is a simultaneous event, note determination (step S66) and note number and gate time writing (step S6)
7, S68) is repeated (step S69). If there is no other simultaneous event, the contents of DUR are written as duration (step S71). At this time, if two written duration data continue,
That is, when the note number and the gate time are not written due to the presence of the rest, two consecutive duration data are added and rewritten to new duration data. After the performance data has been created for all events, end data is written last (step S
70, 72). Through this processing, performance data including a duration, a note number, and a gate time as shown in FIG. 28 is obtained.

Finally, based on the obtained performance data,
Automatic performance processing is executed by creating MIDI data and outputting it to the external MIDI tone generator 2 (step S8). Here, a clock of a fixed cycle corresponding to the tempo is counted, and when the duration of the duration included in the performance data has elapsed, a MIDI note-on message is output. In addition, the clock is counted from the time when the note-on is output, and when the gate time elapses,
Outputs a MIDI note-off message. These M
The IDI data includes, for example, a status byte indicating a note on / off and a performance channel, as shown in FIG.
It is composed of three bytes of velocity, which is information such as a note number indicating a pitch and a sound intensity. Of these, the velocity can be created from the recognition results when dynamics such as accents, crescendos, and decrescendos are also recognized, or when there is no such data in the recognized information. May be input and created through the switch 17 as appropriate.

In this way, the score can be accurately recognized by simple processing, so that the processing speed can be improved and the necessary memory capacity can be reduced, and the personal computer system can be used for practical use. It is possible to perform a recognizable music score. In the above embodiment, once the performance data is created from the music score recognition result, the MID
Although the I data is created, it may be directly converted to MIDI data from the recognition result. The method of obtaining the inclination of the staff is not limited to that of the embodiment, but the coordinates of the left end and the right end of the staff may be obtained, and the inclination may be obtained from the values. The correction of the tilt may be performed in the Y-axis direction after the correction in the X-axis direction.

[0045]

As described above, according to the present invention,
Based on the recognition results of the staff, notes, symbols and their positions in the score, the order of occurrence of events and the note length are determined, and multiple events that occur at the same time should be correlated with each other by the coincidence information. , When there is a plurality of events associated with the simultaneous occurrence information when determining the musical tone generation interval, of those events,
Since the sound generation interval is determined based on the code length of the event having the shortest note length, it is possible to determine the correct sound generation interval by a simple determination process. Further, when the event is a rest, it is possible to easily match the sound generation timing by a very simple method of adding the note length to the sound generation interval.

[Brief description of the drawings]

FIG. 1 is a block diagram of a music score recognition / automatic performance system according to an embodiment of the present invention.

FIG. 2 is a flowchart of the musical score recognition / automatic performance process.

FIG. 3 is a diagram showing an example of a musical score to be recognized.

FIG. 4 is a diagram showing original image data obtained by reading the musical score and a projection thereof.

FIG. 5 is a flowchart of a staff recognition inclination correction process.

FIG. 6 is a diagram showing a horizontal component extraction image in the graphic processing.

FIG. 7 is a diagram illustrating a method of tilt correction in the figure processing.

FIG. 8 shows image data after Y-axis direction tilt correction in the same process.

FIG. 9 is a diagram showing image data and projection after X-axis direction tilt correction in the same process.

FIG. 10 is a flowchart of the staff clearing process.

FIG. 11 is a diagram for explaining a method of discriminating thickness and length in the same process.

FIG. 12 is a diagram showing image data after the staff is erased in the same process.

FIG. 13 is a diagram showing a part of the same image data enlarged and shown together with a comparative example.

FIG. 14 is a diagram for explaining the beam recognition erasing process.

FIG. 15 is a flowchart of the same process.

FIG. 16 is a flowchart of the object recognition process.

FIG. 17 is a diagram illustrating a method of determining a circumscribed rectangle in the same process.

FIG. 18 is a diagram showing a reference pattern in the same processing.

FIG. 19 is a diagram showing a matching evaluation method in the same process.

FIG. 20 is a flowchart of the event recognition process.

FIG. 21 is a diagram showing a chord discrimination method in the same process.

FIG. 22 is a diagram showing a stem end discrimination and dot detection method in the same process.

FIG. 23 is a diagram showing a beam / flag counting method in the same process.

FIG. 24 is a diagram showing an example of a musical score recognized as the event.

FIG. 25 is a diagram showing a recognition result of the musical score.

FIG. 26 is a time chart showing the timing of driving an actual MIDI sound source based on the recognition result.

FIG. 27 is a flowchart of the performance data creation process.

FIG. 28 is a diagram showing performance data created in the same process.

FIG. 29 is a diagram showing MIDI data created from the performance data.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Score recognition device, 2 ... External MIDI sound source device, 3 ... Output device, 11 ... System bus, 12 ... Image scanner, 13 ... CPU, 14 ... ROM, 15 ... RAM, 16
... Timer, 17 ... Switch, 18 ... Display, 19
... MIDI interface.

Claims (2)

(57) [Claims] 1. A staff, a note, a symbol and their positions in the score are recognized from image data obtained by reading an image of the score, and based on the recognition result, a pitch of a musical tone, a sounding timing and In a musical score recognition device for generating information such as pronunciation time, based on the recognition result of the staff, notes, symbols and their positions in the musical score, the pitch of the notes, the event consisting of the notes and the symbols is generated. Event recognition means for obtaining sequence and note length, and simultaneous occurrence information that associates a plurality of the events to be simultaneously generated, and determining the sounding interval of the musical tones based on the sequence and the note length of the events, and the simultaneous occurrence When there are a plurality of events associated by information, the pronunciation interval is determined based on the note length of the event having the shortest note length among the plurality of events. Music recognition apparatus characterized by comprising a sound interval determination means that. 2. The musical score recognition according to claim 1, wherein, when the event is a rest, the pronunciation interval determination means adds the note length to the pronunciation interval. apparatus.