JPH06102869A - Score recognizing device - Google Patents

Abstract

PURPOSE:To efficiently recognize a score with a small calculation quantity and to accurately recognize events on the score and output a musical sound. CONSTITUTION:Five staffs are removed from image data obtained by reading an image of the score and connections of pixel dots with specific thickness corresponding to the thickness of a beam coupling plural notes are discriminated from the image data from which the five staffs are removed (S31). Then connections which are longer than specific length are discriminated among the connections of pixel dots with the specific thickness (S32) and connections which slant within a specific angle range are erased from the connections (S34). Then notes and symbols, and their positions, etc., are recognized from the image data from which the connections of pixel dots are erased.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention recognizes staffs, notes, symbols and their positions in a musical score from image data obtained by reading a musical score with an image scanner or the like, and based on the recognition result, produces a musical tone. The present invention relates to a musical score recognition device that generates and outputs information such as pitch, pronunciation timing, and pronunciation time.

[0002]

2. Description of the Related Art Musical score information is recognized from image data obtained by reading a musical score with an image scanner or the like, and a MIDI (Musical Instrument Digital Interfa
There is an attempt to consistently perform the processing from reading the score to the automatic performance by creating the MIDI data and supplying the created MIDI data to a MIDI tone generator such as an electronic musical instrument.

The general processing flow of this type of system that has been conventionally performed is as follows. (1) Import score data. (2) Staff detection / recognition. (3) Delete the staff. (4) Bar line detection / recognition. (5) Delete bar lines. (6) Note detection. (7) Note recognition. (8) Note deletion. (9) Symbol detection. (10) Symbol recognition. (11) Symbol elimination. (12) Performance data creation. (13) Automatic performance (MIDI data creation / output).

Here, the staff detection / recognition is carried out by detecting a long continuous line segment in the horizontal direction based on a histogram of the number of pixels in the horizontal direction projected on the vertical axis. The detected vertical position of the staff serves as information for determining the pitch of the detected note. The detected staff is deleted so as not to inconvenience the note and symbol recognition processing. The detected bar lines are used to recognize sets of staves that are playing at the same time. The note detection / recognition and the symbol detection / recognition processing are performed using a known pattern matching method. Since the MIDI code can be created up to the symbol recognition in (10), the symbol deletion in (11) is not always necessary, but the symbols may be deleted for convenience of processing.

[0005]

However, since the above-described conventional musical score recognition processing is premised on the processing by a large-scale computer, when the same processing is performed by a general personal computer, complicated image processing is involved. ,
There is a problem that the processing time becomes long and it cannot be put to practical use. On the other hand, in order to simplify the process, if you try to recognize a note or symbol by using a simple pattern recognition method with a small amount of calculation, the beam that connects the notes and the notes is erroneously recognized as the note head of the note. There is a problem that is.

The present invention has been made in view of the above circumstances, and can efficiently recognize a musical score with a small amount of calculation, and can accurately recognize an event on the musical score and output a musical sound. An object is to provide a recognition device.

[0007]

A musical score recognition apparatus according to the present invention recognizes staffs, notes, symbols and their positions in the musical score from image data obtained by reading an image of the musical score, In a musical score recognition device that generates information such as the pitch of a musical tone, sounding timing, and sounding time based on a recognition result, a staff erasing unit that deletes a staff from the image data, and a staff by this staff deleting unit. A thickness discriminating means for discriminating a series of pixel dots having a predetermined thickness corresponding to the thickness of a beam for combining a plurality of notes from the erased image data, and a predetermined thickness discriminated by the thickness discriminating means. Length discriminating means for discriminating a sequence having a length equal to or longer than a predetermined length among the sequence of pixel dots of, and the inclination of the sequence of pixel dots discriminated by the length discriminating means within a predetermined angular range. It is in And an erasing means for erasing the will,
It is characterized in that the erasing means recognizes notes, symbols and their positions from the image data in which the series of pixel dots has been erased.

[0008]

According to the present invention, it is possible to discriminate a series of pixel dots having a predetermined thickness, which is clearly a beam for combining a plurality of notes, from the image data in which the staff has been deleted, and to determine a predetermined thickness. A series of pixel dots having a predetermined length or more is discriminated from the series, and the series within the predetermined angle range among the series of the discriminated pixel dots is erased in consideration of the inclination of the beam. Therefore, it is possible to erase the beam part as well as the staff.

Further, according to the present invention, by deleting the staff and the beam, it is possible to leave notes, symbols and the like in an island shape and to limit the pattern matching area. Further, since the matching process is performed on the image data in which the beam is erased, it is possible to prevent the beam portion from being erroneously recognized as a note.
As a result, it becomes possible to efficiently recognize the score, and to accurately recognize the event on the score with a small calculation load.

[0010]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of a musical score recognition / automatic performance system according to an embodiment of the present invention. This music score recognition / automatic performance system reads printed music scores and recognizes staffs, notes, symbols and their positions in the music score, and based on the recognition results, musical pitches, generation timings, pronunciation times, etc. A musical score recognition apparatus 1 for generating performance data and generating and outputting MIDI data in accordance with the performance data, and an M output from the musical score recognition apparatus 1.
An external MIDI sound source device 2 such as an electronic musical instrument that executes automatic performance processing according to IDI data, and an output device 3 such as a speaker or an audio device driven by the external MIDI sound source device 2. Score recognition device 1
Is a computer system such as a personal computer, a workstation, etc., and an image scanner 12 and a CPU 1 which are mutually connected via a system bus 11.
3, ROM 14, RAM 15, timer 16, switch 1
7. Display 18 and MIDI interface 19
It consists of

The image scanner 12 is an image input device that optically reads an image of a musical score and takes binary image data consisting of dot data into the musical score recognition device 1 as original image data. The CPU 13 executes the score recognition processing according to the score recognition program stored in the ROM 14. The RAM 15 stores the binary image data captured by the image scanner 12 and also provides a work area for executing the score recognition process. Timer 16
Is an MI based on the performance data obtained by the score recognition processing.
The output timing of the DI data is determined and the CPU 13 receives the M
Interrupt for IDI data output. The switch 17 and the display 18 are man-machine interfaces between the musical score recognition apparatus 1 and the operator. The musical score recognition device 1 outputs the generated MIDI data to the external MIDI tone generator device 2 via the MIDI interface 19.

Next, the operation of the musical score recognition / automatic performance system thus constructed will be described. Figure 2 shows the score recognition
It is a flowchart of an automatic performance process. The score recognition / automatic performance process in this embodiment is roughly divided into the following four parts. (1) Pre-processing (Staff / bar recognition, tilt correction, staff erasing and beam erasing). (2) Object recognition (circumscribing rectangle search and matching processing). (3) Event recognition processing (pitch recognition and pitch recognition processing) and performance data creation. (4) Automatic performance (MIDI data creation and output).

First, the score is read by the image scanner 12, and the original image data of the score is stored in the RAM 15 (step S1). Now, when a score as shown in FIG. 3 is read by the image scanner 12, the score may be read with an inclination with respect to the X axis and the Y axis. In this case,
As shown in, the obtained original image data is tilted by a predetermined angle with respect to the X axis and the Y axis. The presence of such inclination in the original image data has a great influence on the recognition process. That is, with respect to the inclination θ1 with respect to the X-axis, the profile of the projection (Y-axis projection view of the number of dots) P becomes considerably large even with a slight inclination, because the staff is long in the X-axis direction. Because of the shape, the staff position cannot be detected. Further, the inclination θ2 with respect to the Y axis makes it difficult to perform matching processing in object recognition and recognition of chords to be generated simultaneously.

Therefore, in order to correct the inclination with respect to the X axis and the Y axis, the staff recognition inclination correction processing is executed as the first step of the preprocessing (step S2). Figure 5
It is a flowchart which shows the detail of this staff recognition inclination correction process. First, in order to remove an object that hinders recognition of staffs, a horizontal run in which a predetermined number of dots are continuously arranged in the horizontal direction (X-axis direction) is extracted from the original image data (step S11). . In order to effectively remove the object, the number of dots in the horizontal run to be detected should be set to a value longer than the general horizontal length of the object. FIG. 6 shows the result of horizontal run extraction processing performed on the original image data of FIG. A part of the image data other than the staff, such as a beam including a relatively long component in the horizontal direction, may remain, but since most of the object is removed, there is no big problem.

Next, the staff is discriminated from the extracted horizontal component image data (step S12). That is, as shown in FIG. 6, the horizontal component image data is sequentially scanned in the Y-axis direction from the left end, and five horizontal lines with even intervals are detected as five lines. More specifically, as shown in FIG. 6, the intervals between horizontal runs detected by Y-axis scanning are d1, d2,
If d3 and d4, then Dm shown in the following equation 1
When (average interval of each horizontal run) satisfies Ds (total of deviation between average interval and each interval) shown in the following Expression 2 is less than 3 dots, these five horizontal runs are determined to be staffs. That is, if the variation in the interval between the horizontal runs is less than a predetermined value, it is regarded as a staff.

[0016]

[Equation 1] Dm = (d1 + d2 + d3 + d4) / 4

[0017]

[Equation 2] Ds = | Dm-d1 | + | Dm-d2 | + | Dm-d3 | + | Dm-d4 | <3 dots

At this time, first, the positions indicated by black dots in the figure which satisfy the above conditions are regarded as the positions of the staff, and the Y coordinates of these staffs and the width Da from the upper end line to the lower end line of the staff are set. It is stored in the RAM 15. When the resolution of the image scanner 12 is 360 dpi, the staff has a thickness of about 5 to 6 dots. In that case, the Y coordinate of the center of each staff is stored.

Next, as shown in FIG. 6, scanning in the Y-axis direction is executed while gradually shifting in the X-axis direction, and the inclination θ of the staff is calculated by the following equation 3 (step S13).

[0020]

[Equation 3] ∠θ = tan-1 (x / y)

However, x is the number of dots in the X-axis direction, and y is the number of dots in the Y-axis direction. Note that θ may be data of y dots in the Y axis direction per x dots in the X axis direction. Also,
In order to speed up the Y-axis scanning, as shown in FIG. 6, in the scanning after the position of the staff is detected, the range of the width Da of the staff is skipped. The scanning does not necessarily have to be performed in the entire range in the X-axis direction, and the inclination can be sufficiently determined by performing the scanning in the range of 1/3 from the left. In this case, the processing speed can be further increased.

Next, the original image data is sequentially scanned, for example, in the X-axis direction by a method similar to the staff detection, and a line extending long in the Y-axis is detected as a bar line (step S14). For example, in the case of a piano score, where the treble part (with a treble clef) and the bass part (with a treble clef) are played in parallel, if multiple staves are connected to a common bar line, Line width Da 3
A Y-axis direction line having a length more than twice the length is regarded as a bar line. Such conditions may be specified by the switch 17 according to the type of score. The X coordinate and its length are stored in the RAM 15 as bar line information. The information on the position of the bar line can be used for checking when the beat does not match in the determination of the note length described later. Also, the bar length information is used to determine the set of staffs to be played in parallel.

Subsequently, the inclination correction processing of the original image data is executed. This tilt correction processing includes Y-axis direction tilt correction processing (step S15) and X-axis direction tilt correction processing (step S16). In the Y-axis direction inclination correction process (step S15), as shown in FIG. 7A, each staff and its vicinity (in the Y-axis direction involved in the performance, based on the previously obtained inclination angle θ). For each block in the X-axis direction, the dots on the original image are shifted in the Y-axis direction in dot units, and inclination correction is performed in the Y-axis direction for each staff and its periphery. The reason why the inclination correction is performed for each staff and its surroundings is that it is often found that the inclination angle is different for each staff depending on the score. Also in the X-axis direction tilt correction process (step S16), similarly, as shown in FIG. 7B, the dots on the original image are set to the X-axis direction on the basis of the tilt angle θ for each block continuous in the Y-axis direction. Direction is shifted dot by dot to correct the inclination with respect to the Y axis. However, this X-axis direction tilt correction processing is performed with the tilt angle θ.
However, it may be executed only when the angle is a predetermined angle or more. The predetermined angle is set to a limit value that affects object recognition and chord detection in consideration of the resolution of the image scanner. Further, this value may be changed according to the length of the bar line. As a result, in the case of a small inclination that does not affect the recognition, the inclination correction processing for the Y axis can be omitted and the processing time can be shortened.
FIG. 8 shows an example of image data after the Y-axis direction inclination correction processing, and FIG. 9 shows an example of image data after the X-axis direction inclination correction processing. As shown in FIG. 9, the projection profile P ′ in the X-axis direction after the inclination correction has a high peak value at the position of the staff.

Then, the X of the image data after the tilt correction is next performed.
Obtain the projection in the axial direction (step S17),
The Y coordinate of the center position in the Y axis direction of each peak portion of the obtained projection is set as the official position of the staff (step S18). At this time, the official staff interval D is also obtained at the same time. In addition to the method of obtaining the projection, the intermediate position of the X axis is sampled so that the intermediate position of the peak value indicated by the staff is scanned, and the Y coordinate satisfying the condition of Ds <3 dots is set to the official position of the staff. May be set. Since the image data of FIG. 9 is also subjected to inclination correction in the X-axis direction, if the projection in the Y-axis direction is obtained, the accurate X-axis direction position of the object may be determined at the time of object recognition described later. it can.

The above staff recognition inclination correction processing (step S
When 2) is completed, then, only the staff portion is deleted from the image data after the inclination correction (step S3). The reason why the staff is erased in step S3 is that inconvenience occurs if the staff remains during object recognition described later. What should be noted when erasing the staff is that, as shown in FIG. 13A, a process must be performed so as not to erase even the objects constituting the musical score.

The details of the staff erasing process will be described with reference to the flowchart shown in FIG. First, a series of pixel dots having a predetermined thickness corresponding to the average staff thickness and along the recognized staff position is discriminated from the image data after the inclination correction (step S21). Next, while checking the thickness, when the predetermined thickness is not satisfied, the length of the run is discriminated (step S22).
If it is determined that the run is, for example, twice or more the staff interval D, the run is erased (step S2).
3). That is, taking FIG. 11 in which A of FIG. 9 is enlarged and partly omitted, the staff has a thickness of about 5 to 6 dots. The continuous X-axis direction that falls within Dl is tracked, and at time t when the thickness condition is no longer satisfied, it is determined whether the continuous length Ll is 2D or more. If that condition is met,
A slight margin M is set at both ends, and the line segment (Ai area in the figure) between them is erased. As a result, most of the staffs are erased as shown in FIG. 12, and some staffs are left as shown in FIG. 13 (b) in which the portion of FIG. The staff can be erased while leaving the. As for the average staff thickness that is the basis for discrimination, find the average range of staff thicknesses corresponding to the interval between staffs in advance from a standard score and store it. Alternatively, the result of the staff recognition described above may be used.

When the staff erasing process (step S3) is completed, a beam recognition erasing process (step S4) is executed next. The beam is also similar to the case of the staff, and at the time of object recognition, there is a possibility that it may be erroneously recognized as a mass of black circles (noteheads) of chords as shown in FIG. It is deleted from the image data until it is performed. However, in the case of a beam, there are many shapes that do not meet certain conditions, and it is impossible to completely recognize and distinguish it. Therefore, for convenience of subsequent processing, at least a typical shape of the beam is recognized. Try to erase. Even if the extreme form of the beam remains unerased, it is preferable to erasing the object. As a typical condition of the beam, as shown in FIG. 14, the thickness Db, the length Lb, and the angle θb satisfy the following conditions.

Thickness D / 3 <Db <D Length Lb ≧ 2D Angle −45 ° ≦ θb ≦ 45 °

FIG. 15 is a flow chart of this beam recognition erasing process. First, a series of dodds satisfying the condition of thickness Db is discriminated from the image data in which the staff is deleted (step S31), and it is discriminated whether or not the sequence satisfies the condition of length Lb (step S32). If the series satisfies the two conditions, it is finally discriminated whether or not the series satisfies the condition of the angle θb (step S33). Then, the series of dots satisfying all the conditions is erased (step S34).

When the beam recognition erasing process (step S4) is completed, the object recognizing process (step S5) is performed next.
To execute. Details of this object recognition processing are shown in FIG.
This will be described with reference to the flowchart shown in FIG. First, since it is inefficient to perform matching processing on the entire surface of the image, in order to limit the range for performing matching processing, the image data from which the staff and beam have been erased is scanned, and there are lumps of dots. A circumscribed rectangle is set for each range (step S41). For example, as shown in FIG. 17, an image in which the staff and the beam have been erased is sequentially scanned horizontally from the upper left, and when a dot is hit at the point indicated by *, the outline of the dot cluster (indicated by ◇) is displayed. Tracking in the direction of the arrow shown, the maximum values Xmax, Ymax and minimum values Xmin, Ym of XY
Find in, and make the rectangle enclosed by these coordinate values the circumscribed rectangle. Specifically, starting from point * shown in the figure, "check the top, bottom, left, and right, if there is a dot, register that point,
The operation of "erasing the dot" is checked up and down, left and right, and repeated until no dot is found. It should be noted that in order to be able to restore the image in which the staff and beam are erased, at the time when the circumscribing rectangle of the cluster of all dots is found, the image in which the staff and beam are erased is determined as the circumscribing rectangle Either save it separately before, or register all the positions of the erased dots.

Next, the inside of the obtained circumscribed rectangle is scanned with the reference pattern to evaluate the matching degree (step S42). As shown in FIG. 18, the reference pattern used as the matching template in step S42 is a feature point that effectively captures the features of the patterns of various objects, that is, the point where a pixel dot should exist (black circle in the figure). And is formed as a distribution pattern of points (white circles in the figure) where pixel dots should not exist, and is registered in the ROM 14 in advance.
FIG. 18 shows reference patterns of sharp symbols (a) and (b), natural symbols (c), whole notes (d), half notes (e) and quarter notes (f). The sharp symbols (a) and (b) and the natural symbol (c) are quite similar, but the upper right and lower left portions show the points where the pixel dots should exist in the former and the pixel dots exist in the latter. Since they are set so that they cannot be different from each other, it is possible to distinguish them from each other by this difference. The whole note (d) and the half note (e) are also quite similar, but they can be distinguished from each other by using the pattern shown in the figure in consideration of the inclination of the ellipse. In addition, the reference pattern is set to a pattern that does not judge the portion of the remaining unerased staff (margin M left when the staff is erased as described above). In addition, as shown in FIG.
With respect to (a), when the object is deviated by a semitone, the position where the staff does not remain is different, and feature points are not attached to the positions where the staff may remain.
Even if a part of the staff remains, it is not affected.

The reference pattern read from the ROM 14 is reduced / enlarged on the basis of the intervals of the staff to be used for pattern matching processing. By doing this, it is possible to handle music images that have been input at any resolution, and even for objects such as music for children where the size of the object is significantly different from normal. Can respond. In this way, the inside of the circumscribed rectangle is scanned using the reference pattern corresponding to the size of the object, and at each point, the evaluation value is set to “+1” if they match, and if they do not match, the evaluation value shown in FIG. As shown in the figure, the evaluation value is set to “+0.9” when the vertical, horizontal, and diagonal directions are further examined and they match. This is to make it possible to deal with some deformation of the object shape. As a result of such evaluation, if the finally obtained evaluation value is 95% or more of the perfect matching, it is determined that the object corresponds to the reference pattern (step S43). For example, assuming that the reference pattern is composed of eight feature points, if the evaluation value is 7.6 (= 8 × 0.95) or more, it is determined that the object corresponds to the reference pattern. .

In the concrete processing, step S4
The object discrimination of 3 and the matching evaluation of step S42 are linked to "delete the already recognized pattern from the image", "evaluate the matching degree to some extent and if it does not match extremely, the matching degree for the reference pattern Is terminated and the matching degree is evaluated by fitting the next reference pattern. " Further, in this method, since it is difficult to recognize the dots and whole rests, they are recognized by recognizing geometrical features. In other words, all rests are recognized by calculating the area that all rests of the score should have from the interval of staffs, and recognizing by searching for a figure with dense dots and dots. Then, recognition is performed based on the feature that the area is extremely small and the dot dispersion is small. The information of each reference pattern also includes the information of the center coordinates thereof, and when the object is recognized, the RAM 15 is paired with the information of the object whose coordinates in the image data of the center coordinates of the reference pattern are recognized. be registered.

When the object recognition processing (step S5) is completed, the event recognition processing (step S6) is executed next. FIG. 20 is a flowchart showing this event recognition processing. First, the pitch of a note is recognized based on the staff information obtained by the staff recognition processing and the coordinate information of the object obtained by the object recognition processing (step S51). In this step S51, the musical symbols for controlling the pitches such as the sharp, flat, and natural pitches obtained in step S43, and the white and black circles of the notes (and further bar lines if necessary) are collected together in the X-axis direction. , And rearrange them in chronological order, and apply each note to the staff to determine the pitch. In addition, in step S51, the pitch of the additional line is determined at a position that is equal to the interval D of the staff in principle, but in practice, the interval of the additional line is the interval D of the staff. Since the pitch is often wider than the above, the pitch is determined by taking this into consideration so that the interval D between the staffs expands with a coefficient such as 1.2 times.

Next, the degree of proximity of the X coordinate of the note head center position of the note obtained by the object discrimination in step S43 is evaluated, and the sounds to be played at the same time are put together as a chord (step S52). That is, as the form of the chord,
Various forms are conceivable, such as h1, h2, and h3 of 1. However, in the case of h2, X at the center of the note head of the notes to be played simultaneously
The coordinates will shift by Dh. With this in mind,
When the interval Dh of the X coordinate of the notehead center is within the interval D of the staff, it is regarded as a chord. When the chord detection is performed more accurately, the interval between stems, which will be described later, may also be evaluated.

Next, the position of the end of the vertical bar (stem) connected to the note head of the note is evaluated (step S53). This processing is necessary for determining the point to start the search when searching and counting the number of beams or flags.
That is, as shown in FIG. 22, in step S53,
Search for the upper right and lower left ends of the note, find the distance between them and the center position of the black or white circle of the note head (indicated by * in the figure), and the farther one is the stem end (indicated by Δ in the figure) Set as coordinates. In order to cope with the distortion of the stem, the search recursively performs dots in two directions from the center position of the notehead to the upper left, upper, upper right and right and left, lower left, lower and lower right. FIG. 22
If the notehead is a white circle as shown in (b), tracing up, down, left, and right from the center point "*", and recursively starting from the point where the black dot hits causes the printed score to fade. Even if there is such a case, the search can be surely performed. Also, if the notehead of a note recognized as an object is a white circle and the difference between the distance from its upper right end to its center point and the distance from its upper left end to its center point is small, the whole note Consider as a note.

When the position of the stem end is detected, the beam or flag is counted with the stem end as a reference (step S54). That is, as shown in FIG. 23, when the position detected as the stem end is in the upper right corner of the notehead, it is along the Y direction from the point where the stem end “Δ” is swung right and left by D / 2. Start chasing down. The number of consecutive pixel dots in the tracking direction is counted, and this value is used in step S
If the beam width is within the margin of the beam described in 31, counting is performed, and in order to prevent counting the black circles of the chord as a beam, the tracking is stopped if there is a gap to some extent. On the other hand, when the detected stem end is the lower left point of the notehead, tracking is started along the Y direction and upward from the point where the stem end is swung left and right by D / 2. Next, the numbers of left and right beams counted in this way are compared, and the shorter sound length, that is, the larger number of beams is recognized as the sound length of the event. Therefore, FIG.
In the case of 3, the middle note to be searched is recognized as a sixteenth note by giving priority to the two right beams.

Next, a dot detection process (step S55) is executed. For example, as shown in FIG. 22, if there is a dot group having a sufficiently small area and a small dispersion of pixel dots when searching in a rectangular range of a staff interval D horizontally and a D / 3 vertically from the center of the notehead. Recognize it as a dot attached to the note. Finally, the note length is determined based on the note information recognized in step S43 and the beam or flag number information obtained in steps S52 and S54. If there is a detected dot, a half length of the original length is added to the corresponding note length to obtain the note length (step S5).
6). At this time, by using the position information of the bar line obtained previously, the sound length is finally checked for each bar, so that only 3.5 / 4 beats of notes are recorded, for example, 4/4 beats. It is possible to output an error such as not existing.

In this way, the event recognition result is
The score recognition result is appropriately stored in the RAM 15. For example, the recognition result of the musical score of FIG. 24 is as shown in FIG.
The recognition result includes an event number, an event type, a note length (the number of clocks when the quarter note length is 24 clocks), and a coincidence (chord) flag. The event number is
Numbers for identifying notes a to e, h and rest symbols f, g,
The event type is information indicating whether it is a note or a rest, and in the case of a note, pitch information such as C4 and F3. When the coincidence flag is "1", the chord and the event information immediately before that are formed.

When the event recognition (step S6) is completed, next, performance data, which is intermediate data for creating MIDI data, is created from the obtained recognition result (step S7). That is, as shown in FIG.
In order to control the IDI sound source device 2, the gate time (shorter than the note length at the output interval of the actual MIDI note-on data and note-off data) is calculated from the obtained note length, and note-on and note-off are calculated. It is necessary to decide the timing of. It is also necessary to determine the interval from the note-on timing of the preceding event to the note-on timing of the next event, that is, the duration. Therefore, performance data including duration, note number and gate time is generated from the recognition result shown in FIG.

FIG. 27 is a flow chart for creating performance data, and FIG. 28 is a diagram showing an example of performance data created by this processing. First, as the generation timing of the first note event data, "0" is written in the duration and the duration register (DUR) is set to "0".
(Step S61). Next, as shown in FIG. 25, the event information sorted in advance in the X-axis direction is taken out in order from the smallest event number, and whether or not the coincidence flags of the events following the event are “1”. To determine whether or not there is a simultaneous event (steps S62, 63). If there is no coincident event, the code length is stored in the DUR as the duration to the next event (step S65), but if there is a co-occurrence event, the shortest code length is stored until the next event. The duration is stored in the DUR (step S65).

Subsequently, it is determined whether or not the event is a note (step S66), and only when it is a note, the note number corresponding to the pitch is written and the note length x
Write 0.8 as the gate time (step S6)
7, 68). This is because in actual musical instrument performance, the actual pronunciation period of a certain note is often about 0.8 times the note length, except for slur or staccato-like performance. If there is a simultaneous event, note determination (step S66), note number, gate time writing (step S6)
7, S68) is repeated (step S69). If there is no other simultaneous event, the contents of DUR are written as the duration (step S71). At this time, if two written duration data are consecutive,
That is, when the note number and the gate time are not written due to the presence of rests, two consecutive duration data are added and rewritten as new duration data. After creating performance data for all events, write the end data at the end (step S
70, 72). By this processing, performance data including duration, note number and gate time as shown in FIG. 28 is obtained.

Finally, based on the obtained performance data,
By creating MIDI data and outputting it to the external MIDI tone generator device 2, automatic performance processing is executed (step S8). Here, a clock having a constant cycle corresponding to the tempo is counted, and when the duration of the duration included in the performance data has elapsed, a MIDI note-on message is output. Also, the clock is counted from the time when the note-on is output, and when the gate time elapses,
Outputs MIDI note-off message. These M
The IDI data is, for example, as shown in FIG. 29, a status byte indicating note on / off and a performance channel,
It consists of a note number that indicates the pitch and 3 bytes of velocity, which is information such as the strength of the sound. Among these, for velocity, if accents, crescendos, decrescendos, and other dynamic symbols are also recognized, it may be created from these recognition results, or if there is no such data in the recognized information. May be appropriately input / created through the switch 17.

In this way, since the musical score can be accurately recognized by a simple process, the processing speed can be improved and the required memory capacity can be reduced, and the personal computer system can be used for practical use. It becomes possible to carry out score recognition that can be endured. It should be noted that in the above embodiment, the MID is generated after the performance data is once created from the recognition result of the musical score.
Although the I data is created, the recognition result may be directly converted into MIDI data. Further, the method of obtaining the inclination of the staff is not limited to that of the embodiment, and the coordinates of the left and right ends of the staff may be obtained and the inclination may be obtained from the values. The inclination may be corrected in the Y-axis direction after the X-axis direction.

[0045]

As described above, according to the present invention,
By deleting the staff and beam, notes and symbols can be left in an island shape, the pattern matching area can be limited, and the beam portion is prevented from being mistakenly recognized as a note. Therefore, it becomes possible to efficiently recognize the score, and it is possible to accurately recognize the event on the score with a small calculation load.

[Brief description of drawings]

FIG. 1 is a block diagram of a score recognition / automatic performance system according to an embodiment of the present invention.

FIG. 2 is a flowchart of the musical score recognition / automatic performance process.

FIG. 3 is a diagram showing an example of a score of the recognition target.

FIG. 4 is a diagram showing original image data obtained by reading the score and a projection thereof.

FIG. 5 is a flowchart of the same staff recognition inclination correction process.

FIG. 6 is a diagram showing a horizontal component extracted image in the graphic processing.

FIG. 7 is a diagram showing a method of tilt correction in the graphic processing.

FIG. 8 shows image data after Y-axis direction tilt correction in the same process.

FIG. 9 is a diagram showing image data and projection after X-axis tilt correction in the same process.

FIG. 10 is a flowchart of the same staff deleting process.

FIG. 11 is a diagram for explaining a method of discriminating thickness and length in the same process.

FIG. 12 is a diagram showing image data after the staff is erased in the same process.

FIG. 13 is a diagram showing an enlarged part of the image data together with a comparative example.

FIG. 14 is a diagram for explaining the same beam recognition erasing process.

FIG. 15 is a flowchart of the same process.

FIG. 16 is a flowchart of the same object recognition process.

FIG. 17 is a diagram showing a method of determining a circumscribed rectangle in the same process.

FIG. 18 is a diagram showing a reference pattern in the same process.

FIG. 19 is a diagram showing a matching evaluation method in the same process.

FIG. 20 is a flowchart of the same event recognition process.

FIG. 21 is a diagram showing a chord discrimination method in the same process.

FIG. 22 is a diagram showing a stem end determination and dot detection method in the same process.

FIG. 23 is a diagram showing a method of beam / flag counting in the same process.

FIG. 24 is a diagram showing an example of a musical score recognized in the event.

FIG. 25 is a diagram showing a recognition result of the musical score.

FIG. 26 is a time chart showing the timing of driving an actual MIDI sound source based on the recognition result.

FIG. 27 is a flowchart of the performance data creating process.

FIG. 28 is a diagram showing performance data created by the same processing.

FIG. 29 is a diagram showing MIDI data created from the performance data.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Score recognition device, 2 ... External MIDI sound source device, 3 ... Output device, 11 ... System bus, 12 ... Image scanner, 13 ... CPU, 14 ... ROM, 15 ... RAM, 16
... Timer, 17 ... Switch, 18 ... Display, 19
… MIDI interface.

Claims (1)

[Claims] 1. A staff, a note, a symbol and their positions in the score are recognized from image data obtained by reading an image of the score, and based on the recognition result, the pitch of the musical tone, the sounding timing and In a musical score recognition apparatus for generating information such as pronunciation time, a staff deleting means for deleting a staff from the image data and a plurality of notes from the image data from which the staff is deleted by the staff deleting means. A thickness discriminating means for discriminating a series of pixel dots of a predetermined thickness corresponding to the thickness of the beam, and a predetermined length or more of the series of pixel dots of a predetermined thickness discriminated by the thickness discriminating means. A length discriminating means for discriminating a run having a length of, and an erasing means for erasing a run whose inclination is within a predetermined angle range among the run of pixel dots discriminated by the length discriminating means, This erase Music recognition apparatus characterized by succession of the pixel dots stage is to recognize notes, symbols and their position and the like from the erased image data.