JPS6315373A - Musical score five stuffs detecting system - Google Patents

Abstract

PURPOSE:To detect the five stuffs of a musical score of optional size and to prevent the crossing of five stuffs of a musical score in left and right by making the value of the most frequent occurrence out of interval of black segments line interval, excluding black segments out of conjugate, and connecting black segments in sampling position in order. CONSTITUTION:Sampling lines are drawn in the direction of Y axis at sampling positions S1, S2,..., and the sampling area of length (r), width 1 making the lines starting end or terminating end is considered. The number of black picture elements is counted for all sampling areas, and when the number is above predetermined number (a), the area is made to a black area, and when below (a) the area is made to a white area. Transition of the sampling area that changes from upper part to lower part, and from white to black, and transition from black to white and parts of no transition are abandoned. In such case, the frequency of occurrence is highest in five stuffs interval. By making group connection and selecting lines having black segments having more than five stuffs reference number (m), the five stuffs can be determined.

Description

[Detailed description of the invention] (g) Field of industrial use The present invention relates to a system for optically detecting music staffs.

More specifically, from the binarized bata-n of the musical score,
Concerning an improved method for detecting staves.

It can be used for automatic musical score recognition devices, automatic musical instrument performance devices, and Braille translation devices for musical scores.

(B) Background Art Japanese Patent Application No. 59-186076 proposes a musical score staff detecting device capable of detecting the staff from a binarized image of a musical score, calculating and correcting the inclination.

The staff detection method adopted here will be explained.

(1) A plurality of sample points are provided in the length direction of the staff.

Let the sample points be Sl, S2,...

(2) Provide a filter of constant width extending in the length direction with a starting or ending end at each sample point. This is an elongated window that is one pixel in the vertical direction and r pixels in the horizontal direction, and the filter moves down one pixel in the vertical direction at multiple sample points.

(3) Apply each filter to the binarized image of the musical score. Binary 6 means that a certain pixel is determined to be either white or black. Musical staves, notes, symbols, characters, etc. produce black pixels. The white background becomes white.

(4) The number of white pixels included in the applied filter is p
Let the number of one black pixel be q. p+q=r. If q>p, this is a black line segment, and if q<p, this is a white line segment.

The black line segment υ may be part of the staff.

The white line segment V is never part of the staff.

(5) Determine whether the sample point is a black line segment U or a white line segment V using the width of IXr from top to bottom, and change from the white line segment V to the black line segment U, and the black line at the boundary Input the minute coordinates (X, 3F) into the memory. This black line segment is called a staff candidate line segment.

(6) Find the center of gravity of the first half of the staff candidate line segment.

Find the center of gravity for the latter half as well. This is vertical,
That is, the center of gravity in the Y direction is determined.

(7) A staff is created by connecting the centers of gravity of the first half and the second half. The inclination e between this straight line and the preset X coordinate of the image can also be determined from this. This is called an approximate staff.

(8) Determine the coordinates of the approximate staff at the left end of the staff image, that is, at x=O. With this, Yl and θ1 can be determined, so one of the staffs can be determined.

(9) Repeat the same process to determine the Y coordinates of the remaining four staves. Thus, Yl, el, Y2, e
2,..., Y5, θ5 can be determined.

(lO) Repeat the same process to determine the coordinates of all staves on the same plane.

(b) Problems with the prior art The method described above does not necessarily detect the staff correctly because there is a lack of consideration of the consistency of the staff between the upper and lower and left and right data at the sample point S. There is a drawback.

Tuplets, dyads, and other symbols that are parallel to the staff are present in the score. Since these are staff candidates, the center of gravity is determined, and an approximate straight line is determined, there is a possibility that a staff that is not a true staff may be detected as a staff.

Since they are musical staffs, they are equally spaced and parallel to each other. In terms of the above symbols, equal spacing is y1+ y2
= W (1)y2-y, =w
(2) y, -y, = w
(3) Y4-Y, ±W(4). Parallelism is expressed by θ1=θ2=θ3=e4=θ5 (5). Let's simply call this θ.

No = Wcosθ (6) Wo defined by (6) is the interval between staffs.

Although the relationships expressed by equations (1) to (5) exist, the staff detection method described above does not use them. For this reason, (1) ~ (
5) There was a possibility that the formula was not satisfied, that is, the wrong staff was detected.

00 Purpose It is an object of the present invention to provide a method for detecting staffs with fewer errors by utilizing the equidistant nature of musical scores.

Another object of the present invention is to provide a method that can detect the staff without error even if the musical score is tilted in the image.

A third aspect of the present invention is to provide a method that allows staff detection without error for any staff spacing.
This is the purpose of

(6) Method of the Invention A musical score is imaged by optical means, and the image is decomposed into pixels by an image processing device to obtain a binarized image. The number of pixels on one screen is assumed to be sufficiently large, for example, 2000 x 3000. For this reason, it is assumed that a plurality of pixels exist in the vertical direction of the staff.

A binarized image is one in which the pixel values are determined to be binary, and can only be black or white. To determine the value, the brightness of the pixel is compared to a reference brightness. White and black are determined by the size of this.

Below, the procedure for detecting musical score staves will be explained in order.

(1) Set the X-axis and Y-axis on the screen. The horizontal direction is the X axis,
The downward direction is the Y-axis. Although the X-axis and the staff should be approximately parallel, they are not necessarily exactly parallel. Forms an angle of θ.

Multiple sampling positions a Ss in the X-axis direction, S2°・
Set. This may be at equal intervals, but may also be at irregular intervals.

Sampling lines are drawn in the Y-axis direction at sampling positions S1, S2, . . . , and a sampling area of length r1 and width 1 is considered with these lines as the starting or ending points. The number of black pixels is counted for all sampling areas, and if the number is greater than or equal to a predetermined number a, it is considered a black area, and if it is less than a, it is considered a white area.

a is an integer represented by Q (a (r). The above process is called black-white discrimination.

(2) Detect the transition of the sampling area changing from white to black from top to bottom. Discard any transitions or non-transitions from black to white. For the staff, this means selecting the sampling area at the top of the staff. Areas in the middle and bottom of the staff are discarded.

Assume that black pixels 31 to 103 as shown in FIG. 1 are thus obtained. In Figure 1, the staff and musical note symbols are shown as two lines and are not painted black. This is to clearly show black pixels. In reality, the staff and symbols appear as a collection of black pixels.

The sheet music is long and wide both horizontally and vertically, but only a portion is shown here for simplicity.

Here, black pixel 31=103 is binarized, Sl, S2, .
・ A horizontal set of black pixels (IX
r). This will be called the black line segment.

Black line segments 35 to 64 are black line segments corresponding to the upper ends of the staffs.

However, in addition to this, black line segments also occur due to musical notes and continuous phases. This occurs depending on the sampling position.

The black line segments 65, 66, 67, and 68 are generated by the continuous phase close to the sampling position.

In the case of black line segment 81.88, this occurred because the ball part of the note was close to the sampling position. Since the black line segment corresponds to the transition from the white area to the black area, the black line segment occurs in the middle part of the ball.

The black line segment 103 also occurs for the same reason.゛Black line segment 101
is caused by an auxiliary horizontal line below the staff. Black line segment 102
is caused by two of these being lined up.

Furthermore, not all staves' tops appear as black line segments.

Black line segments 73, 74, 79, and 80 are black lines that appear above the staff because the horizontal line of the note overlaps with a part of the staff, and the upper end of the staff is covered by the horizontal line. It's a minute. This happens because we are picking up the transition from a white pixel to a black pixel.

In this way, in addition to the black line segment corresponding to the upper end of the staff, there are black line segments that appear stochastically depending on the arrangement of notes. These can be called noise. First of all, you have to cut out the noise.

The above process is called black line segment extraction.

(3) Calculate the interval between black line segments at each sampling position S1, S2, etc., and create an appearance frequency distribution, that is, a histogram, for the interval.

In the lower part of FIG. 1, a histogram for each sampling position is illustrated.

The staff interval W appears most frequently.

However, the frequency of appearance for intervals other than W is not zero. A distribution may appear above and below W. This is the appearance of noise.

In one page of music score, there are sets of staffs of about 8 to 20 lines, so the number of black pixels that appear at one sampling position is large. The number of intervals is only one less than the number of black pixels, so it is still sufficiently large.

The total number of appearance frequency distributions is about 40 to 100, and it is estimated that the staff interval W should appear most frequently.
It is probabilistically correct.

The estimated staff interval at the sampling position Sc can be set to the interval Wk that appears most frequently at k.

If a large number of notes coincidentally overlap on the vertical line of Sk, it is possible that the most frequent interval is not the staff interval. In this case, the most frequently appearing interval is set as the estimated staff interval, including the sampling positions in the left and right vicinity of the sampling position Sk.

Furthermore, the estimated interval can be determined by a moving average instead of the most frequent appearance interval at sampling positions near the left and right.

The above process is called interval determination.

(4) In this way, sampling positions S1, S
Assume that the estimated staff intervals W1, W2, etc. are determined in 2, .

At a certain sampling position Sk, for all black line segments, add the estimated staff interval Wk and the error ±ΔW above and below, and check whether there is another black line segment at the position i, Th±ΔW. Find out. ΔW may be about 1/10 to 115 of Wk.

Both up and down, 4 or just up, or just down, W
If there is a black line segment separated by k±ΔW, this is called a staff candidate line segment. A line segment that does not have such a black line segment either above or below is called a non-staff candidate line segment.

Adjacent black line segments separated by Th±ΔW are called W conjugate line segments.

The black line segment 35 is W-conjugate with the black line segment 41. Therefore, 35
．． 41 is a staff candidate line segment.

Similarly, black line segments 35 to 64 are found to be staff candidate line segments because there are other black line segments that are W conjugate.

Discard non-staff candidate lines.

It is assumed that the interval between the black line segments 31 and 35 is greater than or equal to W1+ΔW. 31 would like to have another black line segment that is W conjugate. This means that 31 is a non-staff candidate line segment. Similarly, it is assumed that the black line segment 34 is also found to be a non-staff candidate line segment.

It is assumed that the black line segment 67 is not W conjugate with the black line segment 69. However, it is assumed that 67 is W conjugate with black line segment 65. Then, the black line segments 65.67 become staff candidate line segments.

It is assumed that the black line segments 66 and 68 are W conjugate to each other.

66.6g becomes a staff candidate line segment.

If the black pixels 81 and 88 due to balls exist alone, they would be non-staff candidates because they would have no W conjugate. However, since 8L 88 exists adjacent to the staff, 8L
88 is a staff candidate.

Since the black picture element 101.102 is W conjugate to 96.97, it is a staff candidate line segment.

The black pixel 103 has no W conjugates above and below.The black line segments 73, 74, 79, and 80 that appear instead of the staff have double continuous phases, and the interval is W±ΔW. They become W conjugate to each other only when

In this case, assume that 73.79 is W conjugate.

73.79 is a staff candidate line segment. 74.80 also remains as a staff candidate line segment.

The above process is called staff candidate selection.

(5) In this way, the staff candidate line segments were selected, but they still contain noise.
Furthermore, it is a line segment, not a straight line.

Staff candidates are stored by X and Y coordinates for each sampling position Sk. The X coordinate can also be said to be the horizontal number of the pixel, and the Y coordinate can be said to be the number from top to bottom of the pixel.

Since the X coordinate of the sampling position is the same, in the end,
It is sufficient that the Y coordinates (Yl, Y2, . . .) of each sampling position are stored.

Staff candidates located at adjacent sampling positions are connected in the X direction.

This will be explained with reference to FIG. X of sampling position
Let L be the distance in the direction. Assume that a black line segment extends to the right from five points at the sampling position. The Y coordinate of this is K
shall be. If a staff candidate line segment exists between K-d and -1-d at an adjacent sampling position, these two lines are connected.

In other words, it is sufficient that a staff candidate line segment exists between F and 0 at adjacent sampling positions.

In this way, at the adjacent sampling position, the Y coordinate K of J
When there are other staff candidates between ±d above and below, this is said to be "near d". It is assumed that staff candidates that are not between the top and bottom ±d are not located near d. This will be explained with reference to FIG.

The topmost staff candidate is line segment 32 of S2.

Even if this is extended to 83, there are no other staff candidates between ±d. Even if it is extended to S, , S, there is no staff candidate line segment between ±d. 32 does not have any staff candidate line segments near d. This can be thrown away. Similarly, line segment 33 is also discarded because it does not have a line segment near d.

35 of Sl is the next highest staff candidate line segment.

Draw a straight line parallel to the X axis from 35 to 82, S! In this case, a staff candidate line segment 36 exists within the range of ±d of this straight line. In other words, 36 is near d of 35. In this case 35 and 36 are connected laterally.

This operation is called a horizontal connection.

Similarly, line segment 37 of S3 is located near d of line segment 36.

36 and 37 are horizontally connected.

In the same way, 38 is near d of 37, and 39 is near d of 38.
Assume that the neighborhood, 40, is the d neighborhood of 39.

In the end, staff candidate lines 35.36.37.38.39.4
0 is horizontally connected.

In the same way, staff candidate line segments 41-46, 47-52
．． 53-58 and 59-64 are horizontally connected.

The reason why the staff candidate line segments 75 to 80 are horizontally connected is also due to the same reason. 82-87, 8-9-94 and 95-100 are also horizontally connected.

Staff candidate line segments 65 and 66 are not near d. Therefore, the two are not horizontally connected. 65.66 does not have staff candidate line segments near d below S3, S4, etc.

Therefore, both 65 and 66 remain isolated staff candidate line segments.

The staff candidate line segment 81.88 does not have other staff candidate line segments near d in S2 and S3.

This musical score is tilted by θ with respect to the X coordinate axis, so
In S6 and after S6, there is a possibility that there are other staff candidate line segments near d. However, these line segments are excluded because they are already laterally connected into other groups. in the end,
81.88 remains an isolated staff candidate line segment.

Staff candidate line segments 101 and 102 located one line below the staff are located near d and are connected to each other. But 102 is S4
does not have other staff candidate line segments near d.

The staff candidate line segment 79.80 is not a part of the staff but a part due to continuous phase. However, since 78 and 79 are located near d, 78 and 79 are horizontally connected.

79 and 80 are also horizontally connected. In this way, staff candidate line segments 75, 76, 77, 78, 79, 80 are horizontally connected.

Staff candidate line segments 69 to 72 are horizontally connected. 72 and 73 are not near d. Therefore, the connection will be broken here. Further, since 73 and 74 are near d, they are horizontally connected to each other.

In this way, by connecting those in the vicinity of d at adjacent sampling positions, it is possible to form a staff in many cases.

However, since the connections between 69-72 and 73.74 are broken midway through, they cannot become part of the staff as they are.

(6) A set of horizontally connected black line segments is called a group.

Connecting groups that are lined up horizontally is called group connection.

Conditions for group connection will be explained with reference to FIGS. 5 and 6.

When two arbitrary groups are selected, the interval between the groups is an integral multiple of L, that is, iL, where i is a natural number. Let Z be the end of one group. Let M be a point extending from Z in the X-axis direction by iL. Take an interval of f above and below M. The upper limit is D1 and the lower limit is E.

However, f is smaller than staff interval W o　　f’　＜w It is.

Figure 5 shows the general relationship. When the coordinates of one group are moved to the right or left by iL, the coordinates of the other group are ±f
Two groups are said to be "near f" if they exist within f. What we want to exist within ±f is said to be "not in the vicinity of f."

The fences near f are connected. This operation is called group connection.

However, if the staff is tilted by e with respect to the X axis,
The base line ZM may be made parallel to the musical staff as shown in FIG.

Since the line segments have already gathered and formed a group, 2
From whichever of the two groups is longer, we can calculate the slope of the group. This should be approximately equal to the slope of the staff.

Knowing the inclination angle in this way, draw a base line ZM from Z so that it is inclined to the X axis, and draw a point f above and below M,
Let it be E. The presence of the start of an adjacent group between DEs may be defined as being in the vicinity of f.

Group connection will be explained with reference to FIG. Black line segment 35.39.
40 are in a group. In other words, the black line segments are close to each other by d. Black line segments 111 to 113 form a group. However, line segments 40 and 111 are not in the vicinity of d. 113 and 11
4 is also not near d. Therefore, there is no horizontal connection here.

It is assumed that 40 and 111 are not near f in the sense of FIG. In this case, groups 35-41 and groups 111-113 are not connected.

However, it is assumed that the black line segments 40 and 114 are in the vicinity of f. In this case, groups 35 to 40 and 114 are connected as a group.

In this way, black pixels 35 to 40.114 were connected together.

The same applies to the three-way wire. 120 and 121 are not near f, but 120 and 123 are near f.

In this case, 51 to 120 and 123 are connected in a group.

Group connections are indicated by dashed lines.

Since the line segments 32 and 33 consisting of isolated black pixels do not have a group near f, they remain isolated.

Since the isolated black line segments 65 and 66 also do not have a group near f, they are not connected either.

Isolated groups 67 and 68 have no neighboring groups near f. This remains isolated.

The same applies to isolated groups 101-102 and 190-192.

These do not have neighboring groups near r. This too remains isolated.

(7) In this way, the lines that make up the staff are obtained. The number of black line segments of the continuous line is equal to or less than the number n of sampling positions. A line containing n black line segments should be a constituent line of a staff. However, even lines containing fewer than n black line segments may be constituent lines of a staff. However, the number of black line segments is one, 32.33.
65.66, or the group 67. where the number of black line segments is two.
68 or 101.102 would like to constitute a staff.

Therefore, if the line obtained by the connection has m or more black line segments, this is determined as a candidate staff. Naturally, 0<m≦n. Since the terminal Sn of the sampling position may not cut the staff, m, which is smaller than n, was used as a criterion for determining whether or not the staff is a candidate staff. If it is a staff, it should be almost continuous from left to right, so m may be a fairly large value. Let m/n = Q, 5 to 1.

m is called the staff standard number.

(8) It should be possible to determine the staff by selecting a line with m or more black line segments.

However, if there were six lines instead of five lines, we would have to exclude one of them.

In Figure 4, black line segments 150-160 and black line segments 170-1
Suppose there are 80 groups and there are 6 lines, although we do not know which one is redundant.

There is almost no chance that this will happen, but it is a case where the auxiliary line (the C note) directly below the staff coincidentally exists at the sampling position. Mana, 190~ between the staffs
There is a musical note ball at position 192, below this by W,
This kind of thing can happen if other musical notes line up and all of them match the sampling positions.

Therefore, a candidate staff with black line segments of m or more is selected and counted in this number of staves. If there are 5 lines, then the staff has been properly selected. You can now cancel the operation.

However, suppose there are six candidate staves within a distance of W from each other.

In this case, the upper end is 150-160 or the lower end is 170-1
80 is incorrect.

Checking whether the number of candidate staves is five is called a five-line checking operation.

If the number of candidate staves is 4, either the staff cannot be detected or
Alternatively, the value of the staff standard number m is subtracted, and the number of candidate staves is set to five.

If there are six pieces, it is more troublesome. This will be explained below.

(9) In this way, when there are six candidate staves, approximate straight lines are determined for the upper end line and the lower end line by the least squares method.

Check the spacing and parallelism between this neighboring straight line and the other four lines. In other words, the inclination angle with respect to the X axis is examined.

Incorrect lines are detected by checking spacing or parallelism. Here, the black line segments 170 to 180 are deemed inappropriate and are excluded.

As explained in detail above, the method of the present invention includes, in addition to the optical means and the image processing device, (1) black and white discrimination means (2) black line segment extraction means (3) interval determination means (W) (4) Staff candidate line segment selection means (W conjugate (5) horizontal connection means (d neighborhood (6) group connection means
(f neighborhood) (7) Staff selection means (Black line fraction>m) (8) Five check means (9) Straight line approximation/one exclusion means.

(2) Effects (1) Among the intervals of black line segments, the most frequently occurring value W is set as the line interval. With this lick, it is possible to detect the staff of a music score of any size.

(2) Since the line interval W is determined in advance and black line segments that are not in the W conjugate are excluded, noise caused by the beads of musical notes or the signs of tuplets can be effectively removed.

(3) Since the black line segments at the sampling positions are connected in order, even if the score is tilted, there is no difference between the left and right staffs.

(4) Devices that image musical scores and convert them into binary images are known. The following operations can be performed by dedicated hardware circuitry. Since connection operations and the like can be executed in parallel, processing time can be shortened.

[Brief explanation of drawings]

FIG. 1 is a binary image diagram illustrating a state in which a large number of sampling positions are set in a binarized image of a musical score and black line segments are extracted. FIG. 2 is a diagram for explaining horizontal connections of black line segments. FIG. 3 is a diagram illustrating a range in which adjacent dotted line segments can be horizontally connected. FIG. 4 is a diagram illustrating group connection in which groups are connected in the horizontal direction. FIG. 5 is a diagram showing the possible range of group connections when the staff lines are parallel to the X-axis. FIG. 6 is a diagram showing the possible range of group connections when the staff is tilted with respect to the X-axis. Inventor Toshi Matsushima Akihara 1) Tadanorien Moto Tatsukin Katsu Mori Teru Teru Kanki 1) Yasushi Patent Applicant Teru Otsu Kanki Sumitomo Electric Industries, Ltd.

Claims (1)

[Claims] an optical means for capturing an image of a musical score; an image processing device that decomposes the captured image into pixels arranged in horizontal and vertical directions and performs a binarization process regarding brightness and darkness to obtain a binary image consisting of black pixels and white pixels; The X axis is horizontal to the screen, the Y axis is from top to bottom, and
Multiple sampling positions S_1, S_2,... in the axial direction
, S_n, and each sampling position S_1, S_2
,..., in S_n, a sampling area having a length of pixels in the X direction is provided at every position in the Y-axis direction, and if the number of black pixels in the sampling area is greater than the number of white pixels, this is considered a black area. , a black-and-white discrimination means for determining a black and white area by treating it as a white area when the number of black pixels is smaller than the number of white pixels; A black line segment extraction means that detects an area, sets it as a black line segment, and stores the XY coordinates, and calculates the interval between vertically adjacent black line segments at each sampling position to find the interval with the highest frequency of appearance. An interval determining means that takes this as the estimated staff interval W, and checks whether or not other black line segments exist at positions W±ΔW above and below all black line segments at each sampling position. If there are line segments, the staff candidate line selection means selects these black line segments as staff candidate lines, and the difference in the Y coordinates of two staff candidate line segments at adjacent sampling positions is calculated from W. horizontal connection means for connecting staff candidate lines for all staff candidate line segments for connecting the two staff candidate line segments if the distance is less than or equal to a small certain distance d;
If two groups of horizontally connected staff candidate line segments are close to each other in the X direction, and the difference in their Y coordinates is a certain distance f or less, which is smaller than W, then A group connecting means that connects adjacent groups, and staff selection that selects this connecting line as a candidate staff when the number of black line segments included in the connecting line is greater than the staff standard number m that is less than or equal to the total number of sampling positions n. a means for checking whether the number of candidate staves that form an interval W from each other is five; and when there are six candidate staves, an upper candidate staff and a lower candidate staff; is linearly approximated by the method of least squares, select the upper or lower candidate staff that has the interval and slope angle closer to the other four candidate staves, and select the upper or lower candidate staff that has the slope angle. A musical score staff detection method characterized by comprising a means for excluding.