KR101793184B1 - Apparatus of Fast lyric area extraction from images of printed music scores and method therefor - Google Patents

Abstract

The present invention relates to an apparatus and method for extracting a lyric area for automatic performance of a captured music score image, and more particularly, to an apparatus and method for extracting a lyric area for automatic performance of a captured music score image, capable of extracting the lyric area in an optical music recognition (OMR) process of a score using information of a frequency, a projection, a height and position of the staff, etc. for the automatic performance of the captured music score image. As described above, the present invention can obtain a high recognition rate with a small calculation amount in the extraction of the lyric area of the music score and extract the lyric area by correcting distortion using staff and bar line information.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for extracting a lyric region for automatically playing a photographed music score image,

The present invention relates to an apparatus and method for extracting a lyric region for automatic performance of a photographed music score image, and more particularly, to an apparatus and method for extracting a lyric region for automatically playing a photographed music score image using information on frequency, projection, The present invention relates to a lyric region extracting apparatus and method for automatically playing a photographed music score image, which extracts a lyric region during an optical music recognition (OMR) process of a score.

Japanese Laid-Open Patent Publication No. 10-2014-0144876 discloses a method of drawing a score using handwriting recognition, which includes a process of converting a score image into a score of a digital format using Optical Character Recognition (OCR) And editing the score of the digital form according to the handwriting input.

In recent years, with the development of technology, digitalization has been required in many parts for quick and accurate information, and music is no exception. Optical score recognition enables computers to be used in a variety of fields ranging from playing, analyzing, comparing, arranging, and composing music through digitalization of printed musical scores. However, in many optical musical score recognition applications, the extraction or recognition of the lyric area that affects the accuracy of the score recognition is not considered. When the lyrics are touched or overlapped with the music symbol in the optical score recognition, the recognition rate of the score is lowered. There was a problem.

Therefore, it is necessary to consider extracting or recognizing the lexical region that affects the accuracy of optical score recognition.

An object of the present invention is to provide a method and apparatus for automatically playing a recorded musical score image that enables a high recognition rate to be obtained even with a small amount of calculation using a frequency method and an empirical rule in a lexical region extraction of a Korean music score And a method for extracting a lyric region.

There is also provided a device and method for extracting a lyric region for automatically playing a photographed music score image, which corrects distortions by using line and line information, and extracts a lyric region using frequency and empirical rules.

According to an aspect of the present invention, there is provided a binarization apparatus comprising: a preprocessing unit for binarizing a score image; A preference information extracting unit (200) for extracting preference information from the preprocessed music score image; And a lyric line extracting unit 300 for setting a potential lyric region in the preprocessed music score image, searching and searching for a zonal region and an execution length, and searching for candidate boundary information to extract a lyric region.

Preferably, the preprocessing unit 100 includes a binarization module 110 for binarizing input music images; And a correction module 120 for correcting the distortion of the score image using the line and line information as a result of the binarization process.

The cruise information extracting unit 200 includes a cruise detection module 210 for detecting a cruise line; And a five-line information extraction module 220 for evaluating the space and height information of the pentagon.

The lyric line extracting unit 300 further includes a lyric area setting module 310 for setting the area between the five-line and the five-line area as a potential lyric area; A zone division module (320) for dividing the potential lyrics area set through the lyrics area setting module into N zones; Execute vertical projection to find consecutive run lengths other than 0 in a partitioned area through the zone partitioning module, and to set the execution length to add the run lengths to one zone when the space between run lengths is less than or equal to the straight line height A search module 330; A candidate boundary information extraction module 340 for correcting and extracting a candidate boundary when there is noise after the binarization processing of the preprocessing unit; A candidate boundary filtering module 350 for filtering a candidate boundary of the candidate boundary information extraction module; And a lyric region extraction module 360 for displaying the lyric region and the score region as O or I as a result filtered through the candidate boundary filtering module.

A method of using a lyric region extracting apparatus for automatically playing a photographed music score image, the method comprising the steps of: (a) pre-processing a score image by a control unit of the lyric region extracting apparatus; (b) extracting the selection information from the preprocessed music score image; And (c) setting a potential lyrics area in the preprocessed music score image, wherein the control part searches for and sorts the zone division area and the execution length, and searches for candidate area information to extract a lyrics area.

Preferably, the step (a) includes the steps of: (a-1) binarizing a score image input by the controller; And (a-2) the controller corrects the distortion of the score image using the five-line and line information as a result of the binarization process.

Preferably, the step (b) includes the steps of: (b-1) detecting the pentode of the control unit; And (b-2) the control unit evaluating the space and height information of the pentagon and extracting the pentagon information.

Preferably, the step (c) includes the steps of: (c-1) setting the area between the pentagon and the pentagon as a potential lyric area; (c-2) causing the control unit to divide the potential lyric area to be set into N zones; (c-3) The control unit performs vertical projection to find non-zero consecutive run lengths in the subdivided zone, so that when the space between run lengths is less than or equal to the syllable height, step; (c-4) if the control unit has noise after the pre-processing of step (a), correcting the candidate boundary and extracting it; (c-5) causing the control unit to filter the candidate boundary of step (c-4); And (c-6) causing the control unit to display the lyric region and the score region as O or I as a result of the filtering in the (c-5) step.

According to the above description, it is possible to obtain a high recognition rate even with a small amount of calculation in extracting the lyrics region of the music score, and it is possible to extract the lyrics region by correcting the distortion using the five-line and the ninth line information.

1 is a block diagram of a lyric region extracting apparatus for automatically playing a photographed music score image according to an embodiment of the present invention,
FIG. 2 is an overall flowchart of a method using a lyric region extracting apparatus for automatically playing a photographed music score image according to an embodiment of the present invention.
3 is an exemplary view illustrating a potential lyric area for a lyric region extracting apparatus for automatically playing a photographed music score image according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating an area division for a lyric region extraction apparatus for automatically playing a photographed music score image according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating line information extraction information for a lyric region extraction device for automatically playing a photographed music score image according to an exemplary embodiment of the present invention. Referring to FIG.
FIG. 6 is an exemplary view illustrating a run-length search for a lyric region extracting apparatus for automatically playing a photographed music score image according to an embodiment of the present invention.
7 is a view illustrating run-lengths filtering for a lyric region extracting apparatus for automatically playing a photographed music score image according to an embodiment of the present invention.
FIG. 8 is a diagram illustrating a modification of a candidate line for a lyric region extracting apparatus for automatically playing a photographed music score image according to an exemplary embodiment of the present invention.
FIG. 9 is an overall flowchart illustrating a method using a lyric region extracting apparatus for automatically playing a photographed music score image according to an embodiment of the present invention.
10 is a detailed flowchart of a lyric region extraction step of a method using a lyric region extraction device for automatically playing a photographed music score image according to an embodiment of the present invention.

Specific features and advantages of the present invention will become more apparent from the following detailed description based on the accompanying drawings. It is to be noted that the detailed description of known functions and constructions related to the present invention is omitted when it is determined that the gist of the present invention may be unnecessarily blurred.

The apparatus and method for extracting a lyric region for automatic performance of a photographed music score image according to an embodiment of the present invention solve the rotation problem (distortion correction) using a pentagon and a line, This technique is characterized by a technique for extracting a lyric area through run-lengths search and filtering using rules.

Hereinafter, the present invention will be described with reference to the accompanying drawings.

1 is a block diagram of a lyric region extracting apparatus for automatically playing a photographed music score image according to an embodiment of the present invention.

1, a lyric region extracting apparatus for automatically playing a photographed music score image according to an exemplary embodiment of the present invention includes a preprocessing unit 100, a preference information extracting unit 200, a lyrics line extracting unit 300).

The preprocessing unit 100 preprocesses the score image, binarizes the score image to be inputted, and corrects distortion of the score image by using the five-line and line information.

The preprocessing unit 100 for performing such a function includes a binarization module 110 for binarizing the inputted score image and a correction module 120 for correcting the distortion of the score image using the binarization result, ).

In order to correct the distortion of the music score image, after detecting the music piece, the music piece is detected by using the characteristic that the music score section is connected to the first line and the fifth line of the line. Next, the dive line is divided into the regional regions between the nodes and the nodes using the line information, and the regions are rearranged horizontally to correct the distortion.

The preference information extracting unit (200) extracts preference information from a preprocessed score image. The sloped information extracting unit 200 can detect a sloped line and perform a function of extracting the sloped information by evaluating the space and height information of the sloped line.

The slope information extraction unit 200 may include a slope detection module 210 for detecting a slope and a slope information extraction module 220 for evaluating slope space and height information.

The lyrics line extracting unit 300 is configured to set a potential lyrics region in the preprocessed music score image, to search for a zone division region and an execution length, and to search for candidate boundary information to extract a lyrics region.

The lyric line extracting unit 300 according to the present embodiment includes a lyric area setting module 310, a zone dividing module 320, an execution length searching module 330, a candidate boundary information extracting module 340, a candidate boundary filtering module 350, and a lyric region extraction module 360.

The ladder area setting module 310 is a configuration for setting the area between the pentagon and the pentagon to be a potential lexical area.

The reasons for setting the potential lyric area are as follows. The arrangement of the lyrics in the score is made between the line and the line. However, since the whole area between the pentagon and the pentagon may not be the lexical region, the candidate between the pentagon and the pentagon is the potential lexical region including the lexical region. Therefore, we set the potential lyric area and extract the correct lyric area through more detailed analysis.

The zone division module 320 is a configuration for dividing a potential lyric area set through the lyric area setting module into N zones.

Here, the reason for dividing into N zones will be described as follows.

Since the potential lyric area may include a part of the musical notes, the histogram is divided into a small area of the N area to distinguish the character area from the note area.

The run length search module 330 performs a vertical projection to find non-zero consecutive run lengths in the divided zones through the zone division module, and when the space between run lengths is less than or equal to the run length, As shown in FIG.

First, for zoning and run-lengths searches, divide the potential lyric area into N zones and perform a vertical projection to find non-zero consecutive run-lengths.

Generally, it is expected to make one run-lengths for each word, but in some cases it is necessary to combine run-lengths as shown in Figure 6 (a). When the space between run-lengths is less than or equal to sl_h (heights of the stave line), sum the run-lengths into one section.

와

이

를 만족할 때,

를

와

로 교체한다. 도 6의 (a)가 (b)와 같이 합해진다.

Wow

this

Lt; / RTI >

To

Wow

. 6 (a) is combined as shown in (b).

With respect to run-length filtering, not all run-lengths come from the lyric area, because high or low notes may be in the latent area. The following filtering rules are required for this situation. Figure 7 shows run-lengths filtering through FR1-4.

The rules FR 1-4 are as follows.

FR1 removes some of the notes that extend across the boundary, FR2 removes run lengths that are less than the height between the lines, FR3, 4 spans the boundary but does not remove if it is on the same line as the previous block, .

The candidate boundary information extraction module 340 can perform a function of extracting a candidate boundary by correcting the noise when there is noise after the binarization processing of the preprocessing unit.

With respect to candidate boundary information extraction, if there is noise after binarization, it is difficult for the boundary to include all of the actual household services. At this time, modify the boundary using the following rule. FIG. 8 shows a process in which the candidate boundary is changed from green to red through LE1 to LE2. These rules LE1 to LE2 will be described as follows.

LE1: If the value obtained by subtracting the run length of the start line from the top edge is less than or equal to 2 * sl_h and the run length of the start line is smaller than the new top edge, the new top edge is changed to the run length of the start line.

LE2: If the value obtained by subtracting the run length of the end line from the bottom edge is less than or equal to 2 * sl_h and the run length of the end line is smaller than the new bottom edge, the new bottom edge is changed to the run length of the end line.

The candidate boundary filtering module 350 is a configuration for filtering the candidate boundary of the candidate boundary information extraction module.

With respect to candidate boundary filtering, the density of pixels is high in the line of sight, often with high density by music symbols, which is anomalous. In this case, the lines are corrected through the following rules FL1 to FL3.

Regarding the rules FL1 to FL3, the candidate lyric area is a text, such as a code or a numeral indicating the length of a note, but there is a portion that is not a lyrics.

FL1: This line is excluded if the horizontal histogram average of one line is less than a certain value of the histogram average of all lines.

FL2: Exclude the target line if it is too close to the below line.

FL3: Exclude if the target line is too far from the text line above.

The lexical region extraction module 360 is configured to display a lyric region and a score region as O or I as a result of filtering through the candidate boundary filtering module. The rules LA1 to LA4 at this time will be described below.

LA1: If the height and width of the outpart are less than the height and width of the inpart, outpart, inpart,

LA2: outpart the height is greater than the height inpart τ values less than l _h / 2, if the width of the outpart less than the width of the inpart outpart and inpart notes

LA3: If outpart height is large inpart higher than the τ values less than l _h / 2, the difference between the right edge of the right edge of the car and the joint part of the left edge of the left edge of the joint part and inpart and inpart less than the predetermined value outpart And inpart are notes

LA4: If the above LA1, LA2, LA3 are not satisfied, outpart is the note inpart,

The control unit 400 controls the pre-processing unit 100, the line-of-sight information extracting unit 200, and the lyrics line extracting unit 300.

Meanwhile, a method using a lyric region extracting apparatus for automatically playing a photographed music score image will be described with reference to FIG.

The controller of the lyric region extracting apparatus for automatically playing the photographed music score image preprocesses the score image (a).

In step (a), the control unit causes the input music image to be binarized (a-1), and the distortion of the music score image is corrected using the binarization result as a result of the binarization process (a-2) .

Next, the control unit extracts the selection information from the preprocessed music score image (b). In step (b), the control unit causes the pentagon to be detected (b-1), and the space and height information of the pentagon is evaluated to extract the pentagon information (b-2).

Next, the control unit sets a potential lyric region in the preprocessed music score image, and searches for and searches for the zonal region and the execution length, and extracts the lyric region by searching for candidate boundary information (c).

10 is a detailed flowchart of a lyric region extraction step of a method using a lyric region extraction device for automatically playing a photographed music score image according to an embodiment of the present invention.

Referring to FIG. 10, step (c) will be described as follows.

In step (c), the control unit sets the area between the five-line and the five-line area as a potential lyric area (c-1). (C-2) divide the potential lyric area, which is set next, into N zones (c-2) and perform vertical projection to find consecutive run lengths other than 0 in the divided zones, When they are less than or equal to each other, the execution length is summed into one zone (c-3).

Next, if there is noise after the preprocessing step of step (a), the control unit may correct the candidate boundary and extract it (c-4). Next, the control unit causes the candidate boundary of step (c-4) to be filtered (c-5), and the filtering result, the lyric area and the score area are displayed as O or I (c-6).

FIG. 2 shows a process of extracting a lyric region from an input music image. First, the binarization is performed and the distortion of the score image is corrected using the line and line information. Then find the line and get information such as the space and height of the line. Then we search for potential lexical domain settings, partitions and run-lengths, and extract run-lengths filtering and candidate boundary information to extract the lexical domain.

The details of each stage are as follows.

Regarding the setting of the potential lyric area, since the lyrics are always located between the pentatons except for the last part, this part is set as a potential lyric area as shown in Fig. The upper part of the lyric area is on the fifth line of the pentagon as much as ss_h (height of stave-space) + sl_h (height of stave-line) and the lower part of the lyric area is ss_h_sl_h on the first line of the pentagon.

For zoning and run-lengths searches, divide the potential lyric area into N zones and perform a vertical projection to find non-zero consecutive run-lengths.

Generally, it is expected to make one run-lengths for each word, but in some cases it is necessary to combine run-lengths as shown in Figure 6 (a). When the space between run-lengths is less than or equal to sl_h (heights of the stave line), sum the run-lengths into one section.

와

이

를 만족할 때,

를

와

로 교체한다. 도 6의 (a)가 (b)와 같이 합해진다.

Wow

this

Lt; / RTI >

To

Wow

. 6 (a) is combined as shown in (b).

With respect to run-length filtering, not all run-lengths come from the lyric area, because high or low notes may be in the latent area. The following filtering rules are required for this situation. Figure 7 shows run-lengths filtering through FR1-4.

의 시작과 끝 행,

: 잠재영역의 위와 아래 가장자리.

The beginning and ending rows,

: The top and bottom edges of the potential area.

With respect to candidate boundary information extraction, if there is noise after binarization, it is difficult for the boundary to include all of the actual household services. At this time, modify the boundary using the following rule. FIG. 8 shows a process in which the candidate boundary is changed from green to red through LE1 to LE2.

는 top edge이고, 는 bottom edge이며,

는 new top edge,

는 new bottom edge이다.
From above,

Is the top edge, Is the bottom edge,

New top edge,

Is the new bottom edge.

With respect to candidate boundary filtering, the density of pixels is high in the line of sight, often with high density by music symbols, which is anomalous. In this case, the lyrics line is modified by the following rule.

Rule FL1:

은 실험적으로 결정된 상수이다.In rule FL1, avg (h) is the average of the vertical histogram of the candidate line,

Is an experimentally determined constant.

Rule FL2:

은 다음 오선

의 윗부분이고,

와

는 라인

의 윗부분과 아랫부분이며,

는 오선

의 아랫부분, 이 규칙은 잠재적 가사 영역에서 첫 번째 라인(i=1)에서 사용된다.

The next line

Lt; / RTI >

Wow

The line

The upper and lower portions thereof,

A penthouse

, This rule is used in the first line (i = 1) in the potential lyric area.

Rule FL3:

는 j번째 오선 아래에 잠재적 가사 영역의 I번째 라인이다.

Is the Ith line of the potential lyric area beneath the jth pentagon.

Regarding the extraction of the lyric region, as a result of filtering through the candidate boundary filtering module, the lyric region and the score region are indicated by O or I.

, marking O and I as lyricsRule LA1:

, marking O and I as lyrics

Rule LA2:

Rule LA3:

Rule LA4: If the above rules (LA1, LA2, and LA3) are not satisfied, I is indicated by lyrics and O by music symbol

는 폭,

는 높이,

는 왼쪽 가장자리,

는 오른쪽 가장자리When in-part I,

The width,

The height,

The left edge,

The right edge

는 폭,

는 높이When out-part is O,

The width,

The height

는 왼쪽 가장자리,

는 오른쪽 가장자리
When it is a joint-part,

The left edge,

The right edge

According to an embodiment of the present invention for eliminating a lyric area at a low calculation amount and a high speed in optical score recognition, a calculation process can be simplified and a strength can be exerted in a mobile environment with limited computing capability.

Further, according to the present invention, it is possible to improve the recognition rate of the musical score by solving the case where the lyrics are touched or overlapped with the musical symbol in the optical musical score recognition, and the optical musical score recognition is performed by digitizing the printed musical score From music to playing, analyzing, comparing, arranging, and composing, you will be able to work fast in various fields.

While the present invention has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. It will be appreciated by those skilled in the art that numerous changes and modifications may be made without departing from the invention. Accordingly, all such appropriate modifications and changes, and equivalents thereof, should be regarded as within the scope of the present invention.

100: preprocessing unit 110: binarization module
120: correction module 200:
210: a pentagon detection module 220: a pentagon information extraction module
300: Lyrics line extraction unit 310: Lyrics area setting module
320: zoning module 330: run length search module
340: candidate boundary information extraction module 350: candidate boundary filtering module
360: Lyrics Region Extraction Module

Claims (8)

A preprocessing unit for binarizing the music score image;
A preference information extracting unit for extracting preference information from the preprocessed music score image; And
And a lyric line extracting unit for setting a potential lyric area in the preprocessed music score image, searching and searching for a zoned area and an execution length, and searching for candidate boundary information to extract a lyric area,
Wherein the pre-processing unit comprises: a binarization module for binarizing the input music image; And
And a correction module for correcting the distortion of the score image using the five-line and line information as a result of the binarization process,
The correction module detects a line segment using the line segment characteristic of the score segment, divides the line segments into respective regional regions between the segments and nodes using the line segment information, and then horizontally And rearranging the area so as to correct the distortion. A lyric area extracting apparatus for automatically playing a photographed music score image.
delete The method according to claim 1,
The preference information extracting unit extracts,
A pentode detection module for detecting pentode; And
And a five-line information extraction module for evaluating the space and height information of the dashed line.
The method according to claim 1,
The lyrics line extracting unit extracts,
A lexical area setting module for setting the area between the pentagon and the pentagon as a potential lexical area;
A zone division module for dividing a potential lyrics area set through the lyrics zone setting module into N zones;
Performing a vertical projection to find successive non-zero run lengths in the zones partitioned by the zone partitioning module, and when the space between run lengths is less than or equal to the straight line height, Length search module;
A candidate boundary information extracting module for correcting and extracting a candidate boundary when there is noise after the binarization processing of the preprocessing unit;
A candidate boundary filtering module for filtering a candidate boundary of the candidate boundary information extraction module; And
A lyric region extraction module for displaying a lyric region and a score region as O or I as a result of filtering through the candidate boundary filtering module; Wherein the lexical region extracting means includes a lexical region extracting means for automatically playing the photographed music score image.
A method using a lyric region extracting apparatus for automatically playing a photographed music score image,
(a) causing the control unit of the lyric region extracting apparatus to preprocess the score image;
(b) extracting the selection information from the preprocessed music score image; And
(c) setting, by the control unit, a potential lyric area in the preprocessed score image, searching for a zone division area and an execution length, filtering the candidates, and extracting a lyric area by searching for candidate boundary information,
Wherein the step (a) comprises the steps of: (a-1) binarizing the score image input by the controller; And (a-2) the controller corrects the distortion of the score image using the five-line and line information as a result of the binarization process,
In the step (a-2), after the control unit detects the pentagon, the control unit detects the bar using the pentatonic characteristic of the score segment, divides the pentagon region into the area regions between the bar and the node And rearranging the regions so as to be horizontally arranged on a region-by-region basis, thereby correcting the distortion.
delete 6. The method of claim 5,
The step (b)
(b-1) causing the control unit to detect a pentode; And
(b-2) extracting the five-line information by evaluating the space and height information of the five-line and the control unit extracting the five-line information.
6. The method of claim 5,
The step (c)
(c-1) causing the control unit to set the area between the five-line and the five-line area as a potential lyric area;
(c-2) causing the control unit to divide the potential lyric area to be set into N zones;
(c-3) The control unit performs vertical projection to find non-zero consecutive execution lengths in the divided areas, and when the space between execution lengths is less than or equal to the five-line height, ;
(c-4) if the noise is present after the preprocessing step in the step (a), correcting and extracting the candidate boundary;
(c-5) causing the controller to filter the candidate boundary of the (c-4) -th step; And
(c-6) causing the controller to display the lyric region and the score region as O or I as a result of the filtering in the (c-5) -th step.

Images