KR102045533B1 - System for recognizing music symbol using deep network and method therefor - Google Patents

Abstract

The present invention relates to a music score recognition system and a method using a deep network, comprising: a photographing unit for photographing an image including a score image; A feature map extractor configured to extract a feature map of the entire image by configuring the image as a sliding layer; A candidate region detection unit that detects a candidate region including a music score from the feature map; And a feature vector extraction unit for extracting feature vectors from the candidate region.

Description

Music score recognition system using deep network and its method {SYSTEM FOR RECOGNIZING MUSIC SYMBOL USING DEEP NETWORK AND METHOD THEREFOR}

The present invention relates to a music score recognition system and a method using a deep network, and more particularly, to a technology for automatically converting, reconstructing, and recognizing music symbols in a machine understandable format for playing music.

The object recognition technology using the convolutional neural network (CNN) is more accurate than the recognition rate of the existing object recognition technology, and researches for recognizing digitized music using the convolutional neural network have been conducted.

In general, sheet music recognition (OMR) systems are used to automatically recognize and reconstruct the read data, convert it into a machine-readable format such as XML, and play music.

However, the technique of capturing musical scores and automatically recognizing musical symbols is difficult to accurately recognize due to large changes in musical styles, symbolic notations and other distortions, and in particular, when the lighting changes, noise is generated to recognize differently from the desired symbols. There is a problem that occurs frequently.

In addition, since musical notation written on the score is different in the notation style and shape of each genre, it is required to develop a technology capable of stably recognizing the musical sign even when the lighting changes and the change in the shape of the musical symbol in consideration of such specificity.

Korean Patent Publication No. 10-2017-0028591

An object of the present invention is to accurately recognize musical symbols from sheet music images, even with changes in ambient lighting.

An object of the present invention is to extract a feature map for a whole score image using a convolutional neural network (CNN).

An object of the present invention is to detect candidate regions containing sheet music symbols using Region Proposal Networks (RPNs), which include a single pooling layer, a softmax classifier, and a box regression analyzer.

According to an aspect of the present invention, there is provided a music score recognition system using a deep network, including: a photographing unit photographing an image including a score image; A feature map extractor configured to extract a feature map of the entire image by configuring the image as a sliding layer; A candidate region detection unit that detects a candidate region including a music score from the feature map; And a feature vector extraction unit for extracting feature vectors from the candidate region.

Preferably, the candidate area detection unit is configured to move each sliding window mapped to a feature vector of 256 dimensions into a convolutinal layer and a pooling layer of fully connected layers for box regression and box classification. ), And constitute a reference set comprising the set accumulation (x1, x2, x3) and aspect ratio (1: 1, 1: 2, 1: 3).

And a training processor for initializing the feature vector detection in the candidate region, controlling to set a new candidate region adjusted through the candidate region detection unit, and controlling to extract the feature vector from the new candidate region through the feature vector extraction unit. It features.

In addition, the present invention provides a method for recognizing a score using a deep network, comprising: (a) receiving an image including a score image through a photographing unit; (B) extracting, by the feature map extractor, the feature map for the image; (C) detecting, by the candidate region detection unit, the candidate region including the music score symbol from the feature map; And (d) extracting the feature vector from the candidate region by the feature vector extractor.

Preferably, the step (c) comprises the step (c-1) of the candidate area detection unit constituting each sliding window mapped to the feature vector of 256 dimensions; (C-2) the candidate region detecting unit applying the sliding window to a convolutinal layer and a pooling layer of fully connected layers; (C-3) forming, by the candidate region detection unit, a reference set including preset accumulations (x1, x2, x3) and aspect ratios (1: 1, 1: 2, 1: 3); And (c-4) detecting, by the candidate region detecting unit, a plurality of candidate regions simultaneously at each sliding window position.

Step (d) comprises: (d-1) the feature vector extractor branching to the last two output layers in the candidate region through a pooling layer; (D-2) the feature vector extractor recognizing the music symbol class and the background class from the feature vector in the first layer through a softmax probability function; And (d-3) calculating a position of each music symbol from the feature vector in the second layer.

According to the present invention as described above, there is an effect that can be applied to the mobile device by accurately recognizing the music symbol from the sheet music image even in the ambient light changes.

According to the present invention, a feature map of a whole score image is extracted using a convolutional neural network (CNN), and a candidate region including a score symbol using region proposal networks (RPNs). There is an effect that can be detected.

1 is a block diagram showing a music score recognition system using a deep network in accordance with an embodiment of the present invention.
2 is an exemplary view showing an input image of a music score recognition system using a deep network according to an embodiment of the present invention.
3 is an exemplary diagram illustrating a sliding layer configuration of a music score recognition system using a deep network according to an embodiment of the present invention.
4 is an exemplary view showing a feature vector derived through a music score recognition system using a deep network in accordance with an embodiment of the present invention.
5 is an exemplary diagram showing a music symbol position derived through a music score recognition system using a deep network according to an embodiment of the present invention.
6 is a flow chart illustrating a music score recognition method using a deep network in accordance with an embodiment of the present invention.
7 is a flow chart showing the detailed process of step S30 of the music score recognition method using a deep network in accordance with an embodiment of the present invention.
8 is a flowchart illustrating a detailed process of step S40 of the music score recognition method using a deep network according to an embodiment of the present invention.

Specific features and advantages of the present invention will become more apparent from the following detailed description based on the accompanying drawings. Prior to this, the terms or words used in the present specification and claims are defined in the technical spirit of the present invention on the basis of the principle that the inventor can appropriately define the concept of the term in order to explain his invention in the best way. It should be interpreted to mean meanings and concepts. In addition, when it is determined that the detailed description of the known function and its configuration related to the present invention may unnecessarily obscure the subject matter of the present invention, it should be noted that the detailed description is omitted.

1 is a block diagram showing a music score recognition system 100 using a deep network in accordance with an embodiment of the present invention.

As shown in FIG. 1, the music score recognition system 100 using a deep network according to an exemplary embodiment of the present invention may include a photographing unit 101, a feature map extracting unit 102, a candidate region detecting unit 103, and a feature vector. It is configured to include an extraction unit 104 and the training processing unit 105.

The photographing unit 101 receives an image including a sheet music image through photographing. At this time, the photographing unit 101 is provided in any one of a camera or a mobile terminal having a camera, the input image may be a score image as shown in FIG.

The feature map extractor 102 configures an image applied from the photographing unit 101 as a 3 * 3 sliding layer to extract a feature map of the entire image. At this time, the feature map is mapped to a feature vector of 256 dimensions as shown in FIG. 3.

The candidate region detection unit 103 detects a candidate region including the music score symbol from the feature map. At this time, the candidate region detection unit 103 uses the sliding window mapped to the feature vectors of 256 dimensions for convolutional and pooling layers of fully connected layers for box regression and box classification. ), And construct a reference set that includes the set accumulation (x1, x2, x3) and aspect ratio (1: 1, 1: 2, 1: 3).

In addition, the candidate area detector 103 is configured to derive a plurality of candidate areas at the same time at each sliding window position, and the maximum number of possible candidates for each position corresponds to the number of reference reference boxes.

The feature vector extractor 104 extracts a feature vector from the candidate region. At this time, the feature vector extractor 104 branches to the last two output layers in the candidate region through a pooling layer, and the first layer recognizes a music symbol class and a background class from the feature vector through a softmax probability function. In the second layer, the position of each music symbol is calculated from the feature vector.

Here, the feature vector extracting unit 104 derives the feature vector as shown in FIG. 4, and the music symbol position is derived as shown in FIG. 5.

The training processor 105 initializes the feature vector detection in the candidate region, controls to set a new candidate region adjusted through the candidate region detection unit 103, and from the new candidate region through the feature vector extraction unit 104. Control to extract feature vectors.

On the other hand, the music symbols included in the feature vector are black note / white note, flag, flat, sharp, natural, quarter rest, 16 It includes any of a semi-quaver rest, whole note, rest, full rest and half rest, beam, or clef.

Also, the music symbols included in the feature vectors are accidental, align, appoggiatura, bar line, bass, bass, beat and stave. Brackets, coda, cue, chord / chord, clef (warm clef, C clef), low clef (bar clef, F [bass] clef) , Treble clef (G clef, G [treble] clef), dot, duet (duet (reference quartet, quintet, sextet, solo, trio)), duration, end / ending, elongation (fermata), the note's tail, flat, grace, key signature, beat / rhythm / measure, natural sign, musical note, two minutes Half note, a quarter (crotchet note), an eighth note, sixteenth note, nudge, eighth note (octave), percussion instrument ( Vibration / hit, percussion) height / rhythm / Pitch, normal (note that there is no point in the note, plain), quartet (quartet / quartette (reference duet, quintet, sextet, solo, trio)), quintet (quintet / quintte (reference duet, quartet) , sextet, solo, trio)), five quintuplets, rest / rest, rhythm / rhythm, beats [triple, three-four] to 2 [3/4] Beat, chord root, chord score, a vocal, excerpt (a short ~), film music (a film ~), seven septuplet, quintuplet , sextuplet, triplet), quartet (sextet / sextette, note duet, quartet, quintet, solo, trio), six doubles (sextuplet, note (quintuplet, septuplet, sextuplet, triplet)) ), F [sharp], seam / slur, solo, note duet, quartet, quintet, sextet, trio, vertical line of notes, stave / vowel system (staff (pl . staves), letters / numeric scores / tablatures, speed / rhythm / 1 tempo per minute, tie, time signature / time signature, trio, note duet, quartet, quintet, sextet, solo), triplet (triplet, reference quintuplet, septuplet, sextuplet), or a voice / voice adjustment / voice (voice) of any one may include.

Hereinafter, a music score recognition method using a deep network according to an embodiment of the present invention based on the system described above with reference to FIG. 6 will be described below.

First, the photographing unit receives an image including a sheet music image by photographing (S10).

Subsequently, the feature map extractor configures the image as a 3 * 3 sliding layer to extract a feature map for the entire image (S20).

Subsequently, the candidate area detection unit detects a candidate area including the music score symbol from the feature map (S30).

Subsequently, the feature vector extracting unit extracts a fixed length feature vector from the candidate region (S40).

Subsequently, the training processor initializes the feature vector detection in the candidate area (S50).

Subsequently, the candidate area detection unit sets a new candidate area adjusted (S60).

In step S30, the feature vector extracting unit extracts the feature vector from the new candidate region.

Preferably, step S30 includes step S31 to step S34 as shown in FIG. 7.

After operation S20, the candidate region detector configures each sliding window mapped to the feature vectors of 256 dimensions (S31).

Subsequently, the candidate region detection unit applies a sliding window to a convolutinal layer and a pooling layer of fully connected layers for box regression and box classification (S32).

Subsequently, the candidate region detection unit constructs a reference set including preset accumulations (x1, x2, x3) and aspect ratios (1: 1, 1: 2, 1: 3) (S33).

Then, the candidate area detector detects a plurality of candidate areas at the same time at each sliding window position (S34).

Preferably, the step S40 is configured to include steps S41 to S43 as shown in FIG.

After operation S30, the feature vector extractor branches to the last two output layers in the candidate region through a pooling layer (S41).

Subsequently, the feature vector extractor recognizes the music symbol class and the background class from the feature vector in the first layer through a softmax probability function (S42).

The feature vector extracting unit calculates a position of each music symbol from the feature vector in the second layer (S43).

As described above and described with reference to a preferred embodiment for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described as described above, it is a deviation from the scope of the technical idea It will be understood by those skilled in the art that many modifications and variations can be made to the invention without departing from the scope of the invention. Accordingly, all such suitable changes, modifications, and equivalents should be considered to be within the scope of the present invention.

100: Music Recognition System Using Deep Network
101: filming unit
102: feature map extractor
103: candidate area detection unit
104: feature vector extraction unit
105: training processor

Claims (6)

In the music score recognition system using a deep network,
A photographing unit which photographs an image including a sheet music image;
A feature map extractor configured to extract the feature map of the entire image by configuring the image as a sliding layer;
A candidate region detector for detecting a candidate region including a musical score symbol from the feature map;
A feature vector extractor extracting a feature vector from the candidate region; And
A training processor for initializing feature vector detection in the candidate region, controlling to set a new candidate region adjusted through the candidate region detection unit, and controlling to extract a feature vector from the new candidate region through the feature vector extraction unit; But
The feature vector extraction unit,
Branching to the last two output layers in the candidate region through a pooling layer, the first layer recognizes the music symbol class and background class from the feature vector through a softmax probability function, and from the feature vector on the second layer. Music score recognition system using a deep network, characterized in that for calculating the position of each music symbol. The method of claim 1,
The candidate region detection unit,
Each sliding window mapped to a feature vector of 256 dimensions is applied to a convolutinal and pooling layer of fully connected layers for box regression and box classification, and the set accumulation (x1). A score recognition system using a deep network, comprising: configuring a reference set including x2, x3) and aspect ratios (1: 1, 1: 2, 1: 3). delete In the music score recognition method using a deep network,
(a) receiving an image including a music score image by the photographing unit;
(b) a feature map extractor extracting a feature map for the image;
(c) a candidate region detecting unit detecting a candidate region including a music score from the feature map;
(d) a feature vector extracting unit extracting a feature vector from the candidate region;
(e) a training processor initiating feature vector detection in the candidate region;
(f) controlling the training processor to set a new candidate region adjusted through the candidate region detection unit; And
(g) controlling the training processor to extract the feature vector from the new candidate region through the feature vector extractor;
In step (c),
(c-1) the candidate region detector constituting each sliding window mapped to the feature vectors of 256 dimensions;
(c-2) the candidate region detecting unit applying the sliding window to a convolutinal layer and a pooling layer of fully connected layers;
(c-3) the candidate region detecting unit constructing a reference set including preset accumulations (x1, x2, x3) and aspect ratios (1: 1, 1: 2, 1: 3); And
(c-4) the candidate region detecting unit detecting a plurality of candidate regions at the same time at each sliding window position,
In step (d),
(d-1) the feature vector extractor branching to the last two output layers in the candidate region through a pooling layer;
(d-2) a feature vector extractor recognizing a music symbol class and a background class from the feature vector through a softmax probability function in a first layer; And
(d-3) the feature vector extracting unit calculating a position of each music symbol from the feature vector in the second layer;
Music score recognition system using a deep network comprising a. delete delete

Images