JP6985543B1 - How to create music information from sheet music images and their computing devices and programs - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method, a computing device and a program for generating music information from a musical score image. SOLUTION: This is a method of creating music information from a musical score image, which includes a step of inputting a musical score image, a step of extracting at least one measure from the musical score image by using an optional deep learning model, and a step of extracting music information from the musical score image. Optionally, a step of correcting the position of the five lines in each bar of at least one bar, and a step of arbitrarily identifying the notes in each bar of at least one bar using multiple deep learning models. Includes the step of creating musical information from the identified notes. [Selection diagram] Fig. 1

Description

The present invention relates to a method, a computing device, and a program for creating music information from a musical score image.

Optical Music Recognition (OMR) is a research field that studies how to read a musical score in a document by a computer. The goal of OMR is to use a computer to read and interpret the score to create a machine-readable version of the written score. The OMR pipeline is classified into four stages: preprocessing, musical symbol recognition, notation reconstruction, and final expression construction (Non-Patent Document 1).

Regarding specific processing, Patent Document 1 recognizes the staff, notes, symbols, their positions, etc. in the score from the image data obtained by reading the image of the score, and based on the recognition result, the musical sound is produced. A musical score recognition device that generates information such as pitch, sound timing, and sound time is disclosed. In this device, (1) pre-processing (staff / bar line recognition, tilt correction, staff erasing and beam erasing), (2) object recognition (external rectangle search and matching processing), (3) event recognition processing. (Pitch recognition and sound length recognition processing) and performance data creation, (4) Automatic performance (MIDI data creation and output) are performed.

Patent Document 2 uses a plurality of musical score symbol recognition methods for an image acquisition means for acquiring an image including information on a musical score on paper from an image reading means and a musical score symbol included in the image acquired by the image acquisition means. It has a musical score symbol recognition means that recognizes and outputs a plurality of musical score symbol recognition results, and the musical score symbol recognition means performs five-line recognition processing, paragraph recognition processing, musical score symbol recognition processing, and processing of the entire musical score. , It is characterized by detecting multiple candidates for the relationship between musical score symbols, and using various information for each candidate, estimating the one that is valid in terms of musical score and selecting the relationship between one musical score symbol. Disclose the score recognition device to be used.

Patent Document 3 describes a pre-recognition processing unit that recognizes a part of musical score symbols based on a musical score image, a correction unit that corrects the recognition result of the pre-recognition processing unit, and a recognition result that is corrected by the correction unit. The pre-recognition processing unit has a main recognition processing unit that recognizes other musical score symbols based on the musical score image, and the pre-recognition processing unit recognizes beat symbols, bar lines, clef symbols and key signatures. The recognition processing unit discloses a musical score recognition device characterized by recognizing notes and rests.

Devices using these techniques have the steps of recognizing staves and bar lines as disclosed, then removing the staves and bar lines, and recognizing notes and the like using techniques such as OCR. ing.

Examples of these conventional OMR devices include Aruspix, Audiovis, Gamera, PhotoScore (used in the musical score software Sibelius) and the like. However, there has been a need to improve OMR accuracy.

An approach using deep learning has been tried to improve this OMR accuracy. Deep learning has transformed the analysis and use of information about data, including static and dynamic images, such as photographs, images, and moving images. The current status and potential of deep learning has been discussed in many documents (eg, Non-Patent Documents 2 and 3). Not only the classification of an arbitrary object but also its position can be determined by various deep learning models such as YOLO (Non-Patent Document 4) and SSD (Non-Patent Document 5). The use of both classification and location makes it a useful and versatile model for many applications, including object detection in recorded video and real-time object detection in live images. Currently, its applications are expanding in various fields and will be widely studied in the future.

Specifically, several deep learning models have been applied to OMR. Calvo-Zaralagoza et al. (Non-Patent Document 6) used the so-called Connectionist Temporal Classification (CTC) loss function to locate the musical symbols of the score in the score. Zhiqing Huang et al. (Non-Patent Document 7) propose an end-to-end detection model based on a deep convolutional neural network and feature fusion. The model can directly process the entire image and then output the symbol category and note pitch and duration. Further, a method of processing note data on a five-line line in time series using a convolutional neural network (CNN) and a recurrent neural network (RNN) is also disclosed (Patent Document 4).

Further, in Non-Patent Document 8, Pacha et al. Use a deep learning model for recognizing measures in order to recognize a musical score image. Then, it is shown that music symbol recognition is possible by using a deep learning model for recognizing symbol categories and notes, as in the method disclosed in Non-Patent Document 7.

Japanese Unexamined Patent Publication No. 6-103416 Japanese Unexamined Patent Publication No. 2012-13809 Japanese Unexamined Patent Publication No. 2015-56149 International Publication No. WO2018 / 194456

Rebelo, Ana; Fujinaga, Ichiro; Paszkiewicz, Filipe; Marcal, Andre RS; Guedes, Carlos; Cardoso, Jamie dos Santos (2012). "Optical music recognition: state-of-the-art and open issues" (PDF). International Journal of Multimedia Information Retrieval. 1 (3): 173-190.doi: 10.1007 / s13735-012-0004-6. Yutaka Matsuo: The Difficulties of Deep Learning and Artificial Intelligence, Journal of the Society of Systems Control and Information Science, Vol.60, No.3, pp.92-98, 2016 Z. Zhao, P. Zheng, S. Xu and X. Wu, "Object Detection With Deep Learning: A Review," in IEEE Transactions on Neural Networks and Learning Systems, vol.30, no.11, pp.3212-3232 , Nov. 2019, doi: 10.1109 / TNNLS. 2018.2876865. Redmon, J., Farhadi, A., YOLOv3: An Incremental Improvement., ArXiv 2018, arXiv: 1804.02767 Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, CY, Berg, AC, SSD: Single Shot MultiBox Detector., In European Conference on Computer Vision; Springer: Cham, Switzerland, 2016; pp.21-37., doi: 10.1007 / 978-3-319-46448-0_2. Jorge Calvo-Zaragoza and David Rizo, End-to-End Neural Optical Music Recognition of Monophonic Scores, Appl. Sci., 2018, 8, 606 Zhiqing Huang, Xiang Jia and Yifan Guo, State-of-the-Art Model for Music Object Recognition with Deep Learning, Appl. Sci., 2019, 9, 2645 https://www.youtube.com/watch?v=Mr7simdf0eA

An object of the present invention is to identify a note from a musical score image with high accuracy.

Specifically, the first aspect of the present invention is a method of creating music information from a musical score image, which includes a step of inputting a musical score image, a step of extracting at least one measure from the musical score image, and at least the above. Provided is a method including a step of identifying a musical note in each bar of one bar and a step of creating musical information from the identified musical note. In this method, notes can be identified with high accuracy, in particular, by going through a step of extracting at least one measure from a musical score image.

In some embodiments, the at least one measure may be extracted by a deep learning model. Preferably, each of the at least one measure is extracted along the staff frame, particularly the top and bottom lines. This has the effect of facilitating the correction of the staff, which will be described later.

In some embodiments, it further comprises correcting the position of the staff within each bar of the at least one bar. This staff position correction step includes, although optional, a step of correcting the inclination of a staff to make the entire input score image horizontal. Further, the extraction of each bar by the deep learning may be performed on the horizontally corrected musical score image. Furthermore, a step of horizontally correcting the staff in each of the extracted measures may be included. Although the position of the staff of each measure horizontally corrected in this way is not limited, it may be corrected by the method described in Examples 6 and 7. This staff correction step is made possible by extracting each measure with a deep learning model, and has a remarkable effect on a staff whose distortion is non-uniform in an image such as a photograph of a musical score.

In some embodiments, it may further include identifying notes within each bar of the at least one bar. In some embodiments, the note within each bar of the at least one bar may be identified using a plurality of deep learning models. By combining deep learning models corresponding to a plurality of feature categories, it has a remarkable effect that various clef symbols and the like can be expressed. Also, rather than training and using one large deep learning model that discriminates between multiple feature types, combining deep learning models from multiple feature categories is more efficient and accurate during learning and inference. Turned out to be better. In addition, it is considered that the point that each measure extracted according to the present invention was standardized and used as learning data also contributed to the improvement of the accuracy of learning and inference, and these have a remarkable effect.

In some embodiments, the plurality of deep learning models are processed in parallel. As a result, the inference time can be remarkably shortened, and the present invention will exert more and more excellent effects with the improvement of CPU / GPU / TPU performance in the future.

In some embodiments, the music information is selected from the group consisting of XML files, musicXML files, MIDI files, mp3 files, wav files, and musical scores.

A second aspect of the present invention is a computing device for creating music information from a musical score image, which includes an input unit for inputting the musical score image, a bar extraction unit for extracting at least one measure from the musical score image, and a bar extraction unit. A five-line correction unit that corrects the position of the five lines in each measure of the at least one measure, and a note identification unit that identifies the notes in each measure of the at least one measure using a plurality of deep learning models. , The music information creation unit that creates music information from the identified musical score, the at least one measure is extracted by the deep learning model, the plurality of deep learning models are processed in parallel, and the music information is obtained. , XML files, musicXML files, MIDI files, mp3 files, wav files, and musical scores. This computing device has the remarkable effect obtained from the first aspect.

A third aspect of the present invention is a program for creating music information from a musical score image, which includes an input unit for inputting the musical score image, a bar extraction unit for extracting at least one measure from the musical score image, and at least the above-mentioned bar extraction unit. Identification with a five-line correction unit that corrects the position of the five lines in each measure of one measure, and a note identification unit that identifies the notes in each measure of at least one measure using multiple deep learning models. A music information creation unit that creates music information from the said musical score, the at least one measure is extracted by the deep learning model, the plurality of deep learning models are processed in parallel, and the music information is XML. Provided is a program selected from a group consisting of a file, a musicXML file, a MIDI file, an mp3 file, a wav file, and a musical score. This program also has the remarkable effect obtained from the first aspect.

According to one aspect of the present invention, a remarkable effect of identifying a note from a musical score image with high accuracy is produced.

It is a flowchart which shows the process of the method of one Embodiment of this invention. It is a figure which shows the result of applying the bar deep learning model to a plurality of musical score images which concerns on Example 1 of this invention, and was made to recognize each bar. It is a figure which shows that the deep learning model of a plurality of feature categories which concerns on Example 5 of this invention was applied to various analysis regions, and the type and position of a feature type were identified. It is a figure which shows the result of leveling the inclined musical score image with respect to the staff staff which concerns on Examples 6 and 7 of this invention. It is a figure which shows the result of having corrected the position and the interval of the staff notation which concerns on Example 7 of this invention. It is a figure which carried out this method which concerns on Example 8 of this invention, created MusicXML from a musical score image, and displayed it on two kinds of general musical score software. FIG. 5 is a diagram showing a photographic image of a tilted musical score according to Example 8 of the present invention, a MusicXML created from the image using the present method, and the result displayed on general musical score software. It is a figure which shows the result of OMR processing of the photographic image of the inclined sheet music which concerns on the comparative example of this invention by the existing technique.

Hereinafter, embodiments of the present invention will be described in detail.
Terms and definitions
Image (image)
Images or images used herein (these terms are interchangeably used herein and have the same meaning unless otherwise indicated) are any kind that can be analyzed by the methods of the invention. It is an image of. The image may be two-dimensional, such as a photograph or screen display, or it may be a three-dimensional image, such as a hologram. Examples of images include images, videos, photographs, etc., which may be computers, servers, storage media (eg, RAM, ROM, cache, SSD, hard disk), or the like, respectively. Or together, it can be displayed and / or saved as a file (eg, .jpg, .jpeg, .tiff, .png, .gif, .mp3, mp4, or .mov file).

Information The information used herein is relevant to the data. The difference is that the information resolves the uncertainty. Data can represent redundant symbols, but approaches information through optimal data compression. Information can be encoded in various forms for transmission and interpretation (eg, information may be encoded in a sequence of codes or transmitted over a signal). This general concept of information can be applied herein. With respect to the form of information, the information may be in documented form, digitized form, audio form, video form, or a combination of such forms and is not limited to any particular form. In Optical Mark Recognition (OMR) technology, information may be provided, for example, as a musical score or any other medium in a digitized, readable, or audible form. Both visualized and audible are acceptable.

Area unit In the present specification, the area unit may be each measure. In OMR technology, a region unit is also referred to as a staff (Staff) containing five lines (staff), but in the present specification, "staff" and "staff" may be compatible. ) May be a measure containing one or more staff members (measure; in the present specification, "measure" and "major" may be compatible).

Positional Reference The positional reference used herein may be one or more of the five lines of the staff.

Feature model The feature model used herein may be any feature model as long as the feature model can extract information from the image. The feature model may be, for example, a general feature model, preferably an AI model, more preferably a machine learning model, and even more preferably a deep learning model. Multiple models may be used for inference in images or at least one analysis area (including each bar). The number of feature models used is 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 50, 100, 350, 500, 750, or 1000 or more. May be good. A number between any two of the above values is also included. The number of feature models used for extracting measures is preferably 1, and the number of feature models used for inference of an analysis region including bars is not particularly limited, but is preferably 1 to 100, and more preferably 1. It is ~ 25, more preferably 1-10, still more preferably 1-5.
Specific examples of the feature model disclosed in the present specification when creating music information include a bar model, a Clef model, a Body model, an Accidental mode, an Arm / Beam model, and / or a Rest model. It is not particularly limited to these. Details of these models will be described later.

Feature Categories The feature categories used herein correspond to the associated feature model. Unless otherwise indicated, feature categories relate to this feature of the model used. The feature category can be of any kind as long as the model can extract data about this feature from the image. The data obtained may be arbitrary and not always useful. Therefore, not all of the extracted data will be used for subsequent analysis. Each category may be selected manually or automatically by another model. This makes it possible to facilitate the automatic generation of music information from the score image.
Since some of the embodiments of the present invention have created their own feature categories, they are referred to as Clef, Accidental, Body, Arm / Beam, and Rest to indicate the feature categories.

Feature Types As used herein, the feature categories of each feature model include one or more feature types. Further, the types of feature types are not particularly limited, and any type may be used alone or in combination. Note feature types may also be annotated using a combination of these one or more feature categories and positional criteria. As used herein, note feature types include those of notes and rests. Therefore, references to notes may include both notes and rests.
In one embodiment of the invention, the Clef feature category includes a treble clef, a clef, and an octave shift feature type. Accidental feature categories include # (sharp), ♭ (flat), and natural feature types. Body feature categories include note black circles, dotted black circles, half-white circles (half notes), dotted half-white circles, all black circles (whole notes), and dotted whole note feature types. Arm / Beam feature categories are unconnected quarter stems (up and down), flagged 8th (up and down), 8th (top, bottom) (start, middle, end) connected, 16th. Includes feature types of continuous parts (top, bottom) (start, middle, end). The Rest feature category includes feature types for full rests, half rests, quarters, 8th and 16th rests. These feature types are shown in Table 5. See FIG. 3 for the specific shape.

Sheet music (score)
Scores include handwritten or printed or electronically readable formats written using musical symbols to indicate the pitch, rhythm, and / or chords of musical works of songs and musical instruments. The term score is a general alternative (more general) term for sheet music. The score or score used herein may be commonly referred to as the score. Examples of musical score images used herein include any form of a visualized or digitized musical score image.

Staff (staff) and measure (bar)
The staff (staff) consists of five horizontal lines and four spaces, each of which represents a different pitch. Staff include, for example, the following embodiments: Appropriate musical symbols are placed on the staff according to the corresponding pitch and function according to the intended effect. The notes are arranged for each pitch. The pitch is determined by the vertical position on the five lines and is played from left to right. Which note is in which position is determined by the clef at the beginning of the staff. The clef identifies a particular line as a particular note, and all other notes are determined relative to that line. The ground staff (large staff notation) is used when two staff members connect music or one performer plays at the same time. Generally, the upper staff (staff) uses the treble clef, and the lower staff uses the bass clef. For example, piano music is written by two staff members, one for the right hand and one for the left hand. The bar line is used to divide the notes on the five lines into bars.
In musical notation, a bar or measure (hereinafter sometimes referred to as a bar) is a segment of time corresponding to a specific number of beats, each beat is represented by a specific note value, and the boundaries of the measures are. Indicated by vertical bar lines. Dividing the music into bars provides a regular reference point for positioning in the composition. In addition, since each measure of the staff can be read and played at once, the music can be followed more easily.

Staff line (five lines)
Each staff consists of 5 lines (staff). Lines and spaces can be numbered from bottom to top. The note can be placed on a line (a line that passes through the center of the note ball) or in a space. This space includes four inner spaces and two outer spaces at the top or bottom.
In one embodiment of the invention, the scales (steps) are assigned in correspondence with the treble clef or the treble clef, with the positions of the five lines of the staff as the position reference. In this specification, scales A (la), B (shi), C (do), D (re), E (mi), F (fa), and G (so) are used in principle.

Music Symbols (Characteristics) Type Examples of music symbols are: Lines (eg, five lines, bar lines, braces, parentheses), notes and rests (eg, whole note, half note, quarter note, eighth note, 16th note). , 32nd, 64th, 128th, 256th, beam, dot or rest), accidental (flat, sharp, natural, double flat, double sharp, etc.), key signature (eg flat) Key signature, sharp note), quarter note (Demiflat, Flat and Half, Demi Sharp, Sharp and Half), rhythmic note (eg, simple rhythmic note displayed by number of beats and beat type, common time, tempo) Key signatures such as), note relationships (eg, ties, slurs, grissands, grissands, taplets, chords, arpeggio chords), dynamics (eg pianissimo, pianissimo, piano, meso piano, mesoforte, forte, fortissimo, etc.) Fortissimo, Sforzand, Crescendo, Diminuend), playing style symbols (eg, Staccattissimo, Staccattissimo, Staccattissimo, Staccattissimo, Staccato, Tenuto, Fermata, Accent, Marquardt), decorative sounds (eg, Trill, Upper) Key signatures, lower modents, gurupets, apodge aturas, acchiaccatura), octave symbols (eg ottaba), iterations and coder (eg tremolo, repetition symbols, simulation symbols, volta brackets, dacapo, darsegno, segno, coder) ), Or other musical notes.
In one embodiment of the invention, several types are modified or created to address the problem of generating information from images of musical scores. The feature types used in this embodiment are listed in Table 5.

Direction Unless otherwise stated, the directions specified herein have the meaning commonly used in the art. Horizontal and vertical orientations are provided for any image. Either the horizontal direction or the vertical direction may be arbitrarily set, but the positions may be provided as x positions and y positions by each feature model. These positions may be used directly or may be reset with reference to any of the position criteria.

Overview
Comparison with existing technology In the technology disclosed in Patent Documents 1 to 3, the staff and the bar line are recognized, then the staff and the like are erased to recognize the clef and the like, and the bar line is used at that time. It reconstructs the recognized note information. Therefore, the technical idea is different from the present invention in which each measure is focused on, each measure is extracted, and the note information is reconstructed thereafter. The process of correcting the inclination of the staff is also described, but there is no description of correcting the position of the staff in each measure.

Non-Patent Document 6 proposes an end-to-end detection model that directly processes the entire image and outputs the symbol category, pitch, and duration. It is not clear how to generate music information from the duration. In addition, the technical idea of reconstructing note information by focusing on measures and extracting each measure is not disclosed.

In Patent Document 4, the note data on the five lines is processed in time series by using the convolutional neural network and the recurrent neural network, but the note data is not created by extracting each measure and processed in time series. ..

In Non-Patent Documents 7 and 8, one end-to-end deep learning detection model is used for detecting note symbols, etc., but it is considered to use multiple models for detecting each symbol category (feature type). It has not been. We need to increase the number of symbol categories and types, but we haven't specifically suggested how to annotate and reconstruct the results. Further, although it is disclosed that the step of each note is identified by the position information of the staff, the technical idea of extracting each measure and correcting the position for each measure is not disclosed.

When using multiple models for the task of detecting and analyzing any of the feature types belonging to each symbol category, it is necessary to find the optimal procedure and processing configuration for generating music information from the output of multiple models. There is.

Non-Patent Document 8 shows that a measure in a musical score image can be recognized by a deep learning model. However, the recognized bar is a ground staff (staff: one that includes two staff), unlike the bars described herein (segments separated by each bar line in one staff). There is. Further, the purpose of recognizing a measure is to provide structural information for identifying whether or not the image is a music image. Further, the recognition of the bar of Non-Patent Document 8 recognizes the one larger than the area of the staff including the bar, and is not a model of recognizing only the area of the staff as much as possible. Therefore, it is different from the technical idea of extracting each measure and correcting the staff information using the unit, or recognizing each clef with a deep learning model. Furthermore, as the author admits, there is no method for reconstructing the obtained clef information and finally making it into music information.
Specific embodiments will be described in detail below.

Embodiment 1
The first embodiment of the present invention is a method for creating music information from a musical score image, and provides a method including a step of extracting at least one measure from the musical score image. This method may include, for example, a step of inputting a musical score image or a step of creating music information from the notes in each bar of the at least one bar. Hereinafter, the process of the present method and the process which may be optionally included will be described in detail based on the flowchart (FIG. 1) explaining the process of the embodiment of the present invention. The order of these steps is subject to change.

(1) Musical score image input process (process S100)
In the score image input step (1), a score image is input. The image of the score image is any image as defined above. The score contains all or part of the music. The score may contain multiple pages, and each page may be the target. The input is performed in any manner readable or recognizable by the following computing devices.

(2) Bar extraction step (step S200)
In the measure extraction step (2), at least one measure is extracted from the musical score image. As used herein, the term "bar" is defined above as a regional unit and may be referred to as a bar or measure. In the present specification, each bar is not preferably that of the ground staff (staff), but refers to a unit within one staff (segments separated by each bar line in one staff). Measures may be extracted as regional units. In addition, notes may be identified for each bar (for example, in bar units) for the extracted measures. It may include a step of reconstructing the extracted measures after analysis to create music information.

The number of measures is not particularly limited, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 50, 100, 150, 200, 250, 50, 100. , 150, 200, 250, 500, 1000 or more. Further, the number may be larger or lower than the above number, and may be between any two of them.

(2-1) Bar Extraction Machine Learning Model Each bar may be extracted by a machine learning model. In this case, any type of bar model may be used. Further, the number of bar models is not particularly limited and may be 1, 2, 3, 4, 5, 10 or more. Further, the number may be larger or smaller than the above number, or may be any number in between. Preferably, the number is 1 in terms of the processing time required to acquire each measure.

Each measure model is preferably an AI model, more preferably a machine learning mode, and even more preferably a deep learning (deep learning; deep learning and deep learning are used interchangeably herein) model. good. Any combination thereof is acceptable and they may be used alone or in combination.

Functions of the bar model include classification and positioning of bar types. Classification and positioning can be performed using one feature model such as SSD or YOLO model. However, a plurality of models may be used in combination. The same applies to other feature models described later.

In Example 1, it was shown that each measure in the score can be recognized very efficiently by applying a deep learning model that classifies the measures shown in Table 1 into three types (x0, x1, y0). .. Therefore, it is shown that the present invention exerts a remarkable effect of being able to recognize each measure efficiently (eg, 94% to 100%).

(2-2) Analysis area based on each measure and step of setting at least one position reference in each measure The analysis area is set based on each measure. This analysis area may be a part of each bar, or may include a part or the whole of each bar. The analysis region may have any shape. The shape of the analysis region may be the same as the shape of each bar, or may be different.

Further, the size and number of analysis regions derived from each measure are not particularly limited, and may be provided in substantially the same manner as the above-mentioned region unit. In this embodiment, the upper margin and the lower margin are set to 1 or 1.2 times the vertical width of the staff. As a result, not only the notes in the staff of the bar but also the notes located on the lower and upper sides can be recognized as musical symbols belonging to each bar.

Set at least one position reference. The position reference is as defined above. The type of position reference is not particularly limited. The type of position reference may be any type as long as the position reference can be used for mapping or annotating musical symbols described later. It is preferably one or more lines of the staff in the staff. You can also apply the spacing between the staves to provide position reference lines on the upper and lower sides of the staff to identify note steps in the upper and lower regions.

The number of position references is not particularly limited and may be, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 50, 75, or 100 or more. .. Further, the number may be larger or smaller than the above number, or may be between any two.

Preferably, each of the at least one measure is extracted along the staff frame, particularly the top and bottom lines. This point is different from the method disclosed in Non-Patent Document 8, and has an excellent effect of facilitating the correction of the staff described later.

(3) Position reference correction process
(3-1) Overall image tilt correction step (step S301)
The position reference correction step (3) is a step of correcting the position of the staff in each bar of at least one bar. This staff position correction step includes, although optional, a step of correcting the inclination of a staff to make the entire input score image horizontal. This method of correcting the inclination of the staff of the entire musical score image is preferably performed before the bar extraction step (2). This makes it possible to extract each measure more efficiently.

This overall image tilt correction can be performed, for example, in the following steps.
1. 1. The input image is grayscaled and the edges of the image are extracted using the Canny method.
2. 2. A straight line is detected using the Hough method.
3. 3. Calculate the tilt angle of the longest straight line to obtain the rotation angle of the image.
4. Rotate the entire image at the obtained rotation angle.

Step (3-1) can effectively correct the tilt of the entire image, but the bar tilt is not uniform in each area of the image, as in a photograph of a musical score (eg, see FIGS. 4A and 4B). On the other hand, although it became possible to extract each measure, there was still a problem in uniformly determining the slope of the staff, which is the position reference. When correcting the staff with the existing technology, the whole staff is corrected uniformly or the inclination of each staff (existing across measures) is corrected only. Therefore, in order to solve the problem of providing a more accurate position reference, the inclination of the staff in each of the following measures may be corrected.

(3-2) Each measure inclination correction step (step S302)
The correction of the staff inclination of each measure can be basically performed in the same manner as in (3-1) Overall image inclination correction. For those with different staff slopes in each region of the image, it is preferable to individually correct the staff slope in each measure. However, the staff in each measure may be selected by the threshold value of a straight line extending in the horizontal direction. This correction of the staff slope for each measure has a remarkable effect not found in the existing technology (eg, FIG. 4C). With this correction, it is possible to provide the staff as a position reference with higher accuracy even when the distortion of the staff such as a photograph of a musical score is not uniform in the image.

(3-3) Staff position / interval correction step (step S303)
The position of the staff is calculated assuming that the bar extracted by the bar model is extracted at the correct position (especially along the upper and lower lines of the staff). The measures extracted in the step (2) in this way have the dual effect of not only extracting each measure but also serving as an index for determining the position reference of each measure. In addition, the analysis area can be set to any size at the top and bottom using the height of the staff as an index. Since the upper and lower analysis areas are wider depending on the score, it may be possible to select whether or not to use the widely detected feature model. The staff assumed in this way may differ from the actual staff. The alpha and beta variables may be introduced to correct this deviation. alpha may be a deviation from the center of the staff, and beta may be a value for correcting the spacing between the staffs. These two values can be automatically obtained using the following algorithm.

1. 1. Let 1 be the vertical width of the entire image (the image of the staff + the upper and lower parts of which the height size of the staff is arbitrarily expanded). The range of alpha is looped between -0.03 and 0.03 in 0.001 increments, and beta is looped between -0.005 to 0.005 in 0.001 increments at each value.
2. 2. Using each alpha and beta, the staff notation is overwritten in the image.
3. 3. The image is grayscaled and the area of the black part of the Gaussian threshold-processed image is obtained.
4. Considering that the area becomes the minimum when the staffs overlap, the minimum value is obtained, and the values of alpha and beta at that time are used for correction.

By this (3-3) staff position / interval correction step, the position of each line of the staff can be accurately positioned to provide a more accurate position reference. Therefore, the music information obtained by accurately determining the step of each note is more useful, and has an excellent effect that the burden of the subsequent correction process by a human can be reduced.

As described above, in one embodiment of the present invention, a method of horizontally correcting an image and correcting the position and spacing of the staff is preferably used. Examples of methods used for automatic correction include the Canny method, the Hough method, Gaussian threshold processing (Example 6), and the unique staff position spacing correction method disclosed herein (Example 7). Note steps and accidentals (eg, #, ♭, natural) by individually correcting the position of the staff, even if the distortion of the staff is not uniform in the image, such as a photo of a score. Etc. can be identified with higher accuracy.

(4) A step of identifying the notes in each measure using a plurality of deep learning models (note identification step S400).
(4-1) Use of Multiple Feature Models and Feature Types In this step, multiple feature models are applied to the analysis area based on each bar for inference. By combining deep learning models corresponding to multiple feature categories, it is possible to express various clef and the like. The feature model may be preferably an AI model, more preferably a machine learning mode, and even more preferably a deep learning model. Any combination thereof is acceptable and they may be used alone or in combination.

The number of feature models is not particularly limited, and is 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 50, 100, 150, 200, 250, 500, 100 or more. You may. Further, the number may be larger than the above number, may be smaller than the above number, or may be a number between any two.

Feature categories (eg, Example 5) relate to feature models that recognize arbitrary musical symbols. Arbitrary musical symbols include the default musical symbol itself and self-made ones, such as those related to each part of the note. Specific examples include the accidental, arm / beam, body, clef, and rest categories shown in Table 2, and a plurality of feature types can be set for each category.

As shown in Example 2, using multiple feature models for inference has several advantages over using a single feature model.

The plurality of feature models may be processed in parallel or in series. However, it is preferred that the plurality of feature models be processed in parallel in order to reduce the time required for inference, as shown in Examples 3 and 4.

(I) Training performance A plurality of feature models with a small number of feature types could be learned more easily than a single feature model with a large number of feature types. Further, the second embodiment demonstrates that the recognition accuracy of each feature type is improved when the feature category is selected so as to have a small number of feature types. As described above, according to the present invention, there is a remarkable effect that the learning performance of the feature model can be improved.

(Ii) Inference performance The number of inference processes increases as the number of extracted area units increases. In the case of a computer setting with a large number of CPUs and GPUs, which will occur in the near future, it is conceivable to use this setting to process inference processing in parallel and shorten the processing time. For example, when the number of analysis regions is 100 and the number of feature models for inference is 10, it is necessary to complete 1,000 independent inference processes. As the number of CPUs and GPUs increases, it is expected that the time required for all inference processing will be shortened when multiple feature models are used in parallel. As shown in Example 3, the processing time is not simply reduced to 1/8 even if parallel processing is performed by an 8-core CPU, so it is possible to actually test with the current verifiable architecture and measure the processing time. is necessary. Therefore, Examples 3 and 4 in which the processing times were actually compared demonstrated the usefulness of parallel processing. In the fourth embodiment, the processing time is about one tenth of the time when the CPUs are processed in series, and it is demonstrated that the processing time can be remarkably shortened by the parallel processing on the GPU. Using multiple CPUs / GPUs for inference by multiple feature models is considered to be superior in terms of total processing time. Therefore, in a preferred embodiment of the present invention, the parallel processing has a remarkable effect that the time required for the inference processing can be shortened.

(4-2) Step of mapping and aligning the positions of the plurality of feature types in each feature model Each inferred by each feature model (eg, accidental, arm / beam, body, clef, rest model). Feature types are mapped. This mapping may be performed using the coordinate system used in the feature model or using a positional reference. In addition, a combination of coordinate system and position reference may be used to map each feature type.

Each feature type may be aligned horizontally, vertically, or bidirectionally. The feature types of one feature category may be aligned, the feature types of one or more feature categories may be aligned, or the feature types of all feature categories may be aligned.

The alignment direction is not particularly limited, and may be either a horizontal direction or a vertical direction. Further, the alignment direction may be one direction or two or more directions.

One or more feature types may be excluded before, during, and / or after alignment.

(4-3) A process of annotating notes by analyzing each feature type using the staff position (position reference) Each feature type is analyzed using at least one position reference, the staff position. They may be used in sequence for note annotation (identification; these may be used interchangeably). The direction of analysis may be set arbitrarily, and may be horizontal or vertical. As for the aligned feature types, some feature types may be excluded from the analysis target, but they may be analyzed sequentially.

The feature type analyzed may be influenced by at least one pre-analyzed feature type from at least one feature model among the plurality of feature models. The feature category of at least one pre-analyzed feature type may be the same as or different from the feature category of the feature type being analyzed. In this way, the annotated feature type obtained as a result of the analysis is analyzed and annotated in a specific direction while the preceding feature type affects the subsequent feature types of the same feature category or different feature categories. You may.

Specifically, in Example 8, each characteristic type of accredental and clef corresponds to at least one previously analyzed feature type.

In a preferred embodiment of the invention, each feature type aligned horizontally or vertically and each feature type aligned vertically or horizontally overlaps each of the new note feature types. Annotate. When all positions of feature types are aligned horizontally or vertically, each feature type to be analyzed is at least vertically or horizontally overlapped from at least one feature model among the plurality of feature models. Annotation may be performed using one feature type.

Specifically, horizontal feature-type sorting of each measure may be performed. You may specify the staff number as 1 or 2 to make a series of staff measures (measures), and take out the measures one by one from the front. Then, all the feature types included in each measure are sorted in the horizontal direction (x) (forward direction). As an element that affects each annotation, each note may be annotated while updating the current state of Clef and the Accidental table (a table that teaches which scale has a sharp or flat). The Accidental table inputs the initial value of the states (which specifies which major or minor), and may reflect the state of the immediately preceding fifths when analyzing the next measure.

It is preferable to analyze each horizontally sorted feature type in order from the front. The analysis can be categorized according to which feature category each type is in.

A. Clef category If the feature type being analyzed is Clef category G or F (cf0 or cf1), the state of Clef is changed.

B. Accidental category If the feature type being analyzed is the Accidental category, the Accidental table is modified by combining the positional criteria.

C. Rest category If the feature type being analyzed is the Rest category, annotate it according to the Rest type and add the element to the output list.

D. Body category (notes are identified by feature types that overlap vertically)
If the feature type being analyzed is the Body category, chords are detected. Then, in order to specify the length of the note by the Arm / Beam type, it is preferable to sort and list the feature types that overlap in the vertical direction. If the Rest type is included in it, it is preferable to specify Voice according to its position (Voice1 can be set if it is at the bottom, and Voice2 can be set if it is at the top). If it is in the middle position, it may be added to the output list by deciding whether to add it as an element before or after the body type according to the position before and after.

Body types can be annotated separately according to the number and position of feature types that overlap in the vertical direction. When a plurality of body types are included, chords can be assigned according to the rules of the musicXML file.

Case 1: When both the bottom and top feature types are Arm / Beam If there are 3 or more Body types, the target one and the one belonging to the lower Arm / Beam (downward stem). The distance between the above Arm / Beam and the one belonging to the above Arm / Beam (of the upward stem) can be calculated and assigned to the closest one. At that time, it is preferable that the one belonging to the lower Arm / Beam is assigned to Voice1 and the one belonging to the upper Arm / Beam is assigned to Voice2.

Case 2: When the bottom is Rest It is preferable to assign Rest to Voice1 when the bottom is Rest, and to assign one or more Body types to Voice2.

Case 3: When the top is Rest It is preferable to assign Rest to Voice2 when the top is Rest, and to assign one or more Body types to Voice1.

Case 4: When the top is Arm / Beam When the top is Arm / Beam, the cases are classified according to the type of Body type. Annotate notes in combination with Arm or Beam, such as feature types bd0 to bd3, and annotate those that do not have Arm and Beam, such as bd4 to bd5, respectively. At this time, Voice may be set to Voice 1 and may be appropriately changed in the Voice adjustment step described later.

Case 5: When the bottom is Arm / Beam Even when the bottom is Arm / Beam, the case is classified according to the type of Body type. Annotate notes in combination with Arm or Beam, such as feature types bd0 to bd3, and annotate those that do not have Arm and Beam, such as bd4 to bd5, respectively. At this time, Voice may be set to Voice 1 and may be appropriately changed in the Voice adjustment step described later.

Case 6: When both the top and bottom are Body In this case, the feature types of bd4 to bd5 are assumed. However, as a result of not recognizing the Arm / Beam feature type or Rest feature type (for example, when it is located at the bottom or top of the bar and cannot be recognized, or when it is not detected by the inference of the feature model). There may be cases. Therefore, when a person of bd0 to bd3 is included, it is preferable to appropriately supplement Arm / Beam. Also in this case, it is preferable that the notes are assigned to Voice1.

In the above-mentioned annotation of each Body type, it is preferable to pass the current Clef and the accidental table as arguments to annotate the note feature type. Then, each Body type step is identified according to the relative distance to the position of the staff.

The analyzed Body, Arm, and Rest types can be put in the exclusion list to prevent them from being analyzed again. Beam can also be used again for analysis of adjacent Body types.

The feature types sorted horizontally in this way are such that certain previously analyzed specific types (Clef, Accidental) subsequently affect the feature type, and the vertically overlapping feature types are vertical. It is preferable to perform annotation using a feature type that affects (eg, Arm / Beam).

In a preferred embodiment, the plurality of feature types and the position reference (staff position) are used in combination to annotate a new note feature type. The number of note feature types is preferably at least 10 times, more preferably at least 100 times, and even more preferably at least 1000 times the total number of the plurality of feature types and the position reference.

(4-4) Voice adjustment process measure of each note has a note length determined by the musical piece. In this step, it is confirmed whether or not the Voice of the note group identified in the above (4-3) note annotation step is correctly assigned. In cases 1 to 3, each note is assigned to Voice 1 or Voice 2, but in cases 4 to 6, each note is assigned to Voice 1 for convenience. Therefore, in this state, the length of each note belonging to Voice1 and Voice2 is calculated in consideration of the chord. Then, if the note length is longer than the specified note length of the measure, the Voice is adjusted. For example, the Body type having Arm / Beam on the upper side may be set to Voice2, and the remaining Body type (eg, bd4 to bd5) may be set to Voice1. Further, the Body type having Arm / Beam on the lower side may be set to Voice1, and the remaining Body type (eg, bd4 to bd5) may be set to Voice2. Further, the whole note (bd4 to bd5) may be set to Voice2. This adjustment step may be repeated.

Example 5 shows an example of creating a new note feature type using a small number of feature types of a small number of feature models. Example 5 demonstrates the remarkable effect of the present invention that a large number of note feature types can be identified and annotated by combining a relatively small number of feature types in a plurality of categories.

(5) A process of creating music information from notes in each measure (music information creation process S500)
(5-1) Step of assembling the data of each feature type annotated for each area unit In this step, the data derived from the note feature type annotated for each bar is assembled. During assembly, one or more feature types used for annotation may be deleted. The deleted feature type may affect another feature type during annotation, but may not be needed to generate the information.

The assembly method is not particularly limited. The direction of assembly may be the same as during analysis or annotation. However, the assembling direction may be opposite to that during analysis or annotation. In addition, since annotations may be processed in time (that is, assembled in chronological order), it is preferable to assemble data in the same direction during annotation.

In a preferred embodiment of the invention, the annotated note feature type data is assembled in the time direction.

(5-2) A step of connecting data related to one or more measures in series and / or in parallel to create music information . Data obtained in one or more measures are connected in series or in parallel to generate information. .. In some cases, the number of measures may be one. In this case, the annotated note feature type data contained in one bar may be used.

When having a plurality of measures, the plurality of measures may be connected in series or in parallel. Further, the data connected in series may be further connected in series, the data connected in parallel may be further connected in series, or the data connected in parallel may be further connected in series for music. Information may be generated. This makes it possible to handle scores with multiple staff.

In the case of a score that includes a large staff, connect the staff for the right hand in series and in parallel (those with different stages) to make Staff 1, and connect the staff for the left hand in series and in parallel (those with different stages). It may be staff 2.

The number of connected measures is not particularly limited, and for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 50, 100, 150, 200, 200, It may be 250, 500, 1000, 2500, 5000, 10000, 25000, 50000, or 10000 or more. Further, it may be larger or smaller than the above number, or it may be a number between any two.

The method of connecting the note data of each measure is not particularly limited. The note data may be directly connected or indirectly connected. When indirectly connected, other data or material may be inserted between the data, or the same data may be repeatedly inserted to generate music information.

In one embodiment of the invention, the measures to be connected may be influenced by feature types (eg, key signatures and accidentals) within the preceding measures.

In one embodiment of the invention, the feature type of the connected bar (eg, ellipsis, etc.) may affect the preceding bar. Alternatively, the measures may be simply connected as they are.

In one embodiment of the invention, the music information is selected from the group consisting of XML files, musicXML files, MIDI files, mp3 files, wav files, and musical scores.

In one embodiment of the present invention, the obtained music information may be implemented as a final product (eg, MusicXML, MIDI, mp3 file, wav file, musical score) as it is.

In Example 8, an annotation of each note and an example of creating a MusicXML file are shown, and it is demonstrated that the method of the present invention exerts a remarkable effect in creating music information from a musical score image.

Embodiment 2
A computing device for carrying out the method of the present invention to create information from an image Embodiment 2 relates to a computing device for carrying out the method of the present invention to create information from an image.

A second embodiment of the present invention is a computing device for creating music information from a musical score image, and provides a computing device including a bar extraction unit that extracts at least one bar from the musical score image. This computing device has, for example, an input unit for inputting a musical score image, a five-line correction unit for correcting the position of the five lines of each measure of the at least one measure, and a plurality of notes in each measure of the at least one measure. It may include a note identification unit identified using the deep learning model of the above, or a music information creation unit that creates music information from the identified musical note. Here, the at least one measure is extracted by the deep learning model, the plurality of deep learning models are processed in parallel, or the music information is the XML file, the musicXML file, the MIDI file, the mp3 file, the wav file, and the music information. It may be a computing device selected from a group of musical scores.

Examples of computing devices include, but are not limited to, RAM, ROM, cache, SSD, and hard disk. It also includes any form of computing device, such as one on the cloud, one on a server, one on an on-premises computer, and so on.

The input unit for inputting the score image executes the (1) score image input step of the first embodiment. The bar extraction unit executes the (2) bar extraction step of the first embodiment. The staff correction unit executes the (3) position reference correction step of the first embodiment. The musical note identification unit executes (4) the musical note identification step of the first embodiment. The music information creation unit executes (5) the music information creation step of the first embodiment. Further, as the preferred embodiment of each part, the embodiment described in Example 1 shall be applied mutatis mutandis.

Embodiment 3
A program for implementing the method of the present invention to create information from an image Embodiment 3 relates to a program for implementing the method of the present invention to create information from an image. The program of the present invention includes the whole program or a part thereof as long as the method of the present invention can be carried out.

The program of the present invention can be written in any language as long as the method of the present invention can be carried out. Examples of the language include, but are not limited to, Python, Java, Kotlin, Swift, C, C #, C ++, PHP, Ruby, JavaScript, Scala, Go, R, Perl, Unity, COBOL and the like.

The third embodiment is a program for creating music information from a musical score image, and provides a program including a bar extraction unit for extracting at least one bar from the musical score image. This program has an input unit for inputting a musical score image, a five-line correction unit for correcting the position of five lines in each measure of at least one measure, and a plurality of deep learning of notes in each measure of at least one measure. It may include a note identification unit identified using a model, or a music information creation unit that creates music information from the identified musical note. Here, the at least one measure is extracted by the deep learning model, the plurality of deep learning models are processed in parallel, or the music information is the XML file, the musicXML file, the MIDI file, the mp3 file, the wav file, and the music information. It may be a program selected from a group of musical scores.

The input unit for inputting the score image executes the (1) score image input step of the first embodiment. The bar extraction unit executes the (2) bar extraction step of the first embodiment. The staff correction unit executes the (3) position reference correction step of the first embodiment. The musical note identification unit executes (4) the musical note identification step of the first embodiment. The music information creation unit executes (5) the music information creation step of the first embodiment. Further, as the preferred embodiment of each part, the embodiment described in Example 1 shall be applied mutatis mutandis.

Other Embodiments According to one embodiment of the present invention, there is a method of creating information from an image, in which a step of extracting a region unit from an image, at least one in an analysis region and the region unit based on the region unit. A process of setting one position reference, a process of applying a plurality of feature models to the analysis area to perform inference, and each feature model performing the inference for a plurality of feature types, the plurality of features in each feature model. A process of mapping and aligning each position of a feature type, a process of analyzing and annotating each feature type using the at least one position reference, a process of assembling data of each feature type annotated with respect to the area unit, Provided is a method comprising at least one step of connecting the data about one or more of the domain units in series and / or in parallel to create information. Further, according to one embodiment of the present invention, there is provided a computing device for carrying out the above method and creating information from an image. Examples of computing devices include, but are not limited to, RAM, ROM, cache, SSD, and hard disk. It also includes any form of computing device, such as one on the cloud, one on a server, one on an on-premises computer, and so on. Further, according to one embodiment of the present invention, a program for carrying out the above method to create information from an image or a recording medium on which this program is recorded is provided. The recording medium may be a non-temporary computer-readable recording medium.

The description "AB" in the present specification includes A and B. Further, although the process and the like according to the present invention have been described in each embodiment, the description is not limited to these, and various changes can be made.

Hereinafter, the present invention will be described in more detail with reference to Examples, but the present invention is not limited to the following Examples.

Example 1
Training and reasoning of deep learning models for measures in sheet music First, we trained a YOLOv5 measure model using 47 overall sheet music drawings (each sheet music contained several to about 50 measures), and mAP. @. 5 (an indicator of accuracy in the model for feature type) achieved 0.95. This bar model category has bar feature types x0, x1, and y0, which are bars starting with a treble clef (G clef) and a bar clef (F clef), respectively, as shown in Table 1 below. The measures starting with and the rest of the measures were shown. The labelImg software (https://github.com/tzutalin/labelImg) was used to create the training data, and a bounding box was assigned to each type in each image. At that time, the bounding box was set along the top and bottom lines of the staff. In addition, training data, test data, and verification data for training were adjusted by Robotflow (https://app.roboflow.com/).

Next, it was applied to the inference in the musical score image that was not used for training this bar model. FIG. 2A shows the inference results in the PDF-derived image obtained by scanning a part of the score of "Sarabande and Variations" by Handel. FIG. 2B shows the inference result in the photographic image obtained by using the camera of the smartphone with the same score.

As a result, 100% of the measures in each musical score image were recognized and extracted with the accuracy of the inference of 0.91 to 0.95.

In addition, a part of the score of Beethoven's "Sorrowful Second Movement" (used for training this bar model) was also extracted with 100% of the bars recognized and extracted with the accuracy of the inference of 0.92 to 0.93. ..

Furthermore, in the score image of Bach's "Minuet", which is another score not used for training this bar model, one bar out of 66 bars is recognized as overlapping at x0 and x1, and two bars are recognized. Was recognized as a fusion. Also, one bar contained one adjacent note. The accuracy of the inference was 0.79 to 0.93, and overall, about 94% of the measures were correctly recognized. The results are shown in FIG. 2C.

This proved that this bar model is useful because it can efficiently extract bars even in PDF-derived images and photographic images of musical scores that were not used for training.

Example 2
Training Performed Using Multiple Deep Learning Models A plurality of YOLOv5 models corresponding to each music symbol feature category (described in Example 5 below) were trained. In addition, by expressing a combination of a plurality of feature types, the number of musical symbol (note) feature types expressed as a whole has increased dramatically, which is an advantageous effect.

Each measure was extracted, the analysis area was determined based on it, enlarged to make the size constant (416 x 416 pixels), and training data was created. To create the training data, a bounding box was assigned using the labelImg software as in Example 1.

For feature categories (details will be described in Example 5), accidental, arm / beam, body, clef, and rest categories were created, and a plurality of feature types were set for each category. The number of feature types was 3 for accidental, 8 for arm / beam, 6 for body, 5 for clef, and 5 for rest, which were smaller than those of the above one deep learning model. The number of images used for training (total number of data for training, testing, and verification) was 199 for accidental, 546 for arm / beam, 537 for body, 149 for clef, and 611 for rest, respectively. After all, it was more than an order of magnitude less than the number of training data in normal deep learning. For example, in the handwritten digit data set MNIST, the number of training sets is 60,000 and the number of test sets is 10,000. Therefore, depending on the type of feature type, it was possible to train deep learning with a smaller number of datasets than previously considered. It is considered that this is because the present invention can express a large number of feature types by combining a small number of feature types. Therefore, it was one of the remarkable effects that deep learning training could be carried out without degrading the quality of training.

As a result of the training, mAP @. In No. 5, the accidental model was 0.99, the arm / beam model was 0.99, the body model was 0.94, the clef model was 0.99, and the rest model was 0.99, respectively. The training was basically carried out using a Google Colorory with a batch size of 16 and a GPU (16G) with 500 epochs. The initial weights used were those used in the previous training. Therefore, it is actually the result of 2 to 4 trainings (transfer learning). The results so far are shown in Table 2.

These results demonstrate that training a relatively small number of feature types with multiple deep learning models using relatively small training data may yield superior results. Combining deep learning models from multiple feature categories is better in terms of learning and inference execution, accuracy, etc. than training and using one large deep learning model that discriminates between multiple feature types. May be. Therefore, it is shown that training and using the multi-machine learning model of the configuration of this embodiment is more advantageous than the conventional method and has a very remarkable effect.

Example 3
Comparison of processing time required when processing multiple models in series or parallel We recognized and processed each measure from the score image using the deep learning model created so far, and prepared an analysis area with the same size. Then, the procedure for creating analysis data was automated by applying the above models of the five feature categories to each analysis area. Then, the time required for processing was measured. At this time, the processing times of the models of the five feature categories were compared by processing them in series or in parallel. The results are shown in Table 3.

The processing times were compared using three types of musical score images. The computer used was an iMac Pro (processor: 3.2 GHz, 8-core Intel Xeon W; memory: 64 GB 2666 MHz DDR4). The average time taken for processing in series was 153.8 seconds, 121.5 seconds, and 138.1 seconds for the minuet (66 bars), Sarabande (48 bars), and the second movement of sadness (58 bars), respectively. It was almost proportional to the number of. The average time taken for parallel processing was 81.3 seconds, 63.0 seconds, and 75.4 seconds for minuet, sarabande, and the second movement of sadness, respectively, which were also almost proportional to the number of measures. By the parallel processing, the processing time for minuet, sarabande, and the second movement of sadness was reduced to 52.9%, 51.9%, and 54.6%, respectively, by about half.

Even in the serial processing, the processing time was not reduced to 1/8 because it is considered that the processing was distributed to 8 cores and the processing progressed to some extent, but the time required for inference of the deep learning model was significantly shortened. Since the number of inferences this time applies 5 feature models to each of about 50 measures, it was necessary to process about 250 processes. In this embodiment, processing was performed by one CPU (8 cores). However, since it is considered that a configuration having a plurality of CPUs and GPUs will become mainstream in the future, the processing configuration of the present invention can further shorten the processing time as the number of CPUs / GPUs increases. Therefore, the configuration of this example has a remarkable effect.

Example 4
Processing speed on GPU We examined whether the processing time could be shortened by actually using GPU. The processing in Example 3 was carried out by AWS EC2 instance g4dn. The processing time was measured using metal. g4dn. The CPU / GPU configuration of the metal was NVIDIA T4 Tensor Core GPU 8 pieces, vCPU 96 pieces, RAM 384GiB and the like. The process was programmed to use GPUs in series or in parallel. The results are shown in Table 4.

The average processing time for processing Menuet scores in series with GPUs is 70.9 seconds, which is higher than the average of 153.8 seconds when using CPUs in series and the average of 81.3 seconds when processing in parallel. It was short. The average processing time in parallel was 16.4 seconds, which was about 1/4 of the processing time in series processing. This processing time is about one tenth of the time when the CPUs are processed in series, and it was demonstrated that the processing time can be remarkably shortened by the parallel processing on the GPU. Therefore, Example 4 shows that the effect of the present invention is further enhanced by processing GPUs in parallel. The greater the power of a computer (eg, the capacity or number of CPUs, GPUs, etc.), the shorter the time it takes to process multiple models in parallel, and the applicability and performance of the present invention grows with increasing computer power. Significantly improved.

Example 5
Creating new note feature types using a small number of feature types in a small number of feature models As shown in Table 2, there are five feature categories and feature types used for training and inference of deep learning models (three are not used). ), There were 3 types of Accidental, 6 types of Body, 8 types of Arm / Beam, and 5 types of Rest. It is shown in Table 5 and FIG.

For the treble clef, 25 scales from D3 to G6 were assigned based on the position of the staff, and for the treble clef, 25 scales from F1 to B4 were assigned. Depending on the location of the body, 2 × 25 × 6 (number of types of body) = 300 types of variations can be expressed. In addition, the length of each note is determined by the type of Arm and Beam (whole notes do not take Arm / Beam, and half notes take only am0 or am1). In addition, three types of Beam, start, middle, and end, are expressed by the position of the chain. Therefore, 300 x 2 (2 types of whole notes) + 300 x 2 (2 types of half notes) x 2 (am0 or am1) + 300 x 2 (type of black circle) x (4 (type of Arm) + 4 (type of Beam) x 3 (start, middle, end)) = 11,400. Since there are three types of Accidental, it does not necessarily apply to all scales, but 11,400 x 3 = 34,200. Therefore, from 19 feature types, about 30,000 new note feature types called musical notes can be expressed. Furthermore, considering chords, since chords are any combination of 2, 3, 4, and 5, the number of the feature types that can be expressed increases dramatically, and 100,000 types of single notes and chords can be expressed lightly. .. Therefore, we have demonstrated the remarkable effect of this example that notes, which are a large number of new note feature types, can be identified and annotated by combining a relatively small number of feature types in multiple categories. A specific annotation method will be described in Example 7.

Example 6
Correction of tilted score image Figure 4A is an image of a photograph taken with the score of Sarabande tilted. Since the staff not function as a position reference unless it is in the horizontal state, the entire score image was first leveled (Fig. 4B). The procedure was as follows.

1. 1. The input image was grayscaled and the edges of the image were extracted using the Canny method.
2. 2. A straight line was detected using the Hough method.
3. 3. The tilt angle of the longest straight line was calculated to obtain the rotation angle of the image.
4. The entire image was rotated at the obtained rotation angle.

In the overall image obtained, each measure was not yet completely leveled (the central part was highly leveled, but the upper and lower parts still needed correction). Each measure was leveled again in the same manner as in the above procedure except that the selection was made by the threshold value of the straight line extending in the lateral direction (FIG. 4C). It was found that each feature type was recognized when the obtained image was inferred by the feature model (Fig. 4D).

This result not only corrects the tilt of the entire screen, but also corrects the tilt for each region unit (bar), which is an element of this embodiment, by a position reference, thereby improving the accuracy of the invention. Play.

It was found that this leveling made it possible to easily correct the inclination of the staff notation, which was a problem in the conventional method, and to efficiently carry out the implementation of the present invention.

Example 7
The staff was used as the correction position reference for the position and spacing of the staff. The position of the staff was calculated assuming that the bar extracted by the bar model was extracted at the correct position. Then, the analysis area was set at the upper and lower parts as 1.2 times the height of the staff. As described in the actual annotation, since the upper and lower analysis areas are wider depending on the score, it is possible to select whether or not to use the widely detected feature model. Here, the positions of the initial values of the staff were different from the actual ones as shown in FIG. 5A. In order to correct this deviation, alpha and beta variables (coefficients) were introduced. alpha was the deviation from the center of the staff, and beta was the value to correct the interval between the staffs. These two values were automatically obtained using the following algorithm.

1. 1. The vertical width of the entire image (staff + an image with 1.2 times the height of the staff at the top and bottom) was set to 1. The range of alpha was looped between -0.03 and 0.03 in 0.001 increments, and beta was looped between -0.005 and 0.005 in 0.001 increments at each value.
2. 2. Using each alpha and beta, the staff was overwritten in the image.
3. 3. The image was grayscaled and the area of the black part of the Gaussian threshold-processed image was obtained.
4. Considering that the area is the smallest when the staffs overlap, the minimum value was obtained, and the values of alpha and beta at that time were used for correction.

The correction result is shown in FIG. 5B. By executing this automatic correction function at the time of annotation of each measure, it has become possible to identify the scale of notes with high accuracy. This made it possible to further improve the effect of the present invention.

Example 8
Annotation of each note and creation of MusicXML file The main points of the annotation method are briefly explained below. Each measure was extracted by the deep learning measure model, and the process of removing the measures that were partially overlapped and recognized was automatically performed based on the overlapped position. After that, the measures that were lined up in parallel for each staff member were taken out and connected in series to obtain the original data for each staff member.

8-1 Horizontal feature type sorting Staff numbers were specified as 1 or 2 to make a continuous list of staff measures. Then, the measures were taken out one by one from the front. Then, all the feature types included in each measure were sorted in the horizontal direction (x) (forward direction). Each note was annotated while updating the current Clef state and the Accidental table (a table that teaches which scale has a sharp or flat) as elements that affect each annotation. In the Accidental table, the initial value of the states (which specifies which major or minor) is input, and when the next measure is analyzed, the state of the immediately preceding fifths is reflected.

8-2 Analysis of each feature type in order from the front The feature types sorted in the horizontal direction were analyzed in order. The analysis was divided into cases according to which feature category each type was in.

A. When the feature type in the Clef category analysis was Clef category G or F (cf0 or cf1), the state of Clef was changed.

B. Accidental category If the feature type in the analysis was the Accidental category, the Accidental table was modified in combination with the positional reference.

C. If the feature type during Rest category analysis was Rest category, it was annotated according to the Rest category and the element was added to the output list.

D. Body category (notes are identified by feature types that overlap vertically)
When the feature type under analysis was the Body category, chords were detected and feature types that overlap vertically were sorted and listed in order to specify the note length by Arm / Beam type. If the Rest type is included in it, Voice is specified according to its position (Voice1 is set if it is at the bottom, and Voice2 is set if it is at the top). If it is in the middle position, it is decided whether to add it as an element before or after the body type according to the position before and after, and it is added to the output list.

The Body type was annotated according to the number and position of the feature types that overlap in the vertical direction. When multiple Body types were included, chords were assigned according to the rules of the musicXML file.

Case 1: When both the bottom and top feature types are Arm / Beam
Case 2: When the bottom is Rest
Case 3: When the top is Rest
Case 4: When the top is Arm / Beam
Case 5: When the bottom is Arm / Beam
Case 6: When both the top and bottom are Body In each Body type annotation, the current Clef and accidental table are passed as arguments to annotate the note feature type.

The analyzed Body, Arm, and Rest types were put in the exclusion list to prevent them from being analyzed again. Beam was also used again for analysis of adjacent Body types.

The feature types sorted horizontally in this way are such that certain previously analyzed specific types (Clef, Accidental) subsequently affect the feature type, and the vertically overlapping feature types are vertical. Annotations were performed using feature types that affect (eg, Arm / Beam). The scale here is alpha and beta, and the position of the scale is corrected for each measure.

The annotation result of the adjustment measure of 8-3 Voice was verified. In cases 4 to 6 above, all the notes were split into Voice1. As a result, if the total length of the annotated notes exceeds the length determined for the bar, the Voice is changed. Specifically, a note having a downward stem was assigned to Voice 1, and a note having an upward stem was assigned to Voice 2. The length of the notes in the measure was recalculated for each Voice, and if the length of the notes in the measure of Voice1 still exceeded the determined length, all the notes were assigned to Voice2.

8-4 Combining each measure in series The data of each measure was connected in series to create the data of the entire staff. The created data was in the form of Elegant Tree (ET), and the elements were registered to structure the data.

8-5 Creation of MusicXML file A MusicXML file was created by converting ET-structured note data into XML using a function that converts it into an XML file.

Result In FIG. 6, the staff 1 of the score image of Bach's menuet in FIG. 2C is identified and converted into XML by the method of the present invention, and the XML file is read and displayed by Sibelius (FIG. 6A) and MuseScore (FIG. 6B). It is the result of making it. The XML file prepared as shown in FIG. 6 could be read and displayed by Sibelius, MuseScore, and Finale (not shown; adjustment of display measures is required).

Next, the accuracy of the annotation was evaluated. The result of displaying the XML shown in FIG. 6 on each score software was compared with the original image image of FIG. 2C. The results are summarized in Table 6.

For Staff 1, the measures were recognized with an accuracy of 97% (32/33), demonstrating the high accuracy of measure extraction. New note feature type scales (based on Clef type and position criteria staff), notes (including length), chords (including length), new note feature types identified by combining individual feature types and position criteria. For Chords (all matches), the accuracy was 98% (125/128), 95% (122/128), and 100% (1/1), respectively. Accidentals (both scales and symbols match) were also recognized 100% (3/3).

For Staff 2, the bar was recognized with an accuracy of 97% (32/33). The accuracy of the scale (step), note (Note), and chord (Chord) was 95% (71/75), 95% (71/75), and 100% (1/1), respectively. Rests (Rest) were recognized as 40% (2/5), and accidentals (Accidental) were recognized as 50% (1/2).

From these results, it was shown that the accuracy of the notes annotated by the method of this example was extremely high, and it was proved that this example had a remarkable effect.

Furthermore, we examined whether the original image can be created not only from the image digitally created from PDF, but also from the image of the score taken as a photograph, which is likely to be actually used. At this time, since it is considered that the staff notation of the photographic image is not horizontal in many cases, XML conversion was performed from the inclined photographic image as shown in FIG. 7A. The obtained results were displayed in the score of Staff 1 using Sibelius (FIG. 7B).

As shown in Table 6, the measures were recognized with an accuracy of 96% (23/24). The accuracy of steps, notes, and chords was 87% (135/156), 86% (134/156), and 78% (29/37), respectively. Rests (Rest) were recognized as 64% (16/25), and accidentals (Accidental) were recognized as 71% (10/14).

In particular, Sarabande contained relatively complex chords in 37 places in Staff 1, but it was a surprising result that they recognized those chords with 78% accuracy. Demonstrated a remarkable effect.

As a comparative example, FIG. 8 shows the result of inputting the score of FIG. 7A into the existing OMR application PhotoScore2020 and executing the OMR process. As shown in FIG. 8, it was not possible to obtain correct note information from a tilted photographic image with the existing technology. Furthermore, since only the PDF image can be analyzed at present in MuseScore3, the photograph of FIG. 7A was converted to PDF and OMR processing was performed, but "unsuccessful" was output and analysis could not be performed at all.

Therefore, the fact that the notes could be recognized with high accuracy (about 86%) even from a position reference (staff) that is not horizontal and is from a photograph as an image image demonstrates a further remarkable effect of this embodiment. ..

Example 9
Reproduction of sound from MusicXML It was confirmed whether or not sound is reproduced from MusicXML produced in the present invention using general software.

It was confirmed that the XML files of the minuet and Sarabande confirmed in Example 8 were read by MuseScore3 and Sibelius First, and the sound was reproduced using the sound source reproduction function.

It was also confirmed that it is possible to output as an mp3 file, a wav file, and a midi file by using the Export function of MuseScore3. Then, it was confirmed that the mp3 file and the wav file were played back on the computer and the sound was output. It was also confirmed that the midi file was read into Logic Pro software and the sound was reproduced.

The image-derived information creation method of the present invention is useful in the field of OMR. In addition, the image-derived information creation method using the deep learning model of the present invention generally includes images of, for example, automatic driving, robot operation, medical diagnosis, medical device (endoscope, catheter) operation, product inspection, and the like. It is useful in the field of operation and judgment by using.

Claims (11)

It is a method of creating music information from a score image,
The process of inputting a score image and
The process of extracting at least one measure from the score image and
The step of identifying the notes in each bar of at least one bar without erasing the staff , and
A method comprising the step of creating musical information from the identified musical note. The method of claim 1, wherein at least one measure is extracted by a deep learning model. The method of claim 1 or 2, further comprising correcting the position of the staff within each bar of at least one bar. The method according to any one of claims 1 to 3, wherein the musical note in each bar of at least one bar is identified by using a deep learning model. The method according to any one of claims 1 to 4, wherein the musical note in each bar of at least one bar is identified by using a plurality of deep learning models. The method of claim 5, wherein the plurality of deep learning models are processed in parallel. The method according to any one of claims 1 to 6, wherein the music information is selected from the group consisting of an XML file, a musicXML file, a MIDI file, an mp3 file, a wav file, and a musical score. A computing device for creating music information from musical score images.
The input section for inputting the score image and
A measure extraction unit that extracts at least one measure from the score image,
A staff correction unit that corrects the position of the staff in each bar of at least one bar,
A note identification unit that identifies notes in each bar of at least one bar using multiple deep learning models without erasing the staff.
Includes a music information creation unit that creates music information from the identified notes.
At least one of the measures is extracted by a deep learning model.
The multiple deep learning models are processed in parallel,
A computing device in which the music information is selected from the group consisting of XML files, musicXML files, MIDI files, mp3 files, wav files, and musical scores. A program for creating music information from sheet music images,
The input section for inputting the score image and
A measure extraction unit that extracts at least one measure from the score image,
A staff correction unit that corrects the position of the staff in each bar of at least one bar,
A note identification unit that identifies notes in each bar of at least one bar using multiple deep learning models without erasing the staff.
Includes a music information creation unit that creates music information from the identified notes.
At least one of the measures is extracted by a deep learning model.
The multiple deep learning models are processed in parallel,
A program in which the music information is selected from a group consisting of an XML file, a musicXML file, a MIDI file, an mp3 file, a wav file, and a musical score. It is a method of creating music information from a score image,
The process of inputting a score image and
The process of extracting at least one measure from the score image and
The step of correcting the slope and spacing of the staff in each bar of at least one bar,
The step of identifying a note in each bar of at least one bar,
A method comprising the step of creating musical information from the identified musical note. It is a method of creating music information from a score image,
The process of inputting a score image and
The process of extracting at least one measure from the score image and
The step of identifying a note in each bar of at least one bar,
Including the step of creating music information from the identified notes.
The extraction is taken along the top and bottom lines of the staff.
Method.

Images