WO2015101908A1 - System and method for optical music recognition - Google Patents
System and method for optical music recognition Download PDFInfo
- Publication number
- WO2015101908A1 WO2015101908A1 PCT/IB2014/067306 IB2014067306W WO2015101908A1 WO 2015101908 A1 WO2015101908 A1 WO 2015101908A1 IB 2014067306 W IB2014067306 W IB 2014067306W WO 2015101908 A1 WO2015101908 A1 WO 2015101908A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- musical
- elements
- data object
- composite
- measure
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims description 40
- 230000003287 optical effect Effects 0.000 title description 10
- 230000001020 rhythmical effect Effects 0.000 claims abstract description 40
- 239000002131 composite material Substances 0.000 claims abstract description 31
- 230000001131 transforming effect Effects 0.000 claims abstract description 6
- 230000002547 anomalous effect Effects 0.000 claims description 14
- 239000011159 matrix material Substances 0.000 claims description 13
- 238000003860 storage Methods 0.000 claims description 6
- 230000007704 transition Effects 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 2
- 238000012545 processing Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 7
- 230000033764 rhythmic process Effects 0.000 description 5
- 235000020299 breve Nutrition 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000010191 image analysis Methods 0.000 description 3
- 239000000203 mixture Substances 0.000 description 3
- SEQDDYPDSLOBDC-UHFFFAOYSA-N Temazepam Chemical compound N=1C(O)C(=O)N(C)C2=CC=C(Cl)C=C2C=1C1=CC=CC=C1 SEQDDYPDSLOBDC-UHFFFAOYSA-N 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000002950 deficient Effects 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 235000019227 E-number Nutrition 0.000 description 1
- 239000004243 E-number Substances 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- JEIPFZHSYJVQDO-UHFFFAOYSA-N ferric oxide Chemical compound O=[Fe]O[Fe]=O JEIPFZHSYJVQDO-UHFFFAOYSA-N 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000033458 reproduction Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000013179 statistical model Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/30—Character recognition based on the type of data
- G06V30/304—Music notations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/19—Recognition using electronic means
- G06V30/191—Design or setup of recognition systems or techniques; Extraction of features in feature space; Clustering techniques; Blind source separation
- G06V30/19187—Graphical models, e.g. Bayesian networks or Markov models
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10G—REPRESENTATION OF MUSIC; RECORDING MUSIC IN NOTATION FORM; ACCESSORIES FOR MUSIC OR MUSICAL INSTRUMENTS NOT OTHERWISE PROVIDED FOR, e.g. SUPPORTS
- G10G1/00—Means for the representation of music
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H1/00—Details of electrophonic musical instruments
- G10H1/0033—Recording/reproducing or transmission of music for electrophonic musical instruments
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H1/00—Details of electrophonic musical instruments
- G10H1/18—Selecting circuits
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
- G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
- G10H2210/051—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for extraction or detection of onsets of musical sounds or notes, i.e. note attack timings
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
- G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
- G10H2210/071—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for rhythm pattern analysis or rhythm style recognition
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
- G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
- G10H2210/081—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for automatic key or tonality recognition, e.g. using musical rules or a knowledge base
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
- G10H2220/155—User input interfaces for electrophonic musical instruments
- G10H2220/441—Image sensing, i.e. capturing images or optical patterns for musical purposes or musical control purposes
- G10H2220/455—Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
- G10H2250/005—Algorithms for electrophonic musical instruments or musical processing, e.g. for automatic composition or resource allocation
- G10H2250/015—Markov chains, e.g. hidden Markov models [HMM], for musical processing, e.g. musical analysis or musical composition
Definitions
- This invention relates generally to computerized processing of musical documents. More particularly, this invention relates to automatic analysis of sheet music.
- OMR optical music recognition
- U.S. Patent No. 8,067,682 to Chen-Shyurng et a/. which is herein incorporated by reference, proposes a technique, wherein a music score is detected and at least one measure in the music score is obtained by searching bar lines, so as to plan a recognition order according to the position of each measure in the music score.
- an image capturing apparatus is controlled to capture one of the measures according to the recognition order, and music information in the captured measure is recognized and outputted immediately.
- the method follows the recognition order to perform the steps of controlling the image apparatus repeatedly, recognizing the captured measure, and outputting the music information on the other measures until each of the measures has been processed.
- Embodiments of the invention increase the quality of OMR by identifying ambiguous key signatures correctly and coping with anomalies in the score, e.g., failure to adhere to the stated meter in individual measures.
- a method of music recognition which is carried out by accepting a musical score of musical elements in a digital format, transforming the digital format into a composite musical data object that models the musical score, defining the key signatures in the composite musical data object probabilistically, computing start times to play musical elements in respective measures of the composite musical data object without regard to rhythmic values of other musical elements in the respective measures, and generating an output including the defined key signatures and computed start times.
- defining the key signatures in the composite musical data object is performed by submitting presumptive key signatures for computation in a hidden Markov model.
- the hidden Markov model includes an initial probability matrix populated by probabilities that the composite musical data object begins with a particular key, and a transitional probability matrix populated by probabilities of a transition from one key to another key in the composite musical data object.
- computing start times includes identifying anomalous rhythmic elements in the respective measures of the composite musical data object that are inconsistent with the rhythmic values of the other musical elements therein, and computing the start times of the anomalous rhythmic elements according to the coordinates of the anomalous rhythmic elements relative to the coordinates of the respective measures.
- Yet another aspect of the method includes generating the digital format by optically scanning the musical score.
- transforming the digital format into a composite musical data object includes producing the musical score in a new digital format that is acceptable to a processor.
- the musical elements comprise a plurality of chords has respective chord intervals and computing start times includes computing a union of the chord intervals.
- Another aspect of the method includes computing a union of a plurality of unions of chord intervals and determining whether the unions are within a single measure.
- One aspect of the method includes making a determination that the union of the chord intervals in one of the measures is less than a predetermined proportion of an expected duration of the one measure, and responsively to the determination reporting that the one measure is a candidate for a pick-up measure.
- An additional aspect of the method includes executing the output as a musical performance.
- Other embodiments of the invention provide computer software product for carrying out the above-described method.
- Fig. 1 is a block diagram of a system of music recognition, in accordance with an embodiment of the invention
- Fig. 2A and Fig. 2B are flow-charts of a method for optical music recognition, in accordance with an embodiment of the invention
- Fig. 3 is a class diagram that models the output of an OMR tool in accordance with an embodiment of the invention.
- Fig. 4 is a class diagram that models a musical data object in accordance with an embodiment of the invention.
- Fig. 5 is a table showing an emission matrix produced by a hidden Markov model in accordance with an embodiment of the invention.
- Fig. 6 is an example of musical notation in accordance with an embodiment of the invention.
- Fig. 7 illustrates a rhythm tree that can be employed in accordance with an embodiment of the invention
- Fig. 8 is a diagram showing defective musical notation that is processed in accordance with an embodiment of the invention.
- Fig. 9 is a diagram of complex musical notation which is interpreted by an embodiment of the invention.
- Fig. 10 is a diagram of complex musical notation which is interpreted by an embodiment of the invention.
- Fig. 11 is an exemplary measure illustrating the determination of a union of chord durations, in accordance with an embodiment of the invention.
- aspects of the present invention may be embodied in software programming code, which is typically maintained in permanent storage, such as a computer readable medium.
- software programming code may be stored on a client or a server.
- the software programming code may be embodied on any of a variety of known non-transitory tangible media for use with a data processing system, such as a diskette, hard drive, or CD-ROM.
- the code may be distributed on such media, or may be distributed to consumers from the memory or storage of one computer system over a network of some type to storage devices on other computer systems for use by consumers of such other systems.
- Fig. 1 is a block diagram of a system 10 of music recognition, in accordance with an embodiment of the invention.
- the system 10 typically comprises a general purpose or embedded computer processor, which is programmed with suitable software for carrying out the functions described hereinbelow. These functions may be carried out in software running on a single processor, or on multiple processors.
- the software may be embodied on any of a variety of nontransitory media for use with a computer system, such as a diskette, or hard drive, or CD-ROM.
- the code may be distributed on such media, or may be distributed to the system 10 from the memory or storage of another computer system (not shown) over a network.
- the system 10 may comprise a digital signal processor or hardwired logic.
- a processing unit 12 includes a central processing unit 14 (CPU), a memory 16, and secondary storage unit 18, which can be a hard disk drive (HDD).
- the processing unit 12 receives input data according to a known graphic format from one or more optical input devices, examples of which are shown in Fig. 1 as a camera 20, optical scanner 22 and vector graphics module 24 ⁇
- the graphic formats for the optical input devices need not be the same.
- the processing unit 12 processes the data in order to enhance the quality of the sheet music in a manner described in further detail below, and outputs a result via an I/O module 26.
- This may be a digital music score 28, e.g., a musicXML file or a MIDI (Musical Instrument Digital Interface) file 30.
- Fig. 2A and Fig. 2B are flowcharts of a method for music recognition, in accordance with an embodiment of the invention.
- the process steps are shown in a particular linear sequence in Fig. 2 for clarity of presentation. However, it will be evident that many of them can be performed in parallel, asynchronously, or in different orders.
- a process could alternatively be represented as a number of interrelated states or events, e.g., in a state diagram.
- not all illustrated process steps may be required to implement the process.
- an image of sheet music is acquired by an optical device, e.g., camera 20 or scanner 22 (Fig. l) and output to a processor as image data in a known graphic format.
- a pre-existing graphics file may be presented to the processor.
- step 34 the data that was output at initial step 32 is formatted into a series of images, such that one image comprises one page of the sheet music.
- the result of step 34 is a series of raw digital images 36.
- the raw images 36 are subjected to image processing steps, which may include in various combinations, cropping, deskewing, unwarping, sharpening, and other enhancements known in the image-processing art in order to achieve well-aligned and digital images that are as noise-free as possible.
- image processing steps may include in various combinations, cropping, deskewing, unwarping, sharpening, and other enhancements known in the image-processing art in order to achieve well-aligned and digital images that are as noise-free as possible.
- the digital images produced in step 38 may be converted to gray-scale images 40. This may be accomplished in various ways, using conventional lossy or loss-less techniques.
- the gray-scale images may be output in a conventional graphical format suitable for viewing, for example as a PDF file 42.
- Step 38 also comprises production of a series of uncompressed monochrome digital images 44, which are used for further digital processing at step 46.
- Step 46 comprises submission of the images 44 to an OMR tool, typically page-by-page of the sheet music.
- OMR tools are suitable, so long as they emit a digital output stream whose format is known.
- step 48 it is determined if more images or pages remain to be processed. If the determination is affirmative, then control returns to step 46. Otherwise, control proceeds to step 50.
- step 50 the digital graphical format output by the OMR is converted to a composite musical data object 52 that models the original musical score.
- the actual conversion is specialized to the particular format of the OMR tool employed in step 46.
- Fig. 3 is a class diagram that models the output of an OMR tool in accordance with an embodiment of the invention.
- Fig. 4 is a class diagram that models the musical data object 52 in accordance with an embodiment of the invention.
- the properties of the musical data object 52 represent musical elements throughout the score, for example measures, notes and accidentals. These elements are identified and assigned position coordinates and other properties as shown in the class diagram in Fig. 4 ⁇
- the classes shown in Fig. 3 and Fig. 4 are used to implement step 50.
- Step 54 comprises identification of regions of interest in the coordinate system containing presumptive key signatures.
- Presumptive key signature information obtained through the above-described identification is submitted to a hidden Markov model (HMM) in order to define the key signature more accurately.
- HMM is a statistical model in which the system being modeled— in this case, the performance of a musical piece— is taken to be a Markov process with states that are not directly observable (“hidden”), but which give an observable output.
- a probabilistic analysis is applied to the observed output in order to infer the sequence of states traversed by the system, e.g., transitions in the score from one key to another.
- the HMM principally considers a count of flats and sharps as defining a key signature. Naturals are usually ignored.
- Initial probabilities for each key signature in the score are submitted to the HMM. While the HMM is not highly sensitive to these probabilities, so long as they are relatively uniformly distributed, it operates more efficiently if these are reasonably accurate than if they are not. Initial probabilities for key signatures in a score may be obtained, for example, from a statistical review of the works of the composer of the current score, from compositions by other composers in the same category as the current score, or from the accumulated experience of scores previously analyzed by the user.
- the present approach uses a two-dimensional state space to model the musical score, with coordinates that correspond to the locations of the musical elements in the score, more particularly, in regions of interest considered to contain key signatures.
- the electronic processor that carries out the computations calculates a probability distribution over the two-dimensional state space, based on three considerations that are detailed below. It uses this probability distribution in determining the most likely number of accidentals in regions having a key signature.
- the processing unit 12 (Fig. l) executes the well-known forward -backward algorithm in the HMM in order to match the regions of interest in the musical score to the actual key signatures.
- Observable states correspond to observable number of accidentals in a region of interest.
- Hidden states correspond to the actual number of accidentals.
- the HMM reports the actual number of accidentals given the observed number.
- Embodiments of the invention re-interpret the OMR analysis by applying musical logic to the image analysis.
- the following considerations are taken into account:
- the initial key signature of the song or movement the different possibilities of key signatures are taken into account, and factored statistically, in a very subtle manner—the fewer accidentals in the key signature, the more likely it is to be correct.
- This premise reflects historical tendencies in music composition. Additionally, the key signatures following that of the first staff are consulted, since key signature changes are not frequent. 2. Image analysis often returns different key signatures for different systems, (a system is a collection of staves that are supposed to be played simultaneously). Once again, a statistical premise helps here— it is far more likely for the key signature to remain the same, than it is for it to change. Once again, the key signatures of the surrounding systems are consulted.
- HMM hidden Markov model
- N 15 ( umber of possible key signatures)
- M is a N ⁇ N ⁇ k matrix
- Tbis result means tbat tbe HMM determined tbat numbers sbown in bold case (positions 2, 5, 13) are image-processing mistakes:
- wbicb is a table showing tbe populated emission matrix of tbe HMM for tbe above sequence.
- Measure 86 is a four quarter-note measure in four-four meter, which is the most popular meter, also known as "common time”.
- Meter indication 88 appears only at tbe beginning of eacb song or movement, and in places where tbe meter changes.
- Fig. 7 illustrates a rhythm tree 90 that can be employed in accordance with an embodiment of tbe invention.
- the rhythmic value of each note in Western notation is determined by several graphic factors:
- the notebead can be rectangular (breve, aka double-whole-note) or circular (all shorter values). It can be hollow (breve, whole-note, half-note) or full (all shorter values).
- Beams/flags these are interchangeable. Beams are used for groupings of notes shorter than quarter-notes. Flags are used for single, non-grouped notes of tbe same length. Tbe number of beams/flags indicates tbe shortness of tbe rhythmic value: one for eighth-notes, two for sixteenth-notes, three for thirty-second-notes, etc.
- Rhythm tree 90 illustrates several common rhythms. Notes in each level of the pyramid are twice as short as the higher level. A breve equals two whole notes. The tree shows the shapes of notes that are actually played. There is a corresponding pyramid of rests (not shown), which OMR implementations deal with in similar ways to those used with played notes.
- Fig. 8 a diagram that illustrates one difficulty in interpreting defective musical notation that is overcome by an embodiment of the invention.
- Conventional OMR implementations try to detect noteheads, stems, beams and flags in order to ascertain the rhythmic value of a given note. Faulty scanning, or blemishes in the print, can result in complete misdetection. For example a blemish in a note 92 could result in a misidentification.
- an original quarter note 94 was printed with a white region 96 inside its notehead. This may result in its detection as a half note 98.
- Fig. 9 illustrates another difficulty in interpreting musical notation that is overcome by an embodiment of the invention.
- Composers sometimes disregard mathematical rhythmic notation conventions in favor of freedom, or expression.
- Polish-French composer Frederic Chopin gives us many examples of that sort.
- Fig. 9 is a passage 100 of a musical score taken from Chopin's Nocturne Op. 9 No. 1, measure 15, left hand part. We see that Chopin wrote two voices into one hand. Half note 102 written as the second note from the left is there in order to tell the performing pianist two things:
- rhythmic value of the half note 102 makes no sense. It comes out of nowhere, and it ends an eighth note before the rest of the passage 100. Most OMR implementations, which try to reconcile the rhythmic value with the timing of the note within the measure, will report this as an error. In fact, in the Alfred edition of this piece, edited by Willard A. Palmer, the following comment was attached to this measure: "Chopin purposely placed the half note F on the 2nd 8th note of the group. Although this notation is not mathematically precise, the meaning is clear, and avoids unnecessary complications in notation.”
- Fig. 10 illustrates another difficulty in interpreting musical notation that is overcome by an embodiment of the invention.
- Fig. 10 is exemplary musical notation taken from Chopin's Nocturne Op. 27 No. 1, measure 51 ⁇ In Fig- ⁇ we see that Chopin wrote a free, improvisation-like passage 104 for the right hand at the end of the measure. Pianists play these notes roughly against their graphical counterparts in left hand part 106.
- the notation Chopin chose for this passage is that of eighth-notes, and will be read as such by most OMR implementations, which will not know how to fit so many eighth-notes into one measure.
- Fig. 10 demonstrates a fundamental paradigm of music printing that has existed since the early 19th century: the beginning of each note is set, graphically, in a location that corresponds to the timing in which it is supposed to be played.
- a relative visual position algorithm exploits the relative position of a note between two bar lines to establish the correct timing for the beginning of the sound.
- the note or chord begins at a point in musical time (e.g., in beats) that is to the musical time interval of the measure as the graphical position of the note is to the graphical length of the measure.
- the algorithm can be expressed as the equation:
- t stgrt and t en j are musical times at the beginning and end of a measure, respectively;
- t note is the musical time that a note or chord begins;
- px s t ar t and px en J are graphical coordinates, e.g., pixels, which are obtained from the musical data object 52.
- the value px s t ar t is set at the first playable musical object (note or break) in a measure;
- px en J is set at the ending bar line of the measure;
- P x note represents the graphical coordinates of the note or chord.
- the start times of anomalous rhythmic elements are assigned by ignoring the rhythmic values of other musical elements in the measure
- equation (l) is applied, treating the start position of each chord as a note.
- step 58 the union of chord durations is determined for each measure.
- FIG. 11 is an exemplary measure illustrating the determination of a union of chord durations, in accordance with an embodiment of the invention.
- An interval 60 represents the union of the intervals of chords 62, 64, 66.
- the respective chord intervals are represented by broken lines 68, 70, 72.
- the spatial intervals of each of the chords 62, 64, 66 overlap with one or more of the other chords 62, 64, 66.
- Exemplary applications of union measurements follow:
- the measure is categorized as a candidate for a pick-up measure.
- step 74 it is determined if there may be volta signs in the current score.
- step 74 determines whether the determination at decision step 74 is affirmative. If the determination at decision step 74 is affirmative, then control proceeds to step 76, where the volta signs are located, using additional image processing if necessary, as not all OMR tools report volta signs. Identifying volta signs improves the accuracy of the playing order in step 78.
- step 76 After performing decision step 76, or if the determination at decision step 74 is negative, control proceeds to step 78, where the piece playing order is determined. The performance of this step is outside the scope of this disclosure and is therefore not discussed further.
- step 80 data files comprising digital music scores are generated from the musical data object 52.
- a MIDI file is produced at step 82, resulting in a MIDI file 84.
- Other examples of useful data files are XML files having layout information and additions to an Sqlite ⁇ database.
- Data files in many other formats known to the art may be produced in step 80. Additionally or alternatively the generated output may be performed using known electronic music recognition methods.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Multimedia (AREA)
- Acoustics & Sound (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Character Discrimination (AREA)
- Auxiliary Devices For Music (AREA)
Abstract
Description
Claims
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/027,248 US9478201B1 (en) | 2013-12-31 | 2014-12-24 | System and method for optical music recognition |
JP2016543138A JP2017507346A (en) | 2013-12-31 | 2014-12-24 | System and method for optical music recognition |
EP14876924.3A EP3090385A4 (en) | 2013-12-31 | 2014-12-24 | System and method for optical music recognition |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201361922140P | 2013-12-31 | 2013-12-31 | |
US61/922,140 | 2013-12-31 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2015101908A1 true WO2015101908A1 (en) | 2015-07-09 |
Family
ID=53493339
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IB2014/067306 WO2015101908A1 (en) | 2013-12-31 | 2014-12-24 | System and method for optical music recognition |
Country Status (4)
Country | Link |
---|---|
US (1) | US9478201B1 (en) |
EP (1) | EP3090385A4 (en) |
JP (1) | JP2017507346A (en) |
WO (1) | WO2015101908A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107945780A (en) * | 2017-11-23 | 2018-04-20 | 北京物灵智能科技有限公司 | A kind of instrument playing method and device based on computer vision |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10725650B2 (en) * | 2014-03-17 | 2020-07-28 | Kabushiki Kaisha Kawai Gakki Seisakusho | Handwritten music sign recognition device and program |
JP7197263B2 (en) * | 2017-10-18 | 2022-12-27 | ヤマハ株式会社 | Image analysis method and program |
US11615772B2 (en) * | 2020-01-31 | 2023-03-28 | Obeebo Labs Ltd. | Systems, devices, and methods for musical catalog amplification services |
US11670188B2 (en) | 2020-12-02 | 2023-06-06 | Joytunes Ltd. | Method and apparatus for an adaptive and interactive teaching of playing a musical instrument |
US11972693B2 (en) | 2020-12-02 | 2024-04-30 | Joytunes Ltd. | Method, device, system and apparatus for creating and/or selecting exercises for learning playing a music instrument |
US11900825B2 (en) | 2020-12-02 | 2024-02-13 | Joytunes Ltd. | Method and apparatus for an adaptive and interactive teaching of playing a musical instrument |
US11893898B2 (en) | 2020-12-02 | 2024-02-06 | Joytunes Ltd. | Method and apparatus for an adaptive and interactive teaching of playing a musical instrument |
CN112926603B (en) * | 2021-03-26 | 2024-01-23 | 平安科技(深圳)有限公司 | Music score recognition method, device, equipment and storage medium |
Family Cites Families (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5773741A (en) * | 1996-09-19 | 1998-06-30 | Sunhawk Corporation, Inc. | Method and apparatus for nonsequential storage of and access to digital musical score and performance information |
WO2001008133A1 (en) * | 1999-07-26 | 2001-02-01 | Buhr Thomas J | Apparatus for musical composition |
US7314994B2 (en) * | 2001-11-19 | 2008-01-01 | Ricoh Company, Ltd. | Music processing printer |
CA2425965C (en) * | 2003-04-11 | 2007-10-16 | David Kestenbaum | Coloured music notation system and method of colourizing music notation |
US7432432B2 (en) * | 2004-09-21 | 2008-10-07 | Microsoft Corporation | System and method for recognizing handwritten music notations |
JP2008524656A (en) * | 2004-12-15 | 2008-07-10 | ミューズアミ,インコーポレイティド | System and method for music score capture and synchronized audio performance with synchronized presentation |
WO2007092479A2 (en) * | 2006-02-06 | 2007-08-16 | Lydia Machell | Braille music systems and methods |
CN101652807B (en) * | 2007-02-01 | 2012-09-26 | 缪斯亚米有限公司 | Music transcription method, system and device |
US20090125799A1 (en) * | 2007-11-14 | 2009-05-14 | Kirby Nathaniel B | User interface image partitioning |
KR101459766B1 (en) * | 2008-02-12 | 2014-11-10 | 삼성전자주식회사 | Method for recognizing a music score image with automatic accompaniment in a mobile device |
US8660678B1 (en) * | 2009-02-17 | 2014-02-25 | Tonara Ltd. | Automatic score following |
US8779268B2 (en) * | 2009-06-01 | 2014-07-15 | Music Mastermind, Inc. | System and method for producing a more harmonious musical accompaniment |
TWI386912B (en) * | 2009-06-12 | 2013-02-21 | Univ Nat Taiwan Science Tech | Method and system for recognizing music scores |
JP5960488B2 (en) * | 2012-04-24 | 2016-08-02 | 株式会社河合楽器製作所 | Musical score playing apparatus and musical score playing program |
US8933312B2 (en) * | 2012-06-01 | 2015-01-13 | Makemusic, Inc. | Distribution of audio sheet music as an electronic book |
US8697972B2 (en) * | 2012-07-31 | 2014-04-15 | Makemusic, Inc. | Method and apparatus for computer-mediated timed sight reading with assessment |
US9424822B2 (en) * | 2014-05-27 | 2016-08-23 | Terrence Bisnauth | Musical score display device and accessory therefor |
JP6481319B2 (en) * | 2014-10-01 | 2019-03-13 | ヤマハ株式会社 | Music score display apparatus and music score display method |
US20160189694A1 (en) * | 2014-10-08 | 2016-06-30 | Richard Lynn Cowan | Systems and methods for generating presentation system page commands |
-
2014
- 2014-12-24 US US15/027,248 patent/US9478201B1/en active Active
- 2014-12-24 WO PCT/IB2014/067306 patent/WO2015101908A1/en active Application Filing
- 2014-12-24 EP EP14876924.3A patent/EP3090385A4/en not_active Withdrawn
- 2014-12-24 JP JP2016543138A patent/JP2017507346A/en active Pending
Non-Patent Citations (6)
Title |
---|
KOPEC GARY E. ET, SPIE LOGY, vol. 2422, 6 February 1995 (1995-02-06), pages 115 - 125, XP000642544, Retrieved from the Internet <URL:HTTP://PROCEEDINGS.SPIEDIGITALLIBRARY.ORG/PROCEEDING.ASPX?articleid=992229> * |
PUGIN LAURENT: "Optical Music Recognition of Early Typographic Prints using Hidden Markov Models", PROCEEDINGS OF THE 7TH INTERNATIONAL CONFERENCE ON MUSIC INFORMATION RETRIEVAL, 8 October 2006 (2006-10-08), VICTORIA (BC), CANADA, pages 53 - 56, XP055355566, Retrieved from the Internet <URL:http://www.aruspix.net/publications/pugin06optical.pdf> * |
REBELO ANA ET AL.: "Global constraints for syntactic consistency in OMR: an ongoing approach", IMAGE ANALYSIS AND RECOGNITION ICIAR 2013, vol. 7950, 26 June 2013 (2013-06-26), PORTUGAL, pages 734 - 741, XP047031381, Retrieved from the Internet <URL:HTTP://WWW.INESCPORTO.PT/~JSC/PUBLICATIONS/CONFERENCES/2013ARebeloICIAR.pdf> * |
REBELO ANA ET AL.: "Optical recognition of music symbols", INTERNATIONAL JOURNAL ON DOCUMENT ANALYSIS AND RECOGNITION (IJDAR, vol. 13, no. 1, 17 November 2010 (2010-11-17), pages 19 - 31, XP019783846, Retrieved from the Internet <URL:http://www.inescporto.pt/~jsc/publications/journals/2010ARebeloIJDAR.pdf> * |
See also references of EP3090385A4 * |
STUCKELBERG MARC VUILLEUMIER ET AL.: "On Musical Score Recognition using Probabilistic Reasoning", PROCEEDINGS OF THE FIFTH INTERNATIONAL CONFERENCE ON BANGALORE, ICDAR '99, 20 September 1999 (1999-09-20), BANGALORE, pages 115 - 118, XP010351182, ISBN: 0-7695-0318-7, Retrieved from the Internet <URL:http://www.eecs.harvard.edu/-cat/cs/omr/docs/stuckelberg99-on-MSR-using-probabilistic-reasoning.pdf> * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107945780A (en) * | 2017-11-23 | 2018-04-20 | 北京物灵智能科技有限公司 | A kind of instrument playing method and device based on computer vision |
Also Published As
Publication number | Publication date |
---|---|
US20160300555A1 (en) | 2016-10-13 |
US9478201B1 (en) | 2016-10-25 |
EP3090385A4 (en) | 2017-08-23 |
JP2017507346A (en) | 2017-03-16 |
EP3090385A1 (en) | 2016-11-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9478201B1 (en) | System and method for optical music recognition | |
JP6197631B2 (en) | Music score analysis apparatus and music score analysis method | |
US6930236B2 (en) | Apparatus for analyzing music using sounds of instruments | |
Wu et al. | Polyphonic music transcription with semantic segmentation | |
US7064262B2 (en) | Method for converting a music signal into a note-based description and for referencing a music signal in a data bank | |
US9368093B2 (en) | Recordings resulting from methods for transcoding music notation | |
CN106816151B (en) | Subtitle alignment method and device | |
CN113674584B (en) | Comprehensive conversion method and comprehensive conversion system for multiple music scores | |
US9478200B2 (en) | Mapping estimation apparatus | |
Paleari et al. | A multimodal approach to music transcription | |
CN111063327A (en) | Audio processing method and device, electronic equipment and storage medium | |
US20150268847A1 (en) | Digital music device and methods for managing digital music | |
Church et al. | Improving Rhythmic Transcriptions via Probability Models Applied Post-OMR. | |
Peter et al. | Automatic Note-Level Score-to-Performance Alignments in the ASAP Dataset. | |
KR20140139201A (en) | Braille score generation apparatus using crowd sourcing based game and method thereof | |
CN113076967B (en) | Image and audio-based music score dual-recognition system | |
CN115331648A (en) | Audio data processing method, device, equipment, storage medium and product | |
CN113053337A (en) | Intonation evaluation method, intonation evaluation device, intonation evaluation equipment and storage medium | |
JPH07117833B2 (en) | Musical note similarity calculator | |
Chuan et al. | The KUSC classical music dataset for audio key finding | |
US11749237B1 (en) | System and method for generation of musical notation from audio signal | |
US20240087549A1 (en) | Musical score creation device, training device, musical score creation method, and training method | |
de Valk et al. | Crafting TabMEI, a module for encoding instrumental tablatures | |
JP5189952B2 (en) | Music score recognition apparatus and computer program | |
Khatri et al. | Guitar Tuning Identification |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 14876924 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 15027248 Country of ref document: US |
|
REEP | Request for entry into the european phase |
Ref document number: 2014876924 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2014876924 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 2016543138 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |