DE10341311A1 - Production of a digital fingerprint of a music sheet as a method for indexing and authenticating music sheet copies, whereby the fingerprint is based on the graphical form of the music sheet notes - Google Patents

Abstract

Method for producing a digital fingerprint (16) of a music sheet or a music sheet section (11), whereby the fingerprint is based on the graphical form of the notes on the music sheet or music sheet section. Independent claims are also included for the following:- (a) a device for generating a digital fingerprint of a music sheet segment; (b) a device for characterizing a music sheet and; a method for characterizing a music sheet.

Description

The The present invention relates to the generation of fingerprints for a sheet of music or a score sheet based on the graphic design the notation and the use of fingerprints for characterization or identifying sheet music.

application

Big data are increasingly being printed in electronic form and no longer in print Paper form saved and managed. This is also playing in the meantime more and more involved in the marketing of music scores, at where an international distribution and transport of printed scores is not economical due to the usually relatively low prices. Often notes are copied illegally and for the most part due to the lack of availability music scores, less for financial reasons. A distribution of music scores in electronic form for example over the internet is therefore a cost-effective Alternative. On the other hand, it is predictable that it will be similar in the future Internet architectures like Napster on or from which illegal audio data and increasingly video data can also be loaded, soon also for music scores will give. Since music scores have a significant economic role in the entire music industry play, will be an effective solution to secure the rights to musical notations of classical or contemporary Music when distributed in paper form or in common, electronic representations needed. For the Author or publisher of a music notation or score it is at the public making available important to this work, possibly made and if necessary manipulated copies of the work as an illegal copy of a particular print edition recognized and thus clearly assigned to the author or right holder can be. It's not just about the copyright of the music itself but also about the setting of notes. The setting of notes is specific Subject to rules, for example, is the set of music symbols as notes are limited and the possible vertical localizations of music symbols with respect to the staff lines required. For example, allowed Notes in principle only either exactly between two staves or be positioned exactly on a staff line, the exact position depends on the grade height from. The vertical positioning of clefs in relation to the staves is also set. On the other hand, there are certain degrees of freedom when setting the notes, for example, the orientation of the stems, the horizontal positioning or the horizontal spacing of the music symbols or generally the number of symbols per staff or the distribution the different voices on the staves. Therefore, let yourself different print editions from different publishers of the same piece of music visually discernible and clearly distinguishable.

A in the publishers today used method is a paper carrier with to use a special watermark or markings such as certain ornaments, ornaments or written copyright notices optically visible together with a music notation on the paper carrier to apply. However, these procedures are not for the protection of the rights and the Detection of illegally made copies suitable, since these markings when copying the paper-based work with the simplest Remove agents. Is the music notation in electronic Form before, so can a copy with the help of image processing software edited and the label deleted. In both cases then a conclusion the origin of the music not easily possible.

Out DE10023759 "Method for introducing hidden information into a music notation" is a method of incorporating information hidden in a music notation, so-called digital watermarks The method detects some of the geometric elements of the music notation, such as staves, barlines, staves, or noteheads a predefinable key changed so that the changes bear the information to be incorporated in digital form, but the readability of the music not is reduced.When the key is known, then the wrong information can be read out of illegally copied music notices and thus clearly assigned to the publisher of the original A disadvantage of this method is that the data must first be modified, so to speak, marked in order to be able later to identify copies of the original with the help of the key, and to leave copies of unmarked music already in circulation Do not recognize yourself with this method.

Methods are now known, called Optical Music Recognition (OMR), with which the music symbols can be extracted from scanned music sheets and thus the piece of music can be recognized. These techniques are similar to Optical Character Recognition (OCR) techniques, which extract the ASCII text from a scanned text. The disadvantage of the OMR method for the problem to be solved here is that important formatting instructions, the layout and thus the original note image are lost, and thus the music piece but not different sets of prints of different publishers of the same piece of music is different can become.

For the identification of data, the method of the fingerprint is known for media such as audio, video or image data. These techniques are also often referred to in the literature as perceptual hashing or passive watermarking. From "A Highly Robust Audio Fingerprinting System" by J. Haitsma and T. Kalker, Ismir 2002, ^3rd International Conference on Music Information Retrieval, or DE10133333 "Fingerprint method and apparatus and method and apparatus for identifying an audio signal" are known fingerprinting methods for audio data A fingerprint is a compact representation of the medium to be characterized, typically calculated by a function that is not a unique inverse It is not possible to reconstruct the original from the fingerprint alone, but the fingerprint of the medium to be characterized allows a unique identification of the medium, comparable to the fingerprint of a human being It is also important to be able to distinguish between different recordings of the same piece of music When creating the fingerprint, it is essential to define suitable features that describe the data content as compactly as possible individual characteristics are found. In principle, audio, video and image data differ from media types with symbolic representations such as notes or text in that they theoretically have an analogous (and thus infinitely fine) range of values, in contrast to a bivalent (finite) value range in which only black ones are used or white pixels occur. A direct application of the audio and video data methods for the case of notation identification dealt with here is therefore not possible.

The known methods for Consider image data neither the binary Value range of a sheet music nor the substantive dependencies, at which known music symbols such as notes, staves, staves, barlines etc. occur only within the framework of the rules of the music notation. The rules of the music notation specify, for example, where the clef and the notes may be located in relation to the staves. Around very similar but to be able to differentiate between different print sets must have specific ones Features specially for music sheet are found, since the known fingerprint method for image data no suitable compact representation the content of a score sheet.

Problematic in determining the features to be extracted is that the Computational effort to determine specific features and to generate of a fingerprint should be moderate. In addition, the generated Fingerprint as possible be compact so that the memory overhead of a database of known fingerprints and the computational effort to compare a sheet of music to be identified with the well-known sheet music, whose fingerprints stored in the database, remain within acceptable limits. Further requirements for the fingerprint for sheet music are, on the one hand, that two Music sheet, which come from two different print editions, even if they very similar are, for example, because they are the same piece of music, two different ones fingerprints have to have different as desired Printed editions of the same piece of music different publisher to distinguish. On the other hand, that must be Method of producing a fingerprint robust to typical Processing operations such as duplication. Copies of a Sheet music, for example, where certain copyright marks removed or by photocopying with poor quality the score was noisy, should still be identified. For this must be a necessary feature of the fingerprint that the fingerprints two sheet music, which come from the same print edition, but where one like described opposite the original changed or falsified was, a correspondingly high agreement exhibit. It should be mentioned be that the invention opposite a manipulation that requires a completely new setting of notes includes a printed edition, usually can not grab, because here is the original graphic Designing the printed output when creating the copy lost goes.

Task of invention

The The object of the present invention is a fingerprint concept for sheet music to Identification or indexing of copies of music notations create, without previously hidden in notations or scores Information must be introduced. Here are also different Distinguished printed editions of different publishers of the same piece of music can be.

The object is achieved by the method of independent claims 1 and 16. Advantageous embodiments of the method are the subject of the dependent claims. Claims 22 and 23 indicate devices for carrying out the methods according to claims 1 and 16, respectively. Claims 24 and 25 finally indicate computer-readable data carriers which executable programs according to the methods of claims 1 and 16, respectively.

presentation the invention

Of the The present invention is based on the finding that suitable Characterization of sheet music one especially on sheet music adapted generation of fingerprints is necessary. Because the graphic design of the score image certain predetermined general valid Rules, for example, where the clef and the notes may be located in relation to the staves is it is advantageous in extracting the specific features of a score sheet. In the generation according to the invention The fingerprint of a sheet of music is the determination of the distribution the music symbols on the sheet music essential.

in the Contrary to the one mentioned above already known methods of introducing hidden watermarks in the sheet music must with of the invention, the score images are not altered to later identify them to be able to. Also not marked sheet music can can be identified or characterized by application of the invention. The quality the score must not be changed so that there is no risk that musicians will be affected by the introduced changes to be influenced.

The according to the invention calculated fingerprints are specially adapted to the structure of a score sheet and provide hence a suitable compact representation of a score sheet or sheet music to characterize sheet music. By the special adaptation, the fingerprints are very compact and still suitable also very similar Music sheet, those of different print sets come to be distinguished.

A possible Application of the invention lies in the efficient archiving and Indexing of music notations in electronic form. In the following situation, the invention be used for example: it happens that musicians a Score is only available as a copy, but like to acquire it legally would. With the help of the invention, the printed edition and the publisher of Originals are detected.

A Another important application of the present invention is in the Protection of notations, the present protection mechanism being based on the setting of the Notes in themselves and on the protection of those mediated with these notes Music refers. In particular, the invention can be used to search for distributors used by illegal copies of music notes on the Internet. Possible is also in software programs that use the OMR method mentioned above be realized, with the aid of the invention, a control function to incorporate with the processing of illegal copies of music scores is prevented. In addition, the similarity of two with help Fingerprints generated by the invention a measure of the similarity of two pieces of music and can thus as a measure of Determination of a probable plagiarism can be used.

preferred embodiments The present invention will be described below with reference to FIGS attached drawings explained in detail. Show it:

1 a schematic block diagram of a device according to the invention for generating a fingerprint of a sheet music;

2 a schematic block diagram of a device according to the invention for characterizing a sheet music;

3 a schematic block diagram of a device according to the invention for identifying and localizing staves or staves of a score sheet;

4 a schematic block diagram of a device according to the invention for generating a fingerprint, which is composed of sub-fingerprints for each one stave of the sheet music.

1 shows a schematic block diagram of a device according to the invention for generating a fingerprint of a sheet music or sheet music detail 11 , If the sheet of music is in paper form, the sheet of music or a score sheet is first scanned and the data obtained is used to determine the specific characteristics. It is also possible to generate fingerprints of music sheets that are already available electronically in known formats such as gif or jpeg. Optionally, a preprocessing to remove stains or a Binarisierung, ie a reduction of grayscale to black and white, performed first. To generate the fingerprint are with a device 12 First, specific characteristics of the sheet music or sheet music section determined. These features are based on the graphic design of the score. With a device 13 a feature vector is formed from the specific features. This feature vector is used in another device 14 processed. During processing, depending on the specific characteristics additional characteristics are generated. As new features in particular the relative position of the specific features are provided to each other. From the original feature vector and the new features, a new, high-dimensional feature vector is formed. Finally, in a facility 15 calculates the fingerprint of the entire sheet of music or the sheet of notes, preferably by mapping the high-dimensional feature vector into a less complex space. The fingerprint can be calculated from the feature vector, in particular by quantization, eg deleting Least Significant Bits (LSB). Another possibility consists of the use of decoding algorithms, which are normally used for error correction in connection with a previously occurring encoding, for example, before an error-generating transmission. In the context of the invention, the decoding algorithms are used alone to remove redundancy or to reduce the feature vector to a more compact fingerprint.

In the following, further preferred implementations of the individual method steps will be made 1 described in more detail.

3 shows a possible first step, which can be carried out to determine specific characteristics of a sheet of music: the staves or staves (consisting of five staves) 32 , the sheet music or sheet music section are in the institution 31 identified and localized. A sheet of music usually contains between 8th - 12 Staves. However, it is also conceivable to extract a different kind of music symbols instead of the staff lines and to use these as reference patterns. The extracted music symbols, preferably staves or staves, can be used as reference patterns, in particular for eliminating distortions of the score image or for determining further specific features.

• 1. Es wird lediglich die Verteilung der Pixel auf dem Notenblatt oder Notenblattausschnitt verwendet. In diesem Fall sollte zuvor der Schritt wie in 3 beschrieben durchgeführt und ein Referenzmuster extrahiert worden sein. Die Pixel, die als zu dem Referenzmuster gehörend identifiziert wurden, werden vorher aus dem zu verarbeitenden Rasterbild entnommen. Das Notenblatt wird allein durch die schwarzen Pixel p_i = (x_i, y_i beschrieben, mit (x_i, y_i) ∊ P(„die Menge aller möglichen Positionen").
• 2. Die Verteilung der Musiksymbole auf dem Notenblatt bzw. Notenblattausschnitt wird bestimmt. Die Musiksymbole werden hier durch die Verteilung der Pixel auf dem Notenblatt lokalisiert. Wurde zuvor ein Referenzmuster ermittelt, zum Beispiel an Hand extrahierter Notenlinien, kann hierzu bei einer Häufung von schwarzen Pixel in einer bestimmten Relation zu den Notenlinien auf ein Musiksymbol geschlossen werden. Die Information welches Musiksymbol vorliegt (z.B. halbe oder ganze Note) wird bei dieser Darstellung aber nicht explizit berücksichtigt.
• 3. Es ist auch möglich zusätzlich zu der oben beschriebenen Lokalisierung von Musiksymbolen auf dem Notenblatt zusätzlich, ähnlich wie bei den schon bekannten OMR-Verfahren, die musikalischen Symbole explizit zu identifiziert. Jedes musikalische Symbol kann durch das Tripel t_i = (s_i,(x_i, y_i)) beschrieben werden mit s_i ∊ S(„Die Menge aller Musiksymbole"), und (x_i, y_i) ∊ P(„die Menge aller möglichen Positionen"). Dabei müssen nicht notwendigerweise alle Symbole identifiziert werden. Es genügt bestimmte Musiksymbole, wie z.B. nur Musiknoten, oder nur eine Teilmenge. aller auf dem Notenblatt gesetzten Musiksymbole zu verwenden.

According to the invention, several possibilities for determining the specific features are proposed:

• 1. Only the distribution of pixels on the sheet of music or score sheet is used. In this case, the step should be like in 3 described and extracted a reference pattern. The pixels identified as belonging to the reference pattern are previously extracted from the raster image to be processed. The sheet of music is described solely by the black pixels p _i = (x _i , y _i ) with (x _i , y _i ) ε P ("the set of all possible positions").
• 2. The distribution of the music symbols on the sheet music or sheet music section is determined. The music symbols are located here by the distribution of the pixels on the sheet of music. If a reference pattern has previously been determined, for example by hand extracted staves, this can be concluded by a cluster of black pixels in a certain relation to the staves on a music symbol. The information which music symbol is present (eg half or whole note) is not explicitly taken into account in this presentation.
• 3. It is also possible, in addition to the above-described localization of music symbols on the sheet music additionally, similar to the already known OMR methods, the musical symbols explicitly identified. Each musical symbol can be described by the triple t _i = (s _i , (x _i , y _i )) with s _i ε S ("The set of all music symbols"), and (x _i , y _i ) ε P (" the set of all possible positions "). Not necessarily all symbols must be identified. Suffice certain music symbols, such as only music notes, or only a subset. to use all music symbols set on the score sheet.

The least effort required the method of 1. and is therefore preferred. At the determination specific features with the preferred method of FIG. 1 proposed the feature vector from the summation of the black Each pixel in the x and y direction (projection) to generate.

4 shows a further preferred embodiment in each case for a staff 32 as the smallest possible score sheet cutting a sub-fingerprint 42 is calculated. The calculation of a sub-fingerprint is made with the in 1 shown device 17 carried out. Helpful here is first the staves or staves as in 3 described to identify and locate. The calculation of the sub-fingerprints of a score sheet can be realized in parallel or serially. The fingerprint 41 a sheet of music can thus be assembled as an alternative to the calculation of a fingerprint directly from the entire sheet of music from the sub-fingerprints each for a staff of the sheet music.

The method described for generating a fingerprint of a sheet of music or sheet music can be used to build a database 23 be used by fingerprints of sheet music. In this database, the fingerprints refer to meta information such as the music title, the piece of music, the author, the printed edition or the publisher or right holder of the score from which the sheet music originates. After setting up such a database No sheets, whose origin is unknown, with the help of the 2 be identified device described.

2 shows a schematic block diagram of a device according to the invention for characterizing a sheet music or a sheet of paper cutout. First, in a facility 22 a search fingerprint for the sheet of music to be characterized 22 generated. This generation of the fingerprint can, in principle, according to the in 1 described device 17 be performed. The search fingerprint is now in a facility 24 with those in the database 23 stored fingerprints compared. For this purpose, a distance measure adapted to the fingerprint, in particular at its dimension, is used. In particular, norm-based distance metrics such as the Euclidean metric can be used as the distance measure. If the fingerprint of the sheet of music or sheet music cutout to be characterized has less than a predetermined deviation from the fingerprint associated with the known sheet of music, the device will 25 the correspondence of the score of the sheet of music or sheet of paper to be characterized with the score of the known sheet of music adopted.

The Searches in the database can preferably, similar to J. Haitsma and T. Kalker, ISMIR 2002, was reduced as follows will be: in the database are fingerprints consisting of sub-fingerprints, the for the Staves of the score sheet were calculated, saved. The Under fingerprints are dimensioned so that they only have a certain number of possible Can accept values. The database contains additionally a table with all sorts Values of the sub-fingerprints. Everyone Value in turn refers to any stored fingerprint that contains a sub-fingerprint with this value. A particularly efficient Search results when first it is determined from which sub-fingerprints the search fingerprint is composed and then the search fingerprint initially compared only to those fingerprints which match the search fingerprint in at least one sub-fingerprint. The Probability is very high, that within this subset the stored fingerprints a match detected with the search fingerprint.

at Determination of compliance of the sheet to be identified with a sheet of music from the Database, several results should be output: Depending on the fingerprint distance may be a similarity measure that's the similarity the music notation of the sheet music or sheet music to be characterized to the music notation of a known sheet music represents and thus indicates the security of the decision for compliance. In addition, will Meta information such as music title, song, author, print edition or Publisher or right holder to whom the fingerprint in the database indicates indicated.

A possible Application of the device for characterizing a music sheet or sheet music section is in the monitoring and control of the Flow of information in the form of sheet music in networks. Network nodes can be installed so that they actively control the flow of information Monitor notes in electronic representation and check the notes for their origin. Thus, in the Network illegal copies of notes are detected. The device But it is also suitable pirated copies in paper form on their origin to identify.

Claims (25)

Method for generating a fingerprint ( 16 ) of a sheet of music or a score sheet ( 11 ), wherein the fingerprint is calculated based on the graphic design of the score of the score sheet or score sheet. Method according to claim 1, which is for generating the fingerprint ( 16 ) of a sheet of music or a score sheet ( 11 ) comprises the following steps: - determination of specific characteristics ( 12 ) a score sheet or a score sheet cutout; and - formation of a feature vector ( 13 ) from the specific characteristics; and - processing the feature vector ( 14 ); and - calculation of the fingerprint ( 15 ) from the formed feature vector. Method according to claim 2, characterized in that that for determining the specific characteristics of a score sheet or a sheet of paper cut by scanning the sheet music or used by scanning a score sheet obtained data become. Method according to one of claims 2 or 3, characterized that before the determination of specific features a preprocessing carried out is, in particular a removal of stains or a Binarization. Method according to one of claims 2 to 4, characterized in that for the determination of specific features ( 12 ) First, a certain type of music symbols, especially the Notenlini or staves ( 32 ), identified and localized ( 31 ) and these are used as reference patterns. Method according to claim 5, characterized in that that using the reference pattern distortion of the score be eliminated. Method according to one of claims 5 or 6, characterized in that for the determination of specific features ( 12 ) the distribution of pixels on the sheet of music or score sheet is used. Method according to claim 7, characterized in that that the feature vector from the summation of the black pixels respectively is generated in the x and y directions. Method according to one of claims 2 to 6, characterized in that for the determination of specific features ( 12 ) the distribution of the music symbols on the sheet of music or score sheet is used, the music symbols being localized by the distribution of the pixels on the sheet of music. Method according to one of claims 2 to 6, characterized in that for determining the specific features ( 12 ) the distribution of the music symbols on the sheet of music or sheet music and in addition the identification of the music symbols are used. Method according to one of claims 2 to 10, characterized in that in the processing of the feature vector ( 14 ) are generated depending on the specific features further features, in particular the relative position of the specific features to each other, and from the original feature vector and the new features a new, high-dimensional feature vector is formed. Method according to one of claims 2 to 11, characterized in that for the calculation of the fingerprint ( 15 ) the high-dimensional feature vector is mapped into a space of lesser dimension. Method according to claim 12, characterized in that that the calculation of the fingerprint from the high-dimensional Feature vector by means of quantization and / or the application of decoding algorithms carried out becomes. Method according to one of claims 2 to 13, characterized in that in each case for a staff ( 32 ) as the smallest possible score sheet cutout a sub-fingerprint ( 42 ) and the fingerprint of the entire sheet of music ( 41 ) composed of sub-fingerprints each for a staff of the sheet music. Method according to one or more of the preceding claims, characterized in that the method for generating a fingerprint of a sheet of music for a plurality of sheet music is carried out to form a database ( 23 ) from fingerprints, with a fingerprint indicating meta-information. A method for characterizing a sheet of music or a sheet of music, comprising the following steps: - generating a search fingerprint ( 22 ) of a sheet of music or a score sheet ( 11 ); and - comparing ( 24 ) the calculated fingerprint having a plurality of stored fingerprints of a plurality of known music sheets representing different print sets of a plurality of scores; and - determining ( 25 ) that the score of the sheet of music or sheet cutout to be characterized coincides with the score of a known sheet of music, when the fingerprint of the sheet of music or sheet cutout to be characterized has a less than a predetermined deviation from the fingerprint associated with the known sheet of music. The method of claim 16, further comprising Step: Output meta information when a match is detected. Method according to one of claims 16 or 17, characterized in that in the comparison ( 24 ) of the search fingerprint with a plurality of stored fingerprints of known sheet music a distance measure adapted to the fingerprint is used; and depending on the fingerprint distance, a similarity measure is given that represents the similarity of the music notation of the sheet of music or sheet music cutout to be characterized to the music notation of a known sheet music score. Method according to claim 18, characterized in that that as distance measure a norm-based Distance metric, in particular the Euclidean metric, is used. Method according to one of Claims 16 to 19, characterized in that the fingerprints stored in the database are composed of sub-fingerprints ( 42 ), each for one staff ( 32 ) of the associated sheet music, and the search fingerprint ( 22 ) also from Unter-Fingerabdrü composed, and initially a comparison is made with those stored fingerprints that match with respect to at least one sub-fingerprint with the search fingerprint. Method according to one of claims 19 to 20, characterized in that the comparison ( 24 ) of the search fingerprint is performed automatically with a plurality of stored fingerprints, for example, when a score sheet in digital representation is to be recognized on the Internet and checked for its origin. Device for generating a fingerprint of a sheet of music or a sheet of music sheet ( 11 ), whereby the fingerprint ( 16 ) is calculable based on the graphic design of the score of a sheet of music or a sheet of notes, comprising: - means for determining specific characteristics ( 12 ) a score sheet or a score sheet cutout; and - means for forming a feature vector ( 13 ) from the specific characteristics; and - means for processing the feature vector ( 14 ); and - a device for calculating the fingerprint ( 15 ) from the formed feature vector. Apparatus for characterizing a sheet of music or a score sheet comprising: - means for generating a search fingerprint ( 22 ) of the sheet music. (11); and a means for comparing ( 24 ) the calculated fingerprint having a plurality of stored fingerprints of a plurality of known music sheets representing different print sets of a plurality of scores; and - a means of detection ( 25 ) that the score of the sheet of music or sheet cutout to be characterized coincides with the score of a known sheet of music, when the fingerprint of the sheet of music or sheet cutout to be characterized has a less than a predetermined deviation from the fingerprint associated with the known sheet of music. A computer-readable medium comprising an executable program for generating a fingerprint of a sheet of music or a score sheet ( 11 ), the fingerprint ( 16 ) is calculable based on the graphical design of the score of a sheet of music or score sheet, and the program contained on the computer-readable medium comprises: - means for determining specific characteristics ( 12 ) a score sheet or a score sheet cutout; and - means for forming a feature vector ( 13 ) from the specific characteristics; and - means for processing the feature vector ( 14 ); and - a device for calculating the fingerprint ( 15 ) from the formed feature vector. A computer-readable medium containing an executable program for characterizing a sheet of music or score sheet and comprising the program contained on the computer-readable medium: - means for generating a search fingerprint ( 22 ) of the score sheet ( 11 ); and - a means for comparing ( 24 ) the calculated fingerprint having a plurality of stored fingerprints of a plurality of known music sheets representing different print sets of a plurality of scores; and - a means for determining ( 25 ) that the score of the sheet of music or sheet cutout to be characterized coincides with the score of a known sheet of music, when the fingerprint of the sheet of music or sheet cutout to be characterized has a less than a predetermined deviation from the fingerprint associated with the known sheet of music.