CN113763991B - Comparison method and system of performance sound information and music score information - Google Patents
Comparison method and system of performance sound information and music score information Download PDFInfo
- Publication number
- CN113763991B CN113763991B CN202111041068.0A CN202111041068A CN113763991B CN 113763991 B CN113763991 B CN 113763991B CN 202111041068 A CN202111041068 A CN 202111041068A CN 113763991 B CN113763991 B CN 113763991B
- Authority
- CN
- China
- Prior art keywords
- pitch
- matrix
- path
- information
- comparison
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 239000011159 matrix material Substances 0.000 claims abstract description 133
- 238000011156 evaluation Methods 0.000 claims abstract description 67
- 230000006870 function Effects 0.000 claims abstract description 31
- 238000006243 chemical reaction Methods 0.000 claims abstract description 25
- 230000008569 process Effects 0.000 claims abstract description 17
- 230000033764 rhythmic process Effects 0.000 claims abstract description 9
- 239000013598 vector Substances 0.000 claims description 57
- 238000000605 extraction Methods 0.000 claims description 22
- 238000004364 calculation method Methods 0.000 claims description 17
- 238000001228 spectrum Methods 0.000 claims description 15
- 238000003780 insertion Methods 0.000 claims description 12
- 230000037431 insertion Effects 0.000 claims description 12
- 238000012163 sequencing technique Methods 0.000 claims description 11
- 230000011218 segmentation Effects 0.000 claims description 10
- 238000012217 deletion Methods 0.000 claims description 8
- 230000037430 deletion Effects 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 5
- 230000008859 change Effects 0.000 claims description 3
- 239000002131 composite material Substances 0.000 claims description 2
- 239000011295 pitch Substances 0.000 description 84
- 238000012549 training Methods 0.000 description 8
- 238000004458 analytical method Methods 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000010276 construction Methods 0.000 description 4
- 239000000284 extract Substances 0.000 description 4
- 230000004888 barrier function Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/90—Pitch determination of speech signals
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
- G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
- G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
- G10H2210/076—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for extraction of timing, tempo; Beat detection
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/90—Pitch determination of speech signals
- G10L2025/906—Pitch tracking
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Landscapes
- Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Auxiliary Devices For Music (AREA)
Abstract
The invention discloses a comparison method and a comparison system of performance sound information and music score information, which mainly solve the problems that the prior evaluation system and method in the prior art mechanically require the matching of actual performance data and standard performance data, neglect the non-fluency of the actual performance process, lack the evaluation of fine technical points such as voice region errors, snatch a shot, drags and beats, unstable rhythm and the like, and cause lower performance evaluation. The invention obtains the playing sound information and the music score information; then, respectively performing code conversion on the performance sound information and the music score information to generate a matrix; then inputting the matrix into a corresponding comparison function to generate a comparison matrix; then planning a path according to the comparison matrix, and producing an optimal path; then, the matrix generated first is segmented according to the position index of the optimal comparison path; and finally, evaluating according to the evaluation dictionary and outputting an evaluation language. Through the scheme, the aim of detail comment can be achieved.
Description
Technical Field
The invention relates to the technical field of music analysis, in particular to a comparison method and a system of performance sound information and music score information.
Background
Currently, performance evaluation methods for musical instruments are generally based on systems for additionally evaluating performance states on electronic musical instruments; the method and system mainly store standard performance data in an electronic musical instrument or electronic musical instrument metadata, and perform performance evaluation by comparing the standard performance data with actual performance data.
The method has a plurality of barriers in musical instrument playing training and playing evaluation suitable for music education, and mainly comprises the following steps: highly dependent on electronic musical instruments or electronic musical instrument accessories, but most of music education adopts non-electronic musical instruments; in the process of performing performance training, a musical instrument learner needs to gradually improve the performance technology, and the requirement of standard performance data is generally difficult to directly reach, while the method generally mechanically requires the matching of actual performance data and standard performance data, neglects the non-fluency of the actual performance process, and thus causes lower performance evaluation; the core purpose of performance training of a learner of a musical instrument is to improve performance technology, and the above evaluation method generally involves only overall evaluation, lacks evaluation of fine technical points such as a voice region error, snatch a shot, a trawling, a rhythm instability, and the like, and cannot well assist performance training of the learner of a musical instrument by adopting the above technology.
Disclosure of Invention
The invention aims to provide a comparison method and a comparison system of performance sound information and music score information, which are used for solving the problems that the conventional evaluation system and method mechanically require the matching of actual performance data and standard performance data, neglect the non-fluency of the actual performance process, lack the evaluation of fine technical points such as voice region errors, snatch a shot, drags and beats, unstable rhythm and the like, and cause lower performance evaluation.
In order to solve the problems, the invention provides the following technical scheme:
the method for obtaining the optimal comparison path of the performance pitch information and the melody pitch information comprises the following steps:
(A1) Acquiring pitch information of a performance sound and pitch information of a melody;
(A2) Performing code conversion on the pitch information of the performance sound and the pitch information of the melody respectively to generate a pitch matrix;
(A3) Inputting the pitch matrix of (A2) into a pitch comparison function to generate a pitch comparison matrix;
(A4) And (3) carrying out path planning on the pitch comparison matrix of the step (A3), and generating a pitch minimum score path as an optimal path.
Specifically, the PITCH information of the melody in step (A1) is taken from the information of < NOTE < PITCH < stem > < OCTAVE > > tag in the melody file in the extensible markup format (XML/MusicXML).
Specifically, in step (A2), the pitch information is code-converted into a two-dimensional boolean matrix of 128×n, the column coordinates represent the position index of each of the played notes in the pitch sequence, the row coordinates represent the position indexes of 128 semitone notes of absolute pitches C-1 to G9 in scientific notation, the element values in the matrix are represented by 0 or 1, 0 represents not playing the corresponding notes, and 1 represents playing the corresponding notes.
Specifically, the pitch comparison function in step (A3) is a composite function including a pitch scoring function and a pitch matching function;
the pitch scoring function is:
The pitch matching function is:
wherein m 1<g1≤u1′;
a j represents the column vector for converting the pitch information of the sound into a pitch matrix in step (A2);
x i represents the column vector of converting the cursive pitch information into a pitch matrix in step (A2);
If the expression x i=xi*aj is satisfied between the ith column of the music spectrum pitch matrix and the jth column of the music spectrum pitch matrix, judging that the two columns are matched, assigning the (i+1, j+1) position of the scoring matrix as a matching score m 1, otherwise, assigning the (i+1, j+1) position of the scoring matrix as a matching score u 1;
g 1 represents the insertion score of g 1 for each bit of representation moved in the pitch matrix from left to right and top to bottom, introducing an insertion operation.
Specifically, the path planning in the step (A4) is minimum score path dynamic planning based on position indexes, and the specific process is that the position indexes of the music notes and the sound notes are shifted by 1 positive unit integrally through position index transformation, 0 is used for representing insertion or deletion, and a position index (r, c) sequence of the optimal path is generated according to a minimum score index function;
Wherein i is the ith column of the cursive pitch matrix; j is the j-th column of the pitch matrix of the sound; let R sequence be the set of all R; the C sequence is the set of all C.
Specifically, the method for obtaining the optimal comparison path of the performance duration information and the score duration information comprises the following steps:
(B1) Acquiring performance sound duration information and melody duration information;
(B2) Performing code conversion on the (B1) performance sound time value information and the music spectrum time value information respectively to generate a time value matrix;
(B3) Inputting the timing value matrix of the step (B2) into a timing value comparison function to generate a timing value comparison matrix;
(B4) And (3) carrying out path planning according to the timing value comparison matrix of the step (B3), and generating a timing value minimum variance path as an optimal path.
Specifically, the music score value information in step (B1) is taken from the < NOTE < DURATION > > tagged information in the music score file in the extensible markup format (XML/MusicXML).
Specifically, in step (B2), the time value information is code-converted into a two-dimensional numerical matrix 1*n, the column coordinates represent the position index of each performance note in the time value sequence, and the element values in the matrix are represented by integer numerical values and represent the number of time frames.
Specifically, the value comparison function in step (B3) is: dura_s (i+1,j+1)=bj/yi′;
wherein, the matrix of the curvelet time value is Sound duration matrix is/>Wherein y i represents the ith column vector of the curvelet matrix, and b j represents the jth column vector of the sound value matrix; dura_s is the duration alignment matrix.
Specifically, the path planning in the step (B4) is a minimum variance path dynamic planning based on the position index, and the specific process is that the alignment matrix of the timing values is traversed by a directed graph, and the path planning from DURA_S (0,0) to DURA_S (0,0) is acquiredCalculating the variance of each path to obtain a minimum variance path, and converting the minimum variance path into a (r, c) sequence based on a position index, wherein 0 represents insertion or deletion, and shifting the whole of the position indexes of the music notes and the sound notes by 1 positive unit through position index conversion;
Where the duration of each position index (i, j) of DURA_S path(i,j) deviates, E (DURA_S path) represents the average of the duration deviations of the entire path, and N represents the number of path positions.
A comparison method of performance sound information and music score information includes the following steps:
(C1) Obtaining an optimal comparison path of the pitch information according to an obtaining method of the optimal comparison path of the performance pitch information and the melody pitch information;
(C2) Obtaining an optimal comparison path of the time value information according to an obtaining method of the optimal comparison path of the performance time value information and the score time value information;
(C3) Obtaining an optimal comparison path according to the comparison paths in the step (C1) (C2);
(C4) Cutting the pitch matrix and the duration matrix of the step (B2) according to the position index of the optimal comparison path of the step (C3) to generate a section, phrase and paragraph matrix;
(C5) Constructing evaluation rules of different levels in advance to generate an evaluation dictionary;
(C6) And (3) generating evaluation words of different matrixes according to the different matrixes generated in the step (C4) and the evaluation dictionary in the step (C5).
Specifically, the evaluation word dictionary in the step (C5) comprises evaluation words with four evaluation levels of pitch, rhythm and fluency, notes, phrases, music pieces and music pieces; the evaluation word includes: region error, snatch a shot, drag beat, rhythm instability, temporary change sign error, pitch error, mispronounce, snatch a shot, drag beat, rhythm instability, cross-bar incoherence, phrase incoherence.
Specifically, the specific process of step (C3) is: the path sequencing value rank=p×w, P is an attribute sequence, W is a weight sequence, and the path corresponding to the rank maximum value is the optimal path; adding R, C in step (A4) to sequences R and C, respectively, wherein the eight attribute values of attribute sequence P are in turn the number proportion of element 0 in sequence R, the number proportion of element 0 in sequence C, the relative starting position of element 0 in sequence R, the relative starting position of element 0 in sequence C, the relative ending position of element 0 in sequence R, the relative ending position of element 0 in sequence C, the maximum continuous number proportion of element 0 in sequence R, the maximum continuous number proportion of element 0 in sequence C, and the weight sequence w=w 1waw3w4w5 w6w7w8; r represents a row coordinate or 0 in the matching path, c represents a column coordinate or 0 in the matching path, wherein 0 represents a placeholder for representing insertion and deletion; the sequence R represents a matching sequence of sound information and consists of row coordinates and 0; sequence C represents a matching sequence of the cursive information, consisting of column coordinates and 0.
Specifically, the specific process of step (C4) is: the method comprises the steps of splitting into position indexes of optimal comparison paths generated in step (C3) according to position indexes of a music spectrum file < measurement > tag of extensible markup (XML/musicXML), pre-constructed phrase position indexes and paragraph position indexes, generating splitting marks, and splitting matrixes in step (A2) and step (B2) into sections, phrases and paragraph matrixes respectively according to the splitting marks.
The comparison system of the performance sound information and the music score information comprises an information extraction module, a vector conversion module, a comparison matrix calculation module, a path planning module and a performance evaluation module which are connected in sequence.
Specifically, the information extraction module comprises an audio receiving module and a stored music score; the audio receiving module and the stored music score are respectively connected with the time value information extracting module and the pitch information extracting module;
The vector conversion module comprises a time value vector conversion module and a pitch vector conversion module which are respectively connected with the time value information extraction module and the pitch information extraction module;
the comparison matrix calculation module comprises a time value comparison matrix calculation module and a pitch comparison matrix calculation module which are respectively connected with the time value vector conversion module and the pitch vector conversion module;
The path planning module comprises a sequencing module, a time value minimum variance path planning module and a pitch minimum score path planning module which are respectively connected with the time value comparison matrix calculation module and the pitch comparison matrix calculation module; the pitch minimum score path planning module is sequentially connected with the time value minimum variance path planning module and the sequencing module;
The performance evaluation module comprises a segmentation module and an evaluation module which are connected with the sequencing module;
The sound value information extraction module and the sound height information extraction module are respectively used for extracting sound value information and sound height information of the music score and the sound received by the sound receiving module;
The time value vector conversion module and the sound height vector conversion module are respectively used for converting pitch information and time value information in the melody and the sound into two-dimensional matrixes with corresponding coding formats;
the time value comparison matrix calculation module and the pitch comparison matrix calculation module are respectively used for calculating a pitch comparison matrix and a time value comparison matrix according to the comparison function;
The time value minimum variance path planning module and the pitch minimum score path planning module are used for carrying out path dynamic planning and sorting on the comparison matrix to obtain an optimal path;
The segmentation module is used for segmenting the pitch matrix and the duration matrix into the section, phrase and paragraph matrix.
Specifically, the sorting value in the sorting module is the product of an attribute sequence P and a weight sequence W, where eight attribute values of the attribute sequence P are in turn the number proportion of elements 0 in the sequence R, the number proportion of elements 0 in the sequence C, the relative start position of elements 0 in the sequence R, the relative start position of elements 0 in the sequence C, the relative end position of elements 0 in the sequence R, the relative end position of elements 0 in the sequence C, the maximum continuous number proportion of elements 0 in the sequence R, the maximum continuous number proportion of elements 0 in the sequence C, and the weight sequence w=w 1w2w3w4 w5w6w7w8; r represents a row coordinate or 0 in the matching path, c represents a column coordinate or 0 in the matching path, wherein 0 represents a placeholder for representing insertion and deletion; the sequence R represents a matching sequence of sound information and consists of row coordinates and 0; sequence C represents a matching sequence of the cursive information, consisting of column coordinates and 0.
A computer device includes a processor and a memory in which a computer program is stored that, when loaded and executed by the processor, performs an alignment method.
A computer readable medium has a computer program stored therein, the computer program being loaded and executed by a processor to implement an alignment method.
Compared with the prior art, the invention has the following beneficial effects:
(1) The invention breaks away from dependence on electronic musical instruments and electronic musical instrument accessories by means of voice recognition analysis, and enlarges the range of musical instrument performance evaluation; the method fully considers the unsmooth of a musical instrument learner in the performance training process, matches the actual performance data with the standard performance data through a comparison algorithm, and has more accurate and flexible evaluation results; the evaluation word dictionary is constructed, the evaluation level is more abundant, the evaluation of the performance technology is not limited to overall evaluation, such as overall scoring, and the evaluation granularity can reach the evaluation of the performance technology point of a single performance note, so that the performance training is better assisted.
(2) The invention constructs a comparison function converted into a matrix by coding and based on column vectors (position index vectors) and a dynamic programming algorithm based on the position index on the basis of the position index, can be used for realizing d-dimensional (d is more than or equal to 2) information comparison and generating a comparison path based on the position index, and expands from one-dimensional character string comparison to multi-dimensional matrix comparison.
(3) The invention adopts the playing sound as the input source, thereby eliminating the dependence on the electronic musical instrument and the electronic musical instrument elements in the current playing evaluation system; aiming at the problem that the instrument learner frequently has unsmooth problems such as rebound, sound leakage and the like in the performance training process, the comparison concepts such as 'insert' and 'delete' are introduced, and the actual performance of the instrument learner is identified through a comparison algorithm; the invention realizes multi-level and multi-dimensional performance evaluation by constructing evaluation rules and evaluation word dictionaries of five levels of notes, bars, phrases, paragraphs, tracks and the like and different performance dimensions of pitch, time value, smoothness, strength and the like.
Drawings
Fig. 1 is a block diagram of an alignment system.
Fig. 2 is an example of a coding scheme of a pitch vector matrix.
Fig. 3 is an example of a section and phrase cut.
Fig. 4 is an example of an evaluation dictionary.
Detailed Description
The invention will now be further described with reference to the accompanying drawings and examples, embodiments of which include, but are not limited to, the following examples.
As shown in fig. 1 to 4, the audio receiving module of the present invention is configured to receive performance sounds; the invention obtains the playing sound information and the music score information; then, respectively performing code conversion on the performance sound information and the music score information to generate a matrix; then inputting the matrix into a corresponding comparison function to generate a comparison matrix; then planning a path according to the comparison matrix, and producing an optimal path; then, the first generated matrix is segmented according to the position index of the optimal comparison path to produce a bar, phrase and paragraph matrix; finally, evaluating the bar, phrase and paragraph matrix according to the evaluation dictionary and outputting an evaluation language; the specific implementation process is as follows:
The PITCH information extraction module extracts and stores PITCH information in a music spectrum, the PITCH information being taken from < NOTE < PITCH < STEPS > < OCTAVE > > tagged information in a music spectrum file in an extensible markup format (XML/musicXML).
The [ M112 ] pitch information extraction module extracts and stores pitch information in sound, and the pitch information extraction refers to the patent application number as follows: 201910669985.X, patent name: a music analysis data set construction method and an extraction method in a pitch and duration value extraction method based on the music analysis data set construction method are used for obtaining pitch information.
The [ M121 ] value information extraction module extracts and stores value information in a music spectrum, the value information being taken from information of < NOTE < DURATION > > tag in a music spectrum file in an extensible markup format (XML/MusicXML).
The [ M122 ] timing value information extraction module extracts and stores timing value information in sound, and the timing value information extraction refers to the patent application number as follows: 201910669985.X, patent name: a music analysis data set construction method and an extraction method in a pitch and duration extraction method based on the music analysis data set construction method are used for obtaining duration information.
The pitch information in [ M211 ] is converted into a (128 x n 1) two-dimensional Boolean vector matrix. Wherein the column coordinates represent the position index of each performing note in the pitch sequence, the row coordinates represent 128 semitone notes representing absolute pitches C-1 through G9 in scientific notation, the value of each position in the matrix is represented by 0/1, 0 represents not performing the corresponding note, and 1 represents performing the corresponding note.
Converting the pitch information in [ M212 ] into a (128 x n 2) two-dimensional Boolean vector matrix; the two-dimensional boolean vector matrix is represented by [ M211 ].
Converting the time value information in the M221 into a 1*n 1 two-dimensional numerical vector matrix; wherein the column coordinates represent the position index of each performance note in the pitch sequence.
Converting the time value information in the M223 into a 1*n 2 two-dimensional numerical vector matrix; the two-dimensional numerical vector matrix is represented by [ M221 ].
The PITCH score matrix pitch_s is constructed, initialized to a value of 0, and shaped ((n 1+1)*(n2 +1)).
Inputting the matrix in the M211 and the M212, and updating the scoring matrix S obtained by the M311 according to a scoring function; wherein the scoring function is described below.
Setting the pitch matrix of the cursive spectrum asThe pitch matrix of the sound is/>Wherein x i represents the column vector of the ith column of the spectrum pitch matrix, a j represents the column vector of the jth column of the sound pitch matrix, then:
The PITCH comparison matrix pitch_m is constructed, initialized to a value of 0 and shaped ((n 1+1)*(n2 +1)).
The scoring matrix of the M314 is taken as input, and the PITCH comparison matrix PITCH_M obtained by the M313 is updated according to a matching function. The matching function is described as follows:
The value comparison matrix DURA_M is constructed [ M321 ], the value is initialized to 0, and the shape is ((n 1+1)*(n2 +1)).
The matrix in the M322 is input, and the timing value comparison matrix DURA_M obtained by updating the M321 according to the comparison function is input. Wherein the alignment function is described as follows:
setting the matrix of the curvespectral time value as Sound duration matrix is/>Where y i represents the ith column vector of the curvelet matrix and b j represents the jth column vector of the sound value matrix, then: dura_s (i+1,j+1)=bj/yi′.
Calculating to obtain the best score paths in the pitch comparison matrix of [ M314 ], wherein the number of the best score paths is more than or equal to 1; the calculation rules are described as follows:
Traversing elements in a PITCH comparison matrix PITCH_M, generating R and C according to iteration and assignment rules, and adding R and C into sequences R and C respectively, wherein the iteration and assignment rules are as follows, and 0 represents insertion or deletion and is used for representing multi-performance notes or few-performance notes in the actual performance process:
Inputting a plurality of best score paths (M411) and a score duration matrix of (M322), calculating duration variance of each path, and selecting a path with the smallest variance as a minimum variance path, wherein the number of the minimum variance paths is more than or equal to 1:
Wherein the duration of each position index (i, j) of DURA-S path(i,j) deviates, E (DURA_S path) represents the average of the duration deviations of the entire path, and N represents the number of path positions.
The eight attribute values of the multiple minimum variance paths in the sequence [ M421 ] are respectively obtained and marked as P, and the eight attribute values are sequentially the number proportion of the elements 0 in the sequence R, the number proportion of the elements 0 in the sequence C, the relative starting position of the elements 0 in the sequence R, the relative starting position of the elements 0 in the sequence C, the relative ending position of the elements 0 in the sequence R, the relative ending position of the elements 0 in the sequence C, the maximum continuous number proportion of the elements 0 in the sequence R and the maximum continuous number proportion of the elements 0 in the sequence C.
Inputting eight attribute values of [ M431 ] into a sequencing model for sequencing respectively to obtain an optimal path; the sorting model is that the distribution of the element 0 in the optimal path obeys a certain rule, a path sorting value rank=p×w is calculated, and the path corresponding to the maximum rank is the optimal path; w is the weight sequence w=w 1w2w3w4w5w6w7w8.
The method comprises the steps of dividing a vector matrix into section vector matrixes according to an optimal path sequence, wherein dividing rules are that section initial note position indexes are extracted from a music spectrum file < MEASURE > tag in an extensible tag format and are matched with position indexes of an optimal comparison path, section dividing tags are generated, and pitch vector matrixes and duration vector matrixes are divided into section pitch vector matrixes and section duration vector matrixes respectively according to the section dividing tags.
Dividing the vector matrix into phrase vector matrixes according to the optimal path sequence, wherein the dividing rule is that the initial note position index of a phrase is extracted from the pre-constructed phrase position index and is matched with the position index of the optimal comparison path, phrase dividing marks are generated, and the pitch vector matrixes and the duration vector matrixes are respectively divided into phrase pitch vector matrixes and phrase duration vector matrixes according to the phrase dividing marks;
The vector matrix is segmented into paragraph vector matrices according to the optimal path sequence, wherein the segmentation rule is that the paragraph starting note position index is extracted from the pre-constructed paragraph position index and matched with the position index of the optimal comparison path, a paragraph segmentation mark is generated, and the pitch vector matrix and the time value vector matrix are segmented into a paragraph pitch vector matrix and a paragraph time value vector matrix respectively according to the paragraph segmentation mark.
And (3) evaluating each data tuple of the step (M521) according to an evaluation rule, and outputting a corresponding position index.
And (M522) matching and selecting the evaluation words from a pre-constructed evaluation word dictionary, and outputting corresponding evaluation words. Wherein the evaluation word dictionary comprises evaluation words with multiple dimensions of pitch, duration, fluency and the like.
The invention acquires performance sound information and standard music score information; respectively encoding and converting the performance sound information and the standard music score information into vector matrixes; respectively inputting the vector matrix into a comparison function to generate a comparison matrix; generating an optimal comparison path by adopting a dynamic programming algorithm, a sequencing model and the like; the evaluation of the performance sound information and the score information includes: constructing an evaluation word dictionary; dividing the optimal comparison path into a multi-level evaluation sequence; and generating evaluation information of different layers of sequences according to the evaluation rule.
The specific process of path planning is that the position indexes of the music notes and the sound notes are shifted by 1 positive unit integrally through position index conversion, and 0 is used for representing insertion or deletion; if the whole is changed from 0 to 8 to 1 to 9, 0 is introduced if insertion or deletion is needed, for example 102 34 500 6 78 9 is the result after the change, which represents that a space needs to be inserted after the first position and 2 spaces are inserted after the 5 th position.
The invention adopts the playing sound as the input source, thereby eliminating the dependence on the electronic musical instrument and the electronic musical instrument elements in the current playing evaluation system; aiming at the problem that the instrument learner frequently has unsmooth problems such as rebound, sound leakage and the like in the performance training process, the comparison concepts such as 'insert' and 'delete' are introduced, and the actual performance of the instrument learner is identified through a comparison algorithm; the invention realizes multi-level and multi-dimensional performance evaluation by constructing evaluation rules and evaluation word dictionaries of five levels of notes, bars, phrases, paragraphs, tracks and the like and different performance dimensions of pitch, time value, smoothness, strength and the like.
The present invention can be well implemented according to the above-described embodiments. It should be noted that, based on the above structural design, even if some insubstantial modifications or color-rendering are made on the present invention, the essence of the adopted technical solution is still the same as the present invention, so it should be within the protection scope of the present invention.
Claims (8)
1. A method of comparing performance sound information with music score information, comprising the steps of:
(C1) Obtaining an optimal comparison path of the pitch information according to an obtaining method of the optimal comparison path of the performance pitch information and the melody pitch information;
(C2) Obtaining an optimal comparison path of the time value information according to an obtaining method of the optimal comparison path of the performance time value information and the score time value information;
(C3) Obtaining an optimal comparison path according to the comparison paths in the step (C1) (C2);
(C4) Cutting the pitch matrix and the time value matrix according to the position index of the optimal comparison path in the step (C3) to generate a bar, phrase and paragraph matrix;
(C5) Constructing evaluation rules of different levels in advance to generate an evaluation dictionary;
(C6) Generating evaluation words of different matrixes according to the different matrixes generated in the step (C4) and the evaluation dictionary in the step (C5);
the method for obtaining the optimal comparison path of the performance pitch information and the melody pitch information comprises the following steps:
(A1) Acquiring pitch information of a performance sound and pitch information of a melody;
(A2) Performing code conversion on the pitch information of the performance sound and the pitch information of the melody respectively to generate a pitch matrix;
(A3) Inputting the pitch matrix of (A2) into a pitch comparison function to generate a pitch comparison matrix;
(A4) Performing path planning on the pitch comparison matrix of the step (A3), and generating a pitch minimum score path as an optimal path;
The pitch comparison function in the step (A3) is a composite function, and comprises a pitch scoring function and a pitch matching function;
the pitch scoring function is:
The pitch matching function is:
wherein m 1<g1<u1;
a j represents the column vector for converting the pitch information of the sound into a pitch matrix in step (A2);
x i represents the column vector of converting the cursive pitch information into a pitch matrix in step (A2);
If the expression x i=xi*aj is satisfied between the ith column of the music spectrum pitch matrix and the jth column of the music spectrum pitch matrix, judging that the two columns are matched, assigning the (i+1, j+1) position of the scoring matrix as a matching score m 1, otherwise, assigning the (i+1, j+1) position of the scoring matrix as a matching score u 1;
g 1 represents that each bit of representation is moved in the pitch matrix from left to right and from top to bottom, an insertion operation is introduced, and the insertion score is g 1;
the method for obtaining the optimal comparison path of the performance time value information and the score time value information comprises the following steps:
(B1) Acquiring performance sound duration information and melody duration information;
(B2) Performing code conversion on the (B1) performance sound time value information and the music spectrum time value information respectively to generate a time value matrix;
(B3) Inputting the timing value matrix of the step (B2) into a timing value comparison function to generate a timing value comparison matrix;
(B4) Performing path planning according to the timing value comparison matrix of the step (B3), and generating a timing value minimum variance path as an optimal path;
The value comparison function in step (B3) is: dura_s (i+1,j+1)=bj/yi;
wherein, the matrix of the curvelet time value is The sound duration matrix is b=b 1b2b3...bn2, where y i represents the ith column vector of the curvespectral duration matrix and B j represents the jth column vector of the sound duration matrix; dura_s is a timing value comparison matrix;
The path planning in the step (B4) is the minimum variance path dynamic planning based on the position index, and the specific process is that the directed graph traversal is performed on the value comparison matrix, and the distance from DURA_S (0,0) to DURA_S (0,0) is obtained Calculating the variance of each path to obtain a minimum variance path, and converting the minimum variance path into a (r, c) sequence based on a position index, wherein 0 represents insertion or deletion, and shifting the whole of the position indexes of the music notes and the sound notes by 1 positive unit through position index conversion;
Wherein the duration of each position index (i, j) of dura_s path(i,j) deviates, E (dura_s path) represents the mean of the duration deviations of the entire path, and N represents the number of path positions.
2. The method of claim 1, wherein the evaluation word dictionary of step (C5) includes evaluation words of four evaluation levels of pitch, rhythm, and fluency, notes, phrases, music pieces, and music pieces; the evaluation word includes: region error, snatch a shot, drag beat, rhythm instability, temporary change sign error, pitch error, mispronounce, snatch a shot, drag beat, rhythm instability, cross-bar incoherence, phrase incoherence.
3. The method of claim 1, wherein the specific process of step (C3) is: the path sequencing value rank=p×w, P is an attribute sequence, W is a weight sequence, and the path corresponding to the rank maximum value is the optimal path;
The specific process of the step (C4) is as follows: the segmentation is to match the position index of the optimal comparison path generated in the step (C3) according to the position index of the scalable marked music spectrum file < MEASURE > mark, the pre-constructed phrase position index and the paragraph position index, so as to generate a segmentation mark, and the segmentation mark is used for segmenting the music score into sections, phrases and paragraph matrixes.
4. A system for comparing performance sound information and score information according to any one of claims 1 to 3, comprising an information extraction module, a vector conversion module, a comparison matrix calculation module, a path planning module, and a performance evaluation module, which are sequentially connected.
5. The alignment system of claim 4, wherein the information extraction module comprises an audio receiving module and a stored score; the audio receiving module and the stored music score are respectively connected with the time value information extracting module and the pitch information extracting module;
The vector conversion module comprises a time value vector conversion module and a pitch vector conversion module which are respectively connected with the time value information extraction module and the pitch information extraction module;
the comparison matrix calculation module comprises a time value comparison matrix calculation module and a pitch comparison matrix calculation module which are respectively connected with the time value vector conversion module and the pitch vector conversion module;
The path planning module comprises a sequencing module, a time value minimum variance path planning module and a pitch minimum score path planning module which are respectively connected with the time value comparison matrix calculation module and the pitch comparison matrix calculation module; the pitch minimum score path planning module is sequentially connected with the time value minimum variance path planning module and the sequencing module;
The performance evaluation module comprises a segmentation module and an evaluation module which are connected with the sequencing module;
The sound value information extraction module and the sound height information extraction module are respectively used for extracting sound value information and sound height information of the music score and the sound received by the sound receiving module;
The time value vector conversion module and the sound height vector conversion module are respectively used for converting pitch information and time value information in the melody and the sound into two-dimensional matrixes with corresponding coding formats;
the time value comparison matrix calculation module and the pitch comparison matrix calculation module are respectively used for calculating a pitch comparison matrix and a time value comparison matrix according to the comparison function;
The time value minimum variance path planning module and the pitch minimum score path planning module are used for carrying out path dynamic planning and sorting on the comparison matrix to obtain an optimal path;
The segmentation module is used for segmenting the pitch matrix and the duration matrix into the section, phrase and paragraph matrix.
6. The alignment system of claim 5 wherein the ranking value in the ranking module is the product of the attribute sequence P and the weight sequence W.
7. A computer device comprising a processor and a memory in which a computer program is stored which, when loaded and executed by the processor, implements the alignment method of any of claims 1 to 3.
8. A computer readable medium, in which a computer program is stored, which is loaded and executed by a processor to implement the alignment method of any of claims 1 to 3.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111041068.0A CN113763991B (en) | 2019-09-02 | 2019-09-02 | Comparison method and system of performance sound information and music score information |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910806968.6A CN110600057B (en) | 2019-09-02 | 2019-09-02 | Method and system for comparing performance sound information with music score information |
CN202111041068.0A CN113763991B (en) | 2019-09-02 | 2019-09-02 | Comparison method and system of performance sound information and music score information |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910806968.6A Division CN110600057B (en) | 2019-09-02 | 2019-09-02 | Method and system for comparing performance sound information with music score information |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113763991A CN113763991A (en) | 2021-12-07 |
CN113763991B true CN113763991B (en) | 2024-04-19 |
Family
ID=68856284
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202111041068.0A Active CN113763991B (en) | 2019-09-02 | 2019-09-02 | Comparison method and system of performance sound information and music score information |
CN202111036222.5A Active CN113744764B (en) | 2019-09-02 | 2019-09-02 | Method for obtaining optimal comparison path of performance time value information and score time value information |
CN201910806968.6A Active CN110600057B (en) | 2019-09-02 | 2019-09-02 | Method and system for comparing performance sound information with music score information |
Family Applications After (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202111036222.5A Active CN113744764B (en) | 2019-09-02 | 2019-09-02 | Method for obtaining optimal comparison path of performance time value information and score time value information |
CN201910806968.6A Active CN110600057B (en) | 2019-09-02 | 2019-09-02 | Method and system for comparing performance sound information with music score information |
Country Status (1)
Country | Link |
---|---|
CN (3) | CN113763991B (en) |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113763991B (en) * | 2019-09-02 | 2024-04-19 | 深圳市平均律科技有限公司 | Comparison method and system of performance sound information and music score information |
CN113643676A (en) * | 2020-04-27 | 2021-11-12 | 汲趣艺术科技(上海)有限公司 | Performance evaluation system |
CN111554257A (en) * | 2020-05-07 | 2020-08-18 | 南京邮电大学 | Note comparison system of traditional Chinese national musical instrument and use method thereof |
CN112258932B (en) * | 2020-11-04 | 2022-07-19 | 深圳市平均律科技有限公司 | Auxiliary exercise device, method and system for musical instrument playing |
CN112836080B (en) * | 2021-02-05 | 2023-09-12 | 小叶子(北京)科技有限公司 | Method and system for searching music score through audio |
CN114417915B (en) * | 2021-12-29 | 2024-08-13 | 星星趣弹(苏州)科技有限公司 | Two-dimensional sequence similarity evaluation system for turn-over spectrum |
CN116070077A (en) * | 2022-12-29 | 2023-05-05 | 江苏理工学院 | Music composing effect evaluation method and device for automatic music composing algorithm |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7750224B1 (en) * | 2007-08-09 | 2010-07-06 | Neocraft Ltd. | Musical composition user interface representation |
CN105070298A (en) * | 2015-07-20 | 2015-11-18 | 科大讯飞股份有限公司 | Polyphonic musical instrument scoring method and device |
CN106228996A (en) * | 2016-07-15 | 2016-12-14 | 黄河科技学院 | Vocality study electron assistant articulatory system |
CN106340286A (en) * | 2016-09-27 | 2017-01-18 | 华中科技大学 | Universal real-time musical instrument playing evaluation system |
CN108711336A (en) * | 2018-04-27 | 2018-10-26 | 山东英才学院 | A kind of piano performance points-scoring system and its method |
CN109448682A (en) * | 2018-12-04 | 2019-03-08 | 无锡市思娱文化科技有限公司 | A kind of system and its application method for realizing that microphone is interacted with jukebox |
CN110600057A (en) * | 2019-09-02 | 2019-12-20 | 深圳市平均律科技有限公司 | Method and system for comparing performance sound information with music score information |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5952597A (en) * | 1996-10-25 | 1999-09-14 | Timewarp Technologies, Ltd. | Method and apparatus for real-time correlation of a performance to a musical score |
US20020116196A1 (en) * | 1998-11-12 | 2002-08-22 | Tran Bao Q. | Speech recognizer |
KR100658219B1 (en) * | 2001-06-25 | 2006-12-15 | 어뮤즈텍(주) | Method and apparatus for designating expressive performance notes with synchronization information |
CA2581919A1 (en) * | 2004-10-22 | 2006-04-27 | In The Chair Pty Ltd | A method and system for assessing a musical performance |
JP5593608B2 (en) * | 2008-12-05 | 2014-09-24 | ソニー株式会社 | Information processing apparatus, melody line extraction method, baseline extraction method, and program |
US8575465B2 (en) * | 2009-06-02 | 2013-11-05 | Indian Institute Of Technology, Bombay | System and method for scoring a singing voice |
CN103871426A (en) * | 2012-12-13 | 2014-06-18 | 上海八方视界网络科技有限公司 | Method and system for comparing similarity between user audio frequency and original audio frequency |
CN103824565B (en) * | 2014-02-26 | 2017-02-15 | 曾新 | Humming music reading method and system based on music note and duration modeling |
CN103823867B (en) * | 2014-02-26 | 2017-02-15 | 深圳大学 | Humming type music retrieval method and system based on note modeling |
JP2016184112A (en) * | 2015-03-26 | 2016-10-20 | ヤマハ株式会社 | Ensemble evaluation apparatus |
CN106228961A (en) * | 2016-07-21 | 2016-12-14 | 赵洪云 | Play evaluation methodology and device |
CN107424476A (en) * | 2017-08-01 | 2017-12-01 | 上海破破飞网络科技有限公司 | A kind of instrumental music training mate system |
CN107767847B (en) * | 2017-09-29 | 2020-10-27 | 小叶子(北京)科技有限公司 | Intelligent piano playing evaluation method and system |
CN107818796A (en) * | 2017-11-16 | 2018-03-20 | 重庆师范大学 | A kind of music exam assessment method and system |
CN110070847B (en) * | 2019-03-28 | 2023-09-26 | 深圳市芒果未来科技有限公司 | Musical tone evaluation method and related products |
-
2019
- 2019-09-02 CN CN202111041068.0A patent/CN113763991B/en active Active
- 2019-09-02 CN CN202111036222.5A patent/CN113744764B/en active Active
- 2019-09-02 CN CN201910806968.6A patent/CN110600057B/en active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7750224B1 (en) * | 2007-08-09 | 2010-07-06 | Neocraft Ltd. | Musical composition user interface representation |
CN105070298A (en) * | 2015-07-20 | 2015-11-18 | 科大讯飞股份有限公司 | Polyphonic musical instrument scoring method and device |
CN106228996A (en) * | 2016-07-15 | 2016-12-14 | 黄河科技学院 | Vocality study electron assistant articulatory system |
CN106340286A (en) * | 2016-09-27 | 2017-01-18 | 华中科技大学 | Universal real-time musical instrument playing evaluation system |
CN108711336A (en) * | 2018-04-27 | 2018-10-26 | 山东英才学院 | A kind of piano performance points-scoring system and its method |
CN109448682A (en) * | 2018-12-04 | 2019-03-08 | 无锡市思娱文化科技有限公司 | A kind of system and its application method for realizing that microphone is interacted with jukebox |
CN110600057A (en) * | 2019-09-02 | 2019-12-20 | 深圳市平均律科技有限公司 | Method and system for comparing performance sound information with music score information |
CN113744764A (en) * | 2019-09-02 | 2021-12-03 | 深圳市平均律科技有限公司 | Method for obtaining optimal comparison path of playing time value information and music score time value information |
Also Published As
Publication number | Publication date |
---|---|
CN110600057B (en) | 2021-12-10 |
CN113763991A (en) | 2021-12-07 |
CN110600057A (en) | 2019-12-20 |
CN113744764A (en) | 2021-12-03 |
CN113744764B (en) | 2024-04-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN113763991B (en) | Comparison method and system of performance sound information and music score information | |
CN110147451B (en) | Dialogue command understanding method based on knowledge graph | |
WO2019158927A1 (en) | A method of generating music data | |
CN113010730B (en) | Music file generation method, device, equipment and storage medium | |
CN114996467A (en) | Knowledge graph entity attribute alignment algorithm based on semantic similarity | |
CN110956039A (en) | Text similarity calculation method and device based on multi-dimensional vectorization coding | |
CN115293139A (en) | Training method of voice transcription text error correction model and computer equipment | |
CN115293138A (en) | Text error correction method and computer equipment | |
CN114021573B (en) | Natural language processing method, device, equipment and readable storage medium | |
CN109102006B (en) | Music automatic marking method based on audio characteristic induction information enhancement | |
CN117350378A (en) | Natural language understanding algorithm based on semantic matching and knowledge graph | |
CN116483337A (en) | API completion method based on prompt learning and data enhancement | |
CN114021572B (en) | Natural language processing method, device, equipment and readable storage medium | |
CN114579763A (en) | Character-level confrontation sample generation method for Chinese text classification task | |
CN114925197A (en) | Deep learning text classification model training method based on topic attention | |
Srivatsan et al. | Checklist models for improved output fluency in piano fingering prediction | |
KR102227415B1 (en) | System, device, and method to generate polyphonic music | |
Chuan | An active learning approach to audio-to-score alignment using dynamic time warping | |
Wai et al. | Myanmar (Burmese) String Similarity Measures based on Phoneme Similarity | |
CN110781678B (en) | Text representation method based on matrix form | |
WO2024042650A1 (en) | Training device, training method, and program | |
JP2014191490A (en) | Abbreviation generation device, abbreviation generation method and program | |
US20240013760A1 (en) | Text providing method and text providing device | |
CN117935859A (en) | Partial multi-label music style characteristic selection method integrating characteristic similarity | |
Tsunenari et al. | Interactive Musical Editing System to Support Human Errors and Offer Personal Preference for an Automatic Piano |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |