CN111724813A

CN111724813A - LSTM-based piano playing automatic scoring method

Info

Publication number: CN111724813A
Application number: CN202010555829.3A
Authority: CN
Inventors: 田君; 程昭德
Original assignee: Dongguan University of Technology
Current assignee: Dongguan University of Technology
Priority date: 2020-06-17
Filing date: 2020-06-17
Publication date: 2020-09-29

Abstract

The invention discloses an LSTM-based piano playing automatic evaluation method, which comprises the following steps: establishing a music database; extracting MIDI music physical characteristics of the music file; vectorizing the characteristic information of all note files of the piano music file to be scored; coding the related note information into a corresponding one-hot vector; encrypting the tone state by a binary vector expression mode; inputting a matrix; establishing a grading model based on LSTM, and training the grading model; and (4) grading the piano music file to be evaluated by using the trained grading model, and outputting a grading result. The music file is evaluated by establishing the LSTM-based scoring model, and the scoring considers the accuracy, and also considers factors such as music literacy, style expression and emotion expression, so that the scoring method is more scientific and objective.

Description

LSTM-based piano playing automatic scoring method

Technical Field

The invention relates to an automatic scoring method for piano playing, in particular to an automatic scoring method for piano playing based on an LSTM (Long Short-Term Memory, Long Short-Term Memory neural network).

Background

With the continuous improvement of the substance living standard of people, piano teachers become scarce resources gradually, and the research and development market demand of piano learning auxiliary software is increased year by year. The evaluation of the traditional piano playing level is carried out by adopting an artificial scoring mode, and particularly the field examination of the piano playing is carried out. After the live piano is played, the playing content of the test music is divided into various specific detail requirements in a form of a table, and each marker carries out scoring respectively according to the level exerted by the live player.

For piano beginners, they still need to exercise themselves without the presence of a music teacher or music professional. However, without the scores of professional persons, the beginners do not really know the performance quality of own pianos, so that the self-learning effect is greatly discounted. Therefore, the exercise level of the exerciser can be intuitively reflected by using the piano scoring system. The principle of the existing piano scoring system in the market at present is mostly a direct comparison method: namely, the note information of the sample to be evaluated played by the practicer is mechanically compared with the note information of the standard sample, and the score is directly given out through the accuracy. Such scoring principles are too mechanical to embody the practicer's true level. Therefore, a piano scoring system has recently appeared, which processes the sample data of the information to be evaluated based on the first method, extracts the information characteristics, and compares the information characteristics with the standard sample. However, the extraction of the sample characteristics is too simple at present, and the influence of factors such as fluency of playing, emotional expression, difficulty degree of songs and the like on the rationality of evaluation is often ignored.

Disclosure of Invention

In order to overcome the defects and problems in the prior art, the invention provides an LSTM-based piano playing automatic scoring method, so as to make more objective and reasonable evaluation on the score of the piano playing level of a piano practicer.

The invention is realized by the following technical scheme: an LSTM-based piano playing automatic scoring method, the method steps comprising:

step S1, establishing a music database to store and manage MIDI file information;

step S2, extracting MIDI music physical characteristics of the piano music file;

step S3, vectorizing the characteristic information of all note files of the piano music file to be scored so as to combine the note information with the corresponding time;

step S4, coding the related note information into a corresponding one-hot vector;

step S5, encrypting the tone state through the expression mode of the binary vector;

step S6, inputting a matrix, wherein the ordinate of the matrix is a time sequence, and the abscissa of the matrix is a tone value;

step S7, establishing a grading model based on LSTM, and training the grading model to obtain model parameters;

and step S8, scoring the piano music file to be evaluated by using the trained scoring model, and outputting a scoring result.

Preferably, in step S5, the encrypting the pitch status by the expression of the binary vector is performed by: a state in which the piano key is not currently depressed is represented by [0, 0 ]; the state in which the piano key is currently depressed is represented by [1, 0 ]; a state in which the piano key is depressed at the present time and also depressed at the previous time is denoted by [1, 1 ].

Preferably, in the step S7, when training is performed using the score model, the training is performed by using a small batch gradient descent method and a log-maximum natural function as a loss function.

Preferably, in step S7, when the score model is used for training, a log-maximum natural function is used as the loss function, where Mini-batch is 10, the number of iterations is 6000, the learning rate is 0.0001, and the optimizer is a RMSProp optimizer.

Preferably, in step S1, the music database is a ySQL database.

Preferably, the MIDI music physical characteristics include tone and intensity.

Preferably, in step S4, when the note information is encoded into a corresponding One-hot vector, One-hot encoding is used.

Preferably, in step S8, the scoring result includes five scoring levels, i.e., good, medium, poor and bad.

Compared with the prior art, the music file scoring method based on the LSTM is used for evaluating the music file by establishing the scoring model based on the LSTM, and the scoring method is more scientific and objective because the accuracy rate is considered and factors such as style expression and emotion expression are considered.

Drawings

FIG. 1 is a flow chart illustrating the steps of the scoring method of the present invention.

Detailed Description

To facilitate understanding of those skilled in the art, the present invention is described in further detail below with reference to the accompanying drawings and specific embodiments.

As shown in fig. 1, an LSTM-based piano playing automatic scoring method includes the steps of:

and step S8, scoring the piano music file to be evaluated by using the trained scoring model, and outputting a scoring result. Specifically, the piano music file to be evaluated is subjected to the processing of the steps S2 to S6, and then the scoring process is performed by using the scoring model trained in the step S7, so that the scoring result can be obtained.

When the scoring method is applied, data in common formats such as dates, characters, numbers and the like are mostly contained in the data tables in the music database, and music files are in the MIDI format. Therefore, when database construction for piano performance automatic scoring is performed, the music database is preferably mySQL database.

In one preferred embodiment, in step S5, the encrypting the tone state through the expression of the binary vector is performed in the following specific way: a state in which the piano key is not currently depressed is represented by [0, 0 ]; the state in which the piano key is currently depressed is represented by [1, 0 ]; a state in which the piano key is depressed at the present time and also depressed at the previous time is denoted by [1, 1 ].

In step S6, the matrix is a long time sequence matrix, which includes characteristic information of pitch, tone, duration, melody, etc. of music; in the LSTM-based scoring model in step S7, LSTM (Long Short-term memory network) is a neural network that processes time-series data, and the matrix can be used as an input parameter of LSTM. In addition, since the piano keyboard is 88 keys and binary vectors are used, the abscissa is 176 dimensions.

In the present embodiment, since the input is a 176-dimensional matrix, in the LSTM-based scoring model, the number of neural nodes of the input layer may be determined to be 176; according to the preferred data after multiple experiments, the hidden layer is preferably three bidirectional LSTM layers, and the number of the neural nodes is: 704. 352, 176. The output layer is a Softmax output layer (in this embodiment, the Softmax output layer selects a Softmax function as an activation function), the number of the neural nodes is 5, and the output layer and the third hidden layer are fully connected. To prevent overfitting, the Dropout algorithm is optimized.

In one specific embodiment, in the step S7, when training by using the scoring model, a small batch gradient descent method is adopted, and a log-maximum natural function is adopted as a loss function for training; when the scoring model is used for training, and a logarithmic maximum nature function is used as a loss function, wherein Mini-batch is 10, the iteration number is preferably 6000, the learning rate is preferably 0.0001, and the optimizer is an RMSProp optimizer. In step S4, when the note information is encoded into the corresponding One-hot vector, One-hot encoding is used.

For ease of understanding, the following description is given in terms of a specific embodiment:

the LSTM-based piano playing automatic scoring method is applied to a piano playing automatic scoring system, the piano playing automatic scoring system mainly comprises a PC end and a MIDI piano, and the PC end and the MIDI piano are subjected to information transmission through a transmission data line. Wherein. In this embodiment, the PC is a Window system with android APP software. Because a keras development environment is adopted for the advantages of high efficiency and strong portability, the MIDI piano is a common brand on the market, Yamaha (YAMAHA) MX88 is selected in the embodiment, and a CH345T chip is adopted in data transmission, so that MIDI signals of the electronic piano can be converted into USB standard serial port signals, and the two-way transmission of the MIDI signals is realized.

When the LSTM-based piano playing automatic scoring method is applied, the specific steps are as follows:

the method comprises the following steps: directly establishing a music database by using mySQL at a PC end in the piano playing automatic scoring system;

step two: extracting MIDI music physical characteristics of music files in a music database: pitch, intensity, timbre, and duration. In the embodiment of the invention, only piano music is evaluated, and the input matrix is a time sequence matrix and contains duration information; the tone color is related to the material and the structure of the sounding body, and because the piano playing automatic scoring method is used for evaluating piano tracks, the piano playing automatic scoring method only has one tone color. The timbre and duration features do not need to be extracted. And pitch, both of which can be extracted directly in the MIDI signal Delta-time.

Step three: vectorizing the characteristic information of all note files of the piano music file to be scored to combine the note information with their corresponding times. Since the music characteristic information is discrete in the Delta-time of MIDI music, it is necessary to convert the time increment into absolute time. The time stamp is modified to approximate the 1/16 th note, i.e., the denominator of each track information is changed from 4 to 16, in order to combine the note information with the corresponding time, and avoid that part of the note information cannot be obtained.

Step four: coding the related note information into a corresponding one-hot vector; because the music characteristic information is not continuous but discrete, the embodiment of the invention adopts One-hot coding to map the discrete music characteristic to the Euclidean space during vectorization.

Step five: encrypting the tone state by a binary vector expression mode; in the embodiment of the present invention, the manner of encrypting the tone status by using the binary vector is shown in table 1 below:

TABLE 1 expression of binary vectors

As will be understood from the above description, at a certain time, several tones may be played simultaneously, and usually, a tone has three states.

Compared with the traditional method of describing the piano key state by using two states, the tone state is encrypted by adopting the expression mode of the binary vector in the embodiment, so that the time for expressing music by the encrypted information can be better, the parameter redundancy is reduced, and the accuracy of subsequent training is improved.

Step six: the matrix is input, the ordinate of the matrix is time series, the abscissa is tone value, since the piano keyboard here is 88 keys, the abscissa is 176 dimensions since the state of the tone is expressed by binary vectors.

Step seven: establishing a grading model based on LSTM, and training the grading model to obtain model parameters; when the scoring model is used for training, a small-batch gradient descent method is adopted, and a log-extreme natural function is adopted as a loss function for training; in addition, when a log-maximum natural function is adopted as a loss function in the training by using the scoring model, Mini-batch is 10, the number of iterations is preferably 6000, the learning rate is preferably 0.0001, and the optimizer is a RMSProp optimizer. To prevent overfitting, optimization was performed using the Dropout algorithm. In this embodiment, a large number of piano music files of known classification results are trained as the training set for the score model training, so that each model parameter can be obtained.

The bidirectional LSTM is a neural network with a memory function, has a forward propagation layer and a backward propagation layer, and takes the past note information and the future note information into consideration when the model analyzes the notes at a certain moment, which is similar to the appreciation process of human beings on music melodies. In other words, musical style expressions and emotional expressions are the time expression of notes, and the two-way LSTM model can capture this information well. Therefore, in the scoring method in the embodiment, factors such as the style expression and the emotion expression are considered, so that the scoring method is more scientific and objective.

Step eight: and (4) grading the piano music file to be evaluated by using the trained grading model, and outputting a grading result. Specifically, the piano music file to be evaluated is subjected to the processing of the steps S2 to S6, and then the scoring processing is performed by using the scoring model trained in the step S7, so that a scoring result can be obtained, and the scoring result is divided into: five grades of scores, good, medium, poor and bad.

According to the automatic piano playing scoring method based on the LSTM, which is provided by the embodiment of the invention, evaluation is carried out on piano music files based on the LSTM model for the first time, the LSTM is very suitable for processing the problems highly related to time sequences, the effect is better than that of the existing BP model algorithm in the aspect of music file evaluation, the accuracy is high, and factors such as style expression and emotion expression are better considered as evaluation factors by utilizing an artificial neural network method, so that compared with the traditional music file evaluation method, the scoring is more comprehensive and reasonable, and the scoring is more scientific and objective.

The above embodiments are preferred implementations of the present invention, and are not intended to limit the present invention, and any obvious alternative is within the scope of the present invention without departing from the inventive concept thereof.

Claims

1. An LSTM-based piano playing automatic scoring method is characterized by comprising the following steps:

2. A scoring method according to claim 1, wherein: in step S5, the encrypting the tone state by the expression mode of the binary vector includes: a state in which the piano key is not currently depressed is represented by [0, 0 ]; the state in which the piano key is currently depressed is represented by [1, 0 ]; a state in which the piano key is depressed at the present time and also depressed at the previous time is denoted by [1, 1 ].

3. A scoring method according to claim 1, wherein: in step S7, when training is performed using the score model, a small batch gradient descent method is used, and a log-maximum natural function is used as a loss function for training.

4. A scoring method according to claim 3, wherein: in step S7, when the score model is used for training, and a log-maximum natural function is used as a loss function, where Mini-batch is 10, the iteration number is 6000, the learning rate is 0.0001, and the optimizer is an RMSProp optimizer.

5. A scoring method according to claim 1, wherein: in step S1, the music database is a ySQL database.

6. A scoring method according to claim 1, wherein: the MIDI music physical characteristics comprise tone and intensity.

7. A scoring method according to claim 1, wherein: in step S4, when the note information is encoded into a corresponding One-hot vector, One-hot encoding is adopted.

8. A scoring method according to claim 1, wherein: in step S8, the scoring result includes five scoring levels, i.e., good, medium, poor and bad.