JP2023081946A - Learning device, automatic music transcription device, learning method, automatic music transcription method and program - Google Patents

Abstract

To provide acoustic processing technology for automatically generating a musical score from audio data where a pitch and a section of each sound are not clear.SOLUTION: A learning device 100 comprises: a learning data acquisition section for acquiring a single tone sound source and pitch information as learning data of a first machine learning model, acquiring a sound source of a transcription object and musical score information as learning data of a second machine learning model, performing preprocessing on the single tone sound source and the sound source of the transcription object, and acquiring respective spectrograms; a first model learning section for performing learning by pitch information so that the spectrogram of the single tone sound source is input as learning input data and prediction probability of the pitch of the single tone sound source is output; and a second model learning section for performing learning by musical score information so that a feature map generated by inputting the spectrogram of the sound source of the transcription object to the learnt first machine learning model is input as learning input data and prediction probability that notes exist in the section of fixed length in the feature map is output.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to sound processing technology.

2. Description of the Related Art Automatic music transcription technology for automatically generating musical scores from audio data has been conventionally known. For example, Japanese Unexamined Patent Application Publication No. 2007-033479 describes a technique for automatically transcribing musical scores from acoustic signals played by a single musical instrument even when a plurality of sounds are played at the same time.

JP 2007-033479

However, in the conventional automatic transcription, it is possible to transcribe music with a relatively high degree of accuracy in the case of audio data in which the musical score is accurately played or sung, and the pitch and interval of each note are clear. In the case of audio data in which pitches and sections of sounds are not clear, automatic transcription as expected is difficult.

In view of the above problems, an object of the present disclosure is to provide a sound processing technique for automatically generating musical scores from various audio data more effectively.

In order to solve the above problems, one aspect of the present disclosure is to learn teacher data that pairs a first spectrogram generated from a single tone sound source and corresponding pitch information, according to the input of the spectrogram , a first model learning unit that learns a first machine learning model that outputs a first feature map indicating the predicted probability of the corresponding pitch; the first feature map that the first model learning unit outputs; and information paired to learn the predicted probability of the pitch output according to the input to the first machine learning model of the second spectrogram generated from the sound source to be transcribed. The present invention relates to a learning device having a second model learning unit that learns a second machine learning model that outputs information for generating a musical score according to the input of the first feature map.

Advantageous Effects of Invention According to the present disclosure, it is possible to provide an acoustic processing technique for automatically generating a musical score from audio data in which the pitch and section of each sound are unclear.

1 is a schematic diagram of an automatic music transcription device having a trained machine learning model according to one embodiment of the present disclosure; FIG. 1 is a block diagram showing the functional configuration of a learning device according to an embodiment of the present disclosure; FIG. FIG. 2 is a schematic diagram illustrating the configuration of a feature map generation model according to one embodiment of the present disclosure; FIG. 2 is a schematic diagram showing the configuration of a note existence probability prediction model according to an embodiment of the present disclosure; FIG. 4 is a conceptual diagram illustrating the relationship between feature maps and default boxes according to one embodiment of the present disclosure; 4 is a flow chart showing learning processing of a feature map generation model according to an embodiment of the present disclosure; 10 is a flowchart showing learning processing of a note presence probability prediction model according to an embodiment of the present disclosure; 1 is a block diagram showing the functional configuration of an automatic music transcription device according to an embodiment of the present disclosure; FIG. 4 is a flow chart showing an automatic music transcription process according to one embodiment of the present disclosure; 1 is a block diagram showing the hardware configuration of a learning device and an automatic music transcription device according to an embodiment of the present disclosure; FIG.

In the following embodiments, an automatic music transcription apparatus is disclosed that generates musical score information from a sound source (audio data that is sound waveform data) using a machine learning model.

Conventional automatic transcription techniques focus on pitch prediction, and one of the problems in automatic transcription is the prediction of onsets and offsets that indicate breaks between notes. The automatic music transcription device according to the present disclosure predicts the onset and offset in the sound source by SSD (Single Shot Detection), which is one of machine learning models.

SSD is a method of detecting an object in an input image using one neural network. That is, the input to the neural network is an image, and the output is the center coordinates, height, width and object type prediction probabilities of a plurality of rectangular regions (called default boxes in SSD). Default boxes are prepared as a preset number of candidates according to the size of the input image, most of the default boxes are removed from candidates by post-processing (NMS: Non-Maximum Suppression, etc.), and the remaining default boxes are used as detection results. That's what it means.

The input to the neural network in the automatic music transcription apparatus according to the present disclosure is the waveform data or spectrogram of the musical tone to be transcribed, and the output is the onset, offset and pitch of the musical tone. and width corresponding to the onset and offset (ie shape or length of the note), and pitch corresponding to the type of object in the SSD.

To summarize the embodiment described later, the automatic transcription device utilizes two pre-trained machine learning models (such as convolutional neural networks), one model outputs the predicted probability of pitch from a monophonic sound source, the other model is to output the predicted probability that a note exists in a fixed-length section of the feature map from the feature map. The automatic transcription device inputs the sound source to be transcribed into the former learned machine learning model (feature map generation model), and applies each feature map generated from the convolution layer of the learned feature map generation model to the latter learned machine. input to the learning model (note existence probability prediction model), and predicted existence of each pitch in a fixed-length interval or default box output from the learned note existence probability prediction model for each point of each feature map Generate score information based on probability.

Feature maps generated by a trained feature map generation model have different temporal resolutions as a result of convolution, and fixed-length intervals or default boxes have different temporal lengths. Therefore, it is possible to identify onsets and offsets of notes of different lengths by detecting notes of the same length as the fixed-length section for each feature map using the note presence probability prediction model.

First, referring to FIG. 1, an automatic music transcription apparatus according to an embodiment of the present disclosure will be described. FIG. 1 is a schematic diagram illustrating an automatic music transcription device having a trained machine learning model according to one embodiment of the present disclosure.

As shown in FIG. 1, an automatic music transcription device 200 according to one embodiment of the present disclosure has two types of trained models implemented as any type of neural network, such as, but not limited to, a convolutional neural network. Then, using the machine learning model learned by the learning device 100 using the learning data storage 50, musical score information is generated from the sound source to be transcribed.

A learning device according to one embodiment of the present disclosure will now be described with reference to FIGS. The learning device 100 uses the learning data in the learning data storage 50 to learn the feature map generation model and the note existence probability prediction model used by the automatic transcription device 200 . FIG. 2 is a block diagram showing the functional configuration of the learning device according to one embodiment of the present disclosure.

As shown in FIG. 2 , the learning device 100 has a learning data acquisition unit 110 , a first model learning unit 120 and a second model learning unit 130 .

The learning data acquisition unit 110 acquires the single tone sound source and the pitch information as learning data for the feature map generation model, acquires the sound source to be transcribed and the musical score information as learning data for the note existence probability prediction model, Preprocessing is performed on the monophonic sound source and the sound source to be transcribed, and spectrograms of each are obtained.

Specifically, the learning data acquisition unit 110 retrieves from the learning data storage 50 single-note or single-note sound sources (for example, 12 types of sound sources from “do” to “shi”) for learning the feature map generation model. ) and pitch information (e.g. pitches from "do" to "b"), and preprocess the waveform data of the obtained monophonic sound source (for example, short-term Fourier transform, etc.). ) to generate a learning data set of the spectrogram and pitch information of each single tone sound source.

In addition, the learning data acquisition unit 110 obtains from the learning data storage 50 waveform data of a monophonic sound source (singing sound source, etc.) and musical score information (time-series change in pitch, etc.) for learning the note existence probability prediction model. and perform preprocessing (for example, short-time Fourier transform) on the obtained monophonic sound source waveform data to generate a learning data set of the monophonic sound source spectrogram and musical score information. . Here, the musical score information may conform to the MIDI (Musical Instrument Digital Interface) standard, for example.

Typically, a spectrogram represents the intensity of signal components on the time and frequency axes and is produced by short-time Fourier transforming waveform data. Various parameters need to be set for the short-time Fourier transform, for example, FFT window width: 1024, sampling frequency: 16 kHz, overlap width: 768, window function: Hanning window, and filter bank: Mel filter bank ( 128 band), etc., a short-time Fourier transform may be performed. After conversion into a spectrogram, a fixed number of samples (for example, 1024 samples) may be extracted along the time axis. Further, the spectrogram according to the present embodiment may be obtained by logarithmically transforming the frequency axis so as to refine the low frequency components.

The first model learning unit 120 receives a spectrogram of a single sound source as input data for learning, and learns a feature map generation model based on pitch information so as to output a predicted probability of the pitch of the single sound source.

For example, as shown in Fig. 3, the feature map generation model is constructed by a convolutional neural network including multiple convolution layers, and is realized as an SSD that converts the spectrogram of the input monophonic sound source into the predicted probability of the pitch. . Here, the pitch is expressed as a discrete value instead of a continuous value, and may be expressed as a one-hot vector. Note that when a noisy sound source such as a percussion instrument is also targeted for learning, a single note or single note of the noisy sound source may be included in the data set. In that case, a class expressing noise may be set as the pitch class and used as the teacher label.

The first model learning unit 120 inputs the spectrogram of the single tone sound source of the input data for learning to the feature map generation model, and adjusts the error between the output from the feature map generation model and the pitch information of the output data for learning to be small. , update the parameters of the feature map generation model by backpropagation. Here, without limitation, the cross entropy between the output of the feature map generation model and the pitch of the learning output data may be used as the loss function indicating the error.

For example, when a predetermined learning end condition is satisfied, such as updating processing for a predetermined number of learning data has been completed, error has converged to a predetermined threshold or less, or error improvement has converged to a predetermined threshold or less. , the first model learning unit 120 sets the updated feature map generation model as a learned machine learning model.

The second model learning unit 130 inputs a feature map generated by inputting a spectrogram of a sound source to be transcribed into a learned feature map generation model as input data for learning, and assigns a fixed-length section of the feature map to musical notes. A note existence probability prediction model is learned from the musical score information so as to output the prediction probability that

For example, the note presence probability prediction model is constructed by a convolutional neural network including multiple convolution layers, as shown in FIG. It is implemented as an SSD that converts a feature map into a predicted probability that there is a note of the same length as a fixed-length interval starting at each point of the feature map. For example, when transcribed with 12 notes from C to B, each point on the feature map has a predicted probability that there are 13 different note or pitch classes for each pitch from C to B and rests (silences). .

As described above, the trained feature map generation model includes multiple convolution layers, from which a feature map of the spectrogram of a monophonic sound source is generated. The generated feature map is a feature map with different temporal resolutions depending on the level of the convolution layer as shown in FIG. Typically, convolutional layers relatively close to the input layer produce feature maps with relatively high temporal resolution (32 Hz in the example shown), as shown in FIG. Close convolutional layers produce feature maps with relatively low temporal resolution (16 Hz in the example shown). With fixed-length intervals or default boxes as shown, intervals in feature maps with higher temporal resolution occupy less time than intervals in feature maps with lower temporal resolution. Therefore, it is possible to derive the existence prediction probabilities of notes having different temporal lengths, and to specify the temporal lengths of the notes.

The second model learning unit 130 inputs the spectrogram of the sound source of the learning input data to the learned feature map generation model, inputs each feature map generated by the learned feature map generation model to the note existence probability prediction model, The parameters of the note presence probability prediction model are updated by back propagation so that the error between the output from the note presence probability prediction model and the musical score information of the learning output data is reduced.

Here, as the loss function indicating the error, without limitation, a weighted sum of the timing error and the confidence error calculated from the output of the note presence probability prediction model and the time-series change of the pitch may be used. The time-series change in pitch is expressed by collecting a plurality of sets of start timing, end timing, and pitch of a piece of music, and is derived from musical score information. The set may also be called a pronunciation. For example, the chronological change in pitch is pronunciation #1: "0:00 to 0:02, A(la)3", pronunciation #2: "0:03 to 0:05, B (b) 3", pronunciation #3: "0:05 to 0:08, C (do) 4", and so on. The default box shown in FIG. 5 represents one pronunciation and has multiple channels. Each first sample of each channel of a default box has a predicted onset value, a predicted offset value, and a pitch class predicted probability of the pronunciation of that default box. That is, there are a total of 2+(number of pitch classes) channels.

The second model learning unit 130 searches for the default box that minimizes the sum of the onset and offset for each pronunciation, and obtains the timing error and confidence error for the detected default box and pronunciation. Here, the timing error may be the sum of the deviation of the start timing considering the predicted onset and the deviation of the end timing considering the predicted offset. However, as a representation of the difference, a relative value based on the length of the default box may be used. Alternatively, the confidence error may be a cross-entropy calculated from the pronounced pitch and the predicted pitch. Note that a class representing silence may also be prepared as a teacher label, in which case it is possible to predict an interval with no pronunciation.

The second model learning unit 130 may reduce the default boxes set for each point of each feature map according to NMS (Non-Maximum Suppression), and use the remaining default boxes as predicted pronunciations. Specifically, the second model learning unit 130 first obtains the note presence prediction probability for each pitch class for each default box. After that, the second model learning unit 130 may delete default boxes whose predicted probabilities are less than or equal to a predetermined threshold value (for example, 0.9). The second model learning unit sets a threshold for the intersection/union of the remaining default boxes, deletes one of the default boxes equal to or greater than the threshold, and eliminates duplicate default boxes. The second model learning unit 130 uses the remaining default boxes as predicted pronunciations.

For example, when a predetermined learning end condition is satisfied, such as updating processing for a predetermined number of learning data has been completed, error has converged to a predetermined threshold or less, or error improvement has converged to a predetermined threshold or less. , the second model learning unit 130 sets the updated note existence probability prediction model as a learned model.

In one embodiment, the first model learning unit 120 learns a feature map generation model for each of multiple types of audio components, and the second model learning unit 130 learns a sound source to be transcribed including multiple types of audio components. A note presence probability prediction model may be trained to output a prediction probability that a note exists for each audio component type.

For example, the feature map generation model and the note presence probability prediction model may be applied to a piece of music including monophonic vocals and accompaniment. In this case, the vocal feature map generation model and the accompaniment feature map generation model are composed of vocal learning data composed of pairs of vocal single-tone sound sources and pitch information, and accompaniment single-tone sound sources and pitch information. Learning is performed in the same manner as the above-described learning process using accompaniment learning data composed of pairs. On the other hand, the vocal note presence probability prediction model and the accompaniment note presence probability prediction model use the learning sound source and score information to generate a learned vocal feature map generation model and a learned accompaniment feature map generation model. A feature map generated by inputting a model is used as an input, and learning is performed in the same manner as the learning process described above.

Alternatively, the feature map generation model and the note presence probability prediction model may be applied to a piece of music including multiple parts for each instrument. The learning process for a piece of music including vocals and accompaniment is similar to the learning process described above, but in this case, the output of the note existence probability prediction model is the prediction probability that a specific note of a specific part exists in a fixed-length section of the feature map. may For example, the predicted probabilities of the presence of specific notes in specific parts such as "male A3 pitch" and "female A3 pitch" may be output.

Alternatively, the present disclosure may be applied to music with beats. In this case, the output of the note presence probability prediction model may be in terms of the onset and offset of the time signature, eg the predicted probability that the default box is one beat.

FIG. 6 is a flow chart showing learning processing of a feature map generation model according to an embodiment of the present disclosure. The learning process is implemented by the learning device 100 described above or the processor of the learning device 100 .

As shown in FIG. 6, in step S101, the learning data acquisition unit 110 acquires a pair of a single sound source and pitch information from the learning data storage 50. FIG. For example, there are 13 pitches of 12 pitches from "do" to "si" and silence, and single tone sound sources corresponding to the 13 pitches may be acquired.

In step S102, the learning data acquisition unit 110 preprocesses the acquired monophonic sound source. Specifically, the learning data acquisition unit 110 performs preprocessing (for example, short-time Fourier transform) on waveform data of a single sound source to acquire a spectrogram of the single sound source.

In step S103, the first model learning unit 120 learns a feature map generation model based on the pair of preprocessed single sound source and pitch information. For example, the feature map generation model consists of a convolutional neural network, which transforms the input sound source into pitch prediction probabilities. Specifically, the first model learning unit 120 inputs the spectrogram of a single tone sound source to the feature map generation model, and uses back propagation to reduce the error between the output of the feature map generation model and the pitch information. Update map generation model parameters.

In step S104, the first model learning unit 120 determines whether the learning end condition is satisfied. The predetermined learning end condition may be, for example, that the updating process has ended for a predetermined number of learning data, the error has converged below a predetermined threshold value, or the error improvement has converged below a predetermined threshold value. good. If a predetermined learning end condition is satisfied (S104: YES), the first model learning unit 120 may set the updated feature map generation model as a learned model. On the other hand, if the predetermined learning end condition is not satisfied (S104: NO), the process proceeds to step S101 and repeats the steps described above.

FIG. 7 is a flow chart showing learning processing of a note existence probability prediction model according to an embodiment of the present disclosure. The learning process is realized by the learning device 100 described above or the processor of the learning device 100 .

As shown in FIG. 7, in step S201, the learning data acquisition unit 110 acquires a pair of monophonic sound source and musical score information from the learning data storage 50. FIG. For example, the monophonic sound source may be waveform data of a singing sound source, and the musical score information indicates the musical score of the monophonic sound source.

In step S202, the learning data acquisition unit 110 preprocesses the acquired monophonic sound source. Specifically, the learning data acquisition unit 110 performs preprocessing (for example, short-time Fourier transform) on waveform data of a monophonic sound source to acquire a spectrogram of the monophonic sound source.

In step S203, the second model learning unit 130 inputs the preprocessed monophonic sound source to the learned feature map generation model, and acquires the feature map generated by the learned feature map generation model. Specifically, the second model learning unit 130 acquires feature maps generated from each convolutional layer of the learned feature map generation model. The generated feature maps have different temporal resolutions depending on the degree of convolution of each convolution layer.

In step S204, the second model learning unit 130 learns a note presence probability prediction model based on the acquired pair of feature map and musical score information. For example, the note presence probability prediction model is configured by a convolutional neural network, and converts an input feature map into note presence prediction probabilities that notes exist in a fixed-length section of the feature map. Specifically, the second model learning unit 130 inputs each feature map to the note presence probability prediction model, and uses back propagation to reduce the error between the output of the note presence probability prediction model and the musical score information. Update the parameters of the existence probability prediction model.

In step S205, the second model learning unit 130 determines whether the learning termination condition is satisfied. The predetermined learning end condition may be, for example, that the updating process has ended for a predetermined number of learning data, the error has converged below a predetermined threshold value, or the error improvement has converged below a predetermined threshold value. good. If the predetermined learning termination condition is satisfied (S205: YES), the second model learning unit 130 may set the updated note presence probability prediction model as a learned model. On the other hand, if the predetermined learning end condition is not satisfied (S205: NO), the process proceeds to step S201 and repeats the steps described above.

Next, with reference to FIGS. 8 and 9, an automatic music transcription device according to one embodiment of the present disclosure will be described. FIG. 8 is a block diagram showing the functional configuration of an automatic music transcription device according to an embodiment of the present disclosure.

As shown in FIG. 8, the automatic music transcription device 200 has a model processor 210 and a musical score generator 220 .

The model processing unit 210 includes a learned feature map generation model that outputs a pitch prediction probability from a single tone sound source, and a learned note existence model that outputs a prediction probability that a note exists in a fixed-length section of the feature map from the feature map. Using a probabilistic prediction model, the sound source to be transcribed is input to the learned feature map generation model, the feature map generated by the learned feature map generation model is input to the learned note presence probability prediction model, and the feature map outputs the predicted probability that a note exists in a fixed-length interval of .

Specifically, the model processing unit 210 performs preprocessing such as a short-time Fourier transform on the sound source to be transcribed to obtain a spectrogram of the sound source, and converts the obtained spectrogram to a learned feature map by the learning device 100. A feature map is obtained from each convolutional layer of the learned feature map generation model by inputting it into the generative model. Then, the model processing unit 210 inputs each of the acquired feature maps into the learned note existence probability prediction model by the learning device 100, and sets a fixed-length section starting from each point of the input feature map or having the same length as the default box. It obtains the predicted probability of the presence of a note, and passes the obtained predicted probability of note presence of each feature map to the musical score generation unit 220 . For example, the note presence prediction probability is the prediction value of the probability of each pitch (e.g., “do”, “re”, . It can be determined that the pitch having the probability corresponds to the note corresponding to the temporal length.

The musical score generation unit 220 generates musical score information based on the predicted probability that notes exist. Specifically, the musical score generator 220 post-processes the output of the trained note presence probability prediction model according to NMS (Non-Maximum Suppression) used in SSD. Typically, a large number of predicted note candidates are output from the learned note existence probability prediction model. It is necessary to specify a predicted note from these predicted note candidates, and SSD often uses NMS to narrow down the predicted note candidates.

For example, the musical score generation unit 220 first determines the note with the maximum note presence prediction probability output for each point on the feature map input to the learned note presence probability prediction model as the predicted note at the time. . Then, the musical score generation unit 220 determines a predicted note for each point on the feature map, lists each point, a predicted note, and a data set of the corresponding note presence prediction probability, and selects the notes in the list in descending order of the note presence prediction probability. Sort the dataset. Then, the musical score generation unit 220 applies predetermined extraction conditions to narrow down the predicted note candidates from the list. For example, the musical score generation section 220 may delete data sets whose note existence prediction probabilities are equal to or less than a predetermined threshold value (for example, 0.9) from the list. In addition, the musical score generation unit 220 eliminates duplication of notes that are detected as duplicates. If the dataset is at the top of the list, only the top list may be left. A score generator 220 generates a score based on the data sets in the final list.

FIG. 9 is a flowchart illustrating automatic transcription processing according to one embodiment of the present disclosure. The automatic music transcription processing is realized by the automatic music transcription device 200 or the processor of the automatic music transcription device 200 described above.

As shown in FIG. 9, in step S301, the model processing unit 210 acquires a sound source to be transcribed. For example, the sound source may be a monophonic sound source, or may include multiple types of audio components.

In step S302, the model processing unit 210 preprocesses the acquired sound source. Specifically, the model processing unit 210 performs preprocessing such as short-time Fourier transform on the acquired sound source to acquire a spectrogram of the sound source.

In step S303, the model processing unit 210 inputs the preprocessed sound source to the learned feature map generation model to obtain a feature map, and inputs the obtained feature map to the learned note presence probability prediction model to obtain the input feature. Obtain the predicted probability that there is a fixed-length interval starting at each point in the map or a note of the same length as the default box.

In step S304, the musical score generator 220 determines a predicted note based on the note presence prediction probability obtained for each point on the feature map. Specifically, the musical score generation section 220 determines the note corresponding to the maximum note presence prediction probability among the note presence prediction probabilities acquired for each point as the prediction note for the point.

In step S305, the musical score generator 220 performs post-processing on the predicted notes at each point of the determined feature map. Specifically, the musical score generation unit 220 narrows down the predicted notes for each point of the feature map according to the NMS in SSD. For example, based on the predicted note determined for each point on the feature map, the musical score generator 220 lists each point, the predicted note, and the corresponding note presence prediction probability data set, and lists them in descending order with respect to the note presence prediction probability. The data sets in the list are sorted, and the data sets whose note existence prediction probability is less than a predetermined threshold (for example, 0.9) are deleted from the list, and the prediction notes are the same and the prediction notes overlap If a data set with a degree greater than or equal to a predetermined threshold (for example, 80%) is at the top of the list, only the top list may be left.

In step S306, the musical score generator 220 generates musical scores based on the data sets in the final list.

The learning device 100 and the automatic music transcription device 200 described above each include, for example, a CPU (Central Processing Unit) 101, a GPU (Graphics Processing Unit) 102, a RAM (Random
Access Memory) 103 , communication interface (IF) 104 , hard disk 105 , input device 106 and output device 107 . The CPU 101 and the GPU 102 may be referred to as processors or processing circuits, and execute various processes of the learning device 100 and the automatic music transcription device 200. In particular, the CPU 101 controls execution of various processes in the learning device 100 and the automatic music transcription device 200. The GPU 102 performs various processes for learning and executing machine learning models. The RAM 103 and the hard disk 105 function as memories that store various data and programs in the learning device 100 and the automatic music transcription device 200. In particular, the RAM 103 functions as a working memory that stores work data in the CPU 101 and the GPU 102, and the hard disk 105 , control programs and/or learning data for the CPU 101 and GPU 102 are stored. The communication IF 104 is a communication interface for acquiring learning data from the learning data storage 50 . The input device 106 is various devices (for example, display, speaker, keyboard, touch screen, etc.) for inputting information and data, and the output device 107 displays various information such as the content, progress, and result of processing. Various devices (eg, displays, printers, speakers, etc.). However, the learning device 100 and the automatic music transcription device 200 according to the present disclosure are not limited to the hardware configuration described above, and may have any other appropriate hardware configuration.

In one aspect of the present disclosure,
A single-tone sound source and pitch information are acquired as learning data for a first machine learning model, a sound source to be transcribed and score information are acquired as learning data for a second machine-learning model, and the single-tone sound source and the musical score information are acquired as learning data for a second machine learning model. a learning data acquisition unit that preprocesses the sound source to be transcribed and acquires each spectrogram;
a first model learning unit that receives the spectrogram of the single sound source as learning input data and learns a first machine learning model using the pitch information so as to output a predicted probability of the pitch of the single sound source;
A feature map generated by inputting the spectrogram of the sound source to be transcribed into the learned first machine learning model is input as input data for learning, and notes are present in a fixed-length section of the feature map. a second model learning unit that learns a second machine learning model with the musical score information to output a predicted probability;
A learning device is provided having:

In one example,
The first machine learning model and the second machine learning model may be configured by a convolutional neural network.

In one example,
The second model learning unit may input a plurality of feature maps having different temporal resolutions generated by the first machine learning model to the second machine learning model.

In one example,
The second model learning unit may realize the first machine learning model and the second machine learning model as an SSD (Single Shot Detection).

In one example,
The first model learning unit learns the first machine learning model for each of multiple types of audio components,
The second model learning unit may learn the second machine learning model so as to output a predicted probability that a note exists for each audio component type with respect to a sound source to be transcribed containing multiple types of audio components. good.

In one aspect of the present disclosure,
A first trained machine learning model that outputs a pitch prediction probability from a single sound source, and a second trained machine learning model that outputs a prediction probability that a note exists in a fixed-length section of the feature map from a feature map. inputting the sound source to be transcribed into the first trained machine learning model, and inputting the feature map generated by the first trained machine learning model into the second trained machine learning model. and a model processing unit that outputs a predicted probability that a note exists in a fixed-length section of the feature map;
a musical score generation unit that generates musical score information based on the predicted probability that the note exists;
is provided.

In one example,
The model processing unit may obtain a spectrogram by performing preprocessing on the sound source to be transcribed, and input the spectrogram to the first trained machine learning model.

In one example,
The model processing unit may determine, as the predicted note, the note having the highest prediction probability output from the second trained machine learning model for each point on the feature map.

In one example,
The score generation unit may generate score information based on predicted notes extracted according to NMS (Non-Maximum Suppression).

In one aspect of the present disclosure,
A processor acquires a single sound source and pitch information as learning data for a first machine learning model, acquires a sound source to be transcribed and score information as learning data for a second machine learning model, and acquires the single sound. performing preprocessing on the sound source and the sound source to be transcribed to obtain respective spectrograms;
the processor inputting the spectrogram of the monophonic sound source as learning input data and learning a first machine learning model with the pitch information to output a predicted probability of the pitch of the monophonic sound source;
The processor inputs, as input data for learning, a feature map generated by inputting the spectrogram of the sound source to be transcribed into the first machine learning model that has already been trained. training a second machine learning model with the musical score information to output a predicted probability that a note is present;
There is provided a learning method comprising:

In one aspect of the present disclosure,
a processor inputting a sound source to be transcribed into a first trained machine learning model that outputs a predicted probability of pitch from a monophonic sound source;
The processor transfers the feature map generated by the first learned machine learning model to a second learned machine learning model that outputs a predicted probability that a note exists in a fixed-length section of the feature map from the feature map. a step of entering;
the processor generating score information based on the predicted probability of the note being present output from the second trained machine learning model;
There is provided an automatic music transcription method comprising:

In one aspect of the present disclosure,
A single-tone sound source and pitch information are acquired as learning data for a first machine learning model, a sound source to be transcribed and score information are acquired as learning data for a second machine-learning model, and the single-tone sound source and the musical score information are acquired as learning data for a second machine learning model. performing preprocessing on the sound sources to be transcribed and obtaining respective spectrograms;
inputting the spectrogram of the monophonic sound source as training input data and learning a first machine learning model with the pitch information so as to output a predicted probability of the pitch of the monophonic sound source;
A feature map generated by inputting the spectrogram of the sound source to be transcribed into the learned first machine learning model is input as input data for learning, and notes are present in a fixed-length section of the feature map. training a second machine learning model with the score information to output predicted probabilities;
A program is provided which causes a processor to execute

In one aspect of the present disclosure,
inputting a sound source to be transcribed into a first trained machine learning model that outputs predicted probabilities of pitches from monophonic sound sources;
inputting the feature map generated by the first trained machine learning model into a second trained machine learning model that outputs from the feature map a predicted probability that a note exists in a fixed-length interval of the feature map; ,
generating score information based on the predicted probability that the note exists output from the second trained machine learning model;
A program is provided which causes a processor to execute

In one aspect of the present disclosure,
A computer readable storage medium storing the above program is provided.

Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the specific embodiments described above, and various modifications can be made within the scope of the gist of the present disclosure described in the claims.・Changes are possible.

50 learning data storage 100 learning device 200 automatic transcription device

In order to solve the above problems, one aspect of the present disclosure is a first machine learning model configured by a convolutional neural network, which includes a first spectrogram generated from a single sound source and corresponding pitch information. By learning paired teacher data, each feature map showing the predicted probability of the corresponding pitch generated from a plurality of convolution layers with different temporal resolutions according to the input of the first spectrogram is output. a first model learning unit for learning a first machine learning model; Output according to the input to the first machine learning model of the third spectrogram generated from the sound source to be transcribed by learning teacher data that pairs each feature map and score information A second machine learning model that outputs the note existence prediction probability that notes of the same length as the fixed-length section or default box starting from each point on each feature map according to the input of each feature map and a second model learning unit for learning the learning device.

Claims (13)

A first feature map indicating the predicted probability of the corresponding pitch according to the input of the spectrogram by learning teacher data that pairs a first spectrogram generated from a single sound source and corresponding pitch information. a first model learning unit that learns a first machine learning model that outputs
The first machine learning of a second spectrogram generated from a sound source to be transcribed by learning teacher data pairing the first feature map output by the first model learning unit and musical score information. Second model learning for learning a second machine learning model that outputs information for generating a musical score according to the input of the first feature map indicating the predicted probability of the pitch that is output according to the input to the model. Department and
A learning device having 2. The learning device according to claim 1, wherein said first machine learning model and said second machine learning model are configured by a convolutional neural network. 3. The learning according to claim 2, wherein said second model learning unit inputs a plurality of said first feature maps having different temporal resolutions generated by said first machine learning model to said second machine learning model. Device. 4. The learning device according to claim 1, wherein said second model learning unit realizes said first machine learning model and said second machine learning model as SSD (Single Shot Detection). The first model learning unit learns the first machine learning model for each of multiple types of audio components,
5. The learning device according to any one of claims 1 to 4, wherein the second model learning unit learns the second machine learning model for a sound source to be transcribed including multiple types of audio components. A first trained machine learning model that outputs a first feature map indicating predicted probabilities of pitches from a single tone sound source, and a second trained machine learning model that outputs information for generating a musical score from the first feature map. input the sound source to be transcribed into the first trained machine learning model, and transfer the first feature map output by the first trained machine learning model to the second trained machine learning model a model processing unit that inputs to and outputs information for generating a musical score;
automatic transcription device. 7. The automatic music transcription apparatus according to claim 6, wherein said model processing unit acquires a spectrogram by performing preprocessing on said sound source to be transcribed, and inputs said spectrogram to said first trained machine learning model. . 8. The model processing unit according to claim 6, wherein for each point on the first feature map, the note having the highest prediction probability output from the second trained machine learning model is determined as the predicted note. Automatic transcription device. a musical score generating unit that generates musical score information based on the predicted probability that the note exists;
9. The automatic music transcription apparatus according to claim 8, wherein said score generation unit generates score information based on predicted notes extracted according to NMS (Non-Maximum Suppression). the processor
A first feature map indicating the predicted probability of the corresponding pitch according to the input of the spectrogram by learning teacher data that pairs a first spectrogram generated from a single sound source and corresponding pitch information. training a first machine learning model that outputs
A second spectrogram generated from a sound source to be transcribed is output according to an input to the first machine learning model by learning teacher data paired with the first feature map and musical score information. learning a second machine learning model that outputs information for generating a musical score in response to the input of a first feature map indicating the predicted probability of pitches that
How to learn to do. the processor
A first feature map indicating the predicted probability of the corresponding pitch according to the input of the spectrogram by learning teacher data that pairs a first spectrogram generated from a single sound source and corresponding pitch information. training a first machine learning model that outputs
A second spectrogram generated from a sound source to be transcribed is output according to an input to the first machine learning model by learning teacher data paired with the first feature map and musical score information. learning a second machine learning model that outputs information for generating a musical score in response to the input of a first feature map indicating the predicted probability of pitches that
An automatic transcription method that performs A first feature map indicating the predicted probability of the corresponding pitch according to the input of the spectrogram by learning teacher data that pairs a first spectrogram generated from a single sound source and corresponding pitch information. training a first machine learning model that outputs
A second spectrogram generated from a sound source to be transcribed is output according to an input to the first machine learning model by learning teacher data paired with the first feature map and musical score information. learning a second machine learning model that outputs information for generating a musical score in response to the input of a first feature map indicating the predicted probability of pitches that
A program that causes the processor to execute inputting a sound source to be transcribed into a first trained machine learning model that outputs predicted probabilities of pitches from monophonic sound sources;
Inputting a feature map indicating the predicted probability of the pitch generated by the first trained machine learning model to a second trained machine learning model that outputs information for generating a musical score in response to an input. a step;
A program that causes the processor to execute

Images