JP2020003536A - Learning device, automatic music transcription device, learning method, automatic music transcription method and program - Google Patents

Abstract

To provide acoustic processing technology for automatically generating a musical score from audio data where a pitch and a section of each sound are not clear.SOLUTION: A learning device 100 comprises: a learning data acquisition section for acquiring a single tone sound source and pitch information as learning data of a first machine learning model, acquiring a sound source of a transcription object and musical score information as learning data of a second machine learning model, performing preprocessing on the single tone sound source and the sound source of the transcription object, and acquiring respective spectrograms; a first model learning section for performing learning by pitch information so that the spectrogram of the single tone sound source is input as learning input data and prediction probability of the pitch of the single tone sound source is output; and a second model learning section for performing learning by musical score information so that a feature map generated by inputting the spectrogram of the sound source of the transcription object to the learnt first machine learning model is input as learning input data and prediction probability that notes exist in the section of fixed length in the feature map is output.SELECTED DRAWING: Figure 2

Description

  The present disclosure relates to sound processing technology.

  2. Description of the Related Art An automatic music transcription technique for automatically generating a musical score from audio data has been conventionally known. For example, JP-A-2007-033479 describes a technique for automatically transcribing a musical score from an acoustic signal played by a single musical instrument even when a plurality of sounds are played at the same time.

JP 2007-033479

  However, in the conventional automatic transcription, the musical score is played or sung accurately, and the pitch and the section of each sound are clear audio data, it is possible to perform relatively high-precision transcription. In the case of audio data whose pitch and section are not clear, automatic transcription as expected is difficult.

  In view of the above problems, an object of the present disclosure is to provide a sound processing technique for automatically and efficiently generating a musical score from various audio data.

  In order to solve the above problem, according to one embodiment of the present disclosure, a single-tone sound source and pitch information are acquired as learning data of a first machine learning model, and a sound source to be transcribed and score information are acquired in a second machine learning mode. A learning data acquisition unit that acquires as model learning data, performs preprocessing on the single-tone sound source and the sound source to be transcribed, and acquires a spectrogram of each, and a learning input of the single-tone sound source spectrogram. A first model learning unit that learns a first machine learning model based on the pitch information so as to output a prediction probability of a pitch of the single-tone sound source, and a spectrogram of a sound source to be transcribed has been learned. A feature map generated by inputting to the first machine learning model is input as learning input data, and a note exists in a fixed length section of the feature map. A second model learning unit that learns a second machine learning model by the score information to output the prediction probability, regarding learning device having a.

  According to the present disclosure, it is possible to provide a sound processing technique for automatically generating a musical score from audio data in which the pitch or section of each sound is not clear.

1 is a schematic diagram illustrating an automatic transcription apparatus having a learned machine learning model according to an embodiment of the present disclosure. 1 is a block diagram illustrating a functional configuration of a learning device according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram illustrating a configuration of a feature map generation model according to an embodiment of the present disclosure. 1 is a schematic diagram illustrating a configuration of a note existence probability prediction model according to an embodiment of the present disclosure. FIG. 6 is a conceptual diagram illustrating a relationship between a feature map and a default box according to an embodiment of the present disclosure. 6 is a flowchart illustrating a learning process of a feature map generation model according to an embodiment of the present disclosure. 11 is a flowchart illustrating a learning process of a note existence probability prediction model according to an embodiment of the present disclosure. 1 is a block diagram illustrating a functional configuration of an automatic transcription apparatus according to an embodiment of the present disclosure. 5 is a flowchart illustrating an automatic music transcription process according to an embodiment of the present disclosure. 1 is a block diagram illustrating a hardware configuration of a learning device and an automatic transcription device according to an embodiment of the present disclosure.

  The following embodiment discloses an automatic transcription apparatus that generates musical score information from a sound source (audio data that is sound waveform data) using a machine learning model.

  In the conventional automatic transcription technology, attention is paid to pitch prediction, and prediction of an onset indicating a break in a note and an offset is one of the problems in automatic transcription. The automatic transcription apparatus according to the present disclosure predicts an onset and an offset in a sound source by using an SSD (Single Shot Detection) which is one of machine learning models.

  SSD is a technique for detecting an object in an input image using one neural network. That is, the input to the neural network is an image, and the output is the predicted probability of the center coordinates, height, width, and object type of a plurality of rectangular regions (called default boxes in SSDs). The default box is prepared as a preset number of candidates according to the size of the input image, most of the default boxes are removed from the candidates by post-processing (NMS: Non-Maximum Suppression, etc.), and the remaining default boxes are used as detection results. That is.

  The input to the neural network in the automatic transcription apparatus according to the present disclosure is waveform data or a spectrogram of a musical sound to be transcribed, the output of which is the onset, offset and pitch of the musical sound. Onset and offset (that is, the shape or length of a musical sound) are specified in accordance with the pitch and the pitch, and the pitch is specified in accordance with the type of object in the SSD.

  To outline the embodiment described later, the automatic transcription apparatus uses two learned machine learning models (such as a convolutional neural network), one of which outputs a pitch prediction probability from a single sound source, and The model outputs a prediction probability that a note exists in a fixed length section of the feature map from the feature map. The automatic transcription apparatus inputs a sound source to be transcribed to the former learned machine learning model (feature map generation model), and converts each feature map generated from the convolutional layer of the learned feature map generation model to the latter learned machine learning model. Predicted presence of a note of each pitch in a fixed-length section or default box output from the learned note existence probability prediction model for each point of each feature map, input to a learning model (note existence probability prediction model) Music score information is generated based on the probability.

  The feature maps generated by the trained feature map generation model have different temporal resolutions as a result of convolution, and fixed-length sections or default boxes have different temporal lengths. For this reason, by detecting a note having the same length as a fixed-length section for each feature map by using the note existence probability prediction model, it becomes possible to specify onsets and offsets of notes of different lengths.

  First, an automatic music transcription device according to an embodiment of the present disclosure will be described with reference to FIG. FIG. 1 is a schematic diagram illustrating an automatic transcription apparatus having a learned machine learning model according to an embodiment of the present disclosure.

  As shown in FIG. 1, the automatic transcription apparatus 200 according to an embodiment of the present disclosure has two types of learned models realized as any type of neural network such as a convolutional neural network without limitation. Then, the musical score information is generated from the sound source to be transcribed using the machine learning model learned by the learning device 100 using the learning data storage 50.

  Next, a learning device according to an embodiment of the present disclosure will be described with reference to FIGS. The learning device 100 uses the learning data in the learning data storage 50 to learn a feature map generation model and a note existence probability prediction model used in the automatic transcription apparatus 200. FIG. 2 is a block diagram illustrating a functional configuration of the learning device according to the embodiment of the present disclosure.

  As shown in FIG. 2, the learning device 100 includes a learning data acquisition unit 110, a first model learning unit 120, and a second model learning unit 130.

  The learning data acquisition unit 110 acquires a single sound source and pitch information as learning data of a feature map generation model, and acquires a sound source to be transcribed and musical score information as learning data of a note existence probability prediction model, Preprocessing is performed on a single sound source and a sound source to be transcribed, and respective spectrograms are obtained.

  More specifically, the learning data acquisition unit 110 reads, from the learning data storage 50, a single sound or single note sound source (for example, 12 types of sound sources from “do” to “shi”) for learning the feature map generation model. ) And a pair of pitch information (such as pitches from “do” to “shi”), and pre-processes (eg, short-time Fourier transform, etc.) ) To generate a learning data set of the spectrogram of each single sound source and the pitch information.

  In addition, the learning data acquisition unit 110 obtains, from the learning data storage 50, waveform data of a single melody sound source (such as a singing sound source) and musical score information (such as a time-series change in pitch) for learning a note existence probability prediction model. By performing preprocessing (for example, short-time Fourier transform) on the acquired waveform data of the monophonic sound source, thereby generating a learning data set of the spectrogram of the monophonic sound source and the musical score information. . Here, the musical score information may conform to, for example, a MIDI (Musical Instrument Digital Interface) standard.

  Typically, a spectrogram represents the intensity of a signal component on a time axis and a frequency axis, and is generated by performing a short-time Fourier transform on waveform data. Various parameters need to be set in the short-time Fourier transform. For example, an FFT window width: 1024, a sampling frequency: 16 kHz, an overlap width: 768, a window function: a Hanning window, and a filter bank: a mel filter bank ( (128 bands) or the like, and a short-time Fourier transform may be executed. After conversion into a spectrogram, a fixed number of samples (for example, 1024 samples) may be extracted in the time axis direction. Further, the spectrogram according to the present embodiment may be one in which the frequency axis is logarithmically transformed so as to make the low frequency component fine.

  The first model learning unit 120 inputs a spectrogram of a single sound source as learning input data, and learns a feature map generation model based on pitch information so as to output a prediction probability of a pitch of a single sound source.

  For example, as shown in FIG. 3, the feature map generation model is configured by a convolutional neural network including a plurality of convolutional layers, and is realized as an SSD that converts an input single sound source spectrogram into a pitch prediction probability. . Here, the pitch may be expressed as a discrete value instead of a continuous value, and may be expressed as a one-hot vector. When a noise source such as a percussion instrument is to be learned, a single sound or a single note sound of the noise source may be included in the data set. In that case, a class that expresses a noise may be set as the pitch class, and may be used as the teacher label.

  The first model learning unit 120 inputs the spectrogram of the single sound source of the learning input data to the feature map generation model, and reduces an error between the output from the feature map generation model and the pitch information of the learning output data. Then, the parameters of the feature map generation model are updated by back propagation. Here, as the loss function indicating the error, the cross entropy between the output of the feature map generation model and the pitch of the output data for learning may be used without limitation.

  For example, when predetermined learning end conditions are satisfied, such as when the update processing is completed for a predetermined number of learning data, the error converges below a predetermined threshold, and the error improvement converges below a predetermined threshold. , The first model learning unit 120 sets the updated feature map generation model as a learned machine learning model.

  The second model learning unit 130 inputs, as learning input data, a feature map generated by inputting a spectrogram of a sound source to be transcribed to a learned feature map generation model, and enters a note in a fixed-length section of the feature map. A musical note existence probability prediction model is learned from the musical score information so as to output a prediction probability of existence.

  For example, as shown in FIG. 4, the note existence probability prediction model is configured by a convolutional neural network including a plurality of convolutional layers, and is generated by inputting a spectrumgram of a monophonic sound source to a learned feature map generation model. The feature map is implemented as an SSD that converts a feature map into a prediction probability that a note having the same length as a fixed-length section starting from each point of the feature map exists. For example, when transcribed with 12 notes from C to C, each point on the feature map has a predicted probability that there are 13 pitches or pitch classes from C to C and rests (silences). .

  As described above, the learned feature map generation model includes a plurality of convolutional layers, and a feature map of a spectrogram of a monophonic sound source is generated from each of the convolutional layers. The generated feature map is a feature map having a different temporal resolution according to the level of the convolutional layer as shown in FIG. Typically, as shown in FIG. 5, a convolutional layer relatively close to the input layer produces a feature map with a relatively high temporal resolution (32 Hz in the example shown) and a In the convolutional layer which is close to the target, a feature map having a relatively low temporal resolution (16 Hz in the illustrated example) is generated. When a fixed-length section or a default box is set as shown in the figure, a section in a feature map having a relatively high temporal resolution occupies a shorter time than a section in a feature map having a relatively low temporal resolution. For this reason, it is possible to derive the predicted existence probabilities of notes having different time lengths, and to specify the time lengths of the notes.

  The second model learning unit 130 inputs the spectrogram of the sound source of the learning input data to the learned feature map generation model, inputs each feature map generated by the learned feature map generation model to the note existence probability prediction model, The parameters of the note existence probability prediction model are updated by back propagation so that the error between the output from the note existence probability prediction model and the musical score information of the output data for learning is reduced.

  Here, as a loss function indicating an error, a weighted sum of a timing error and a reliability error calculated from an output of the note existence probability prediction model and a time-series change in pitch may be used without limitation. The time-series change of the pitch is expressed by collecting a plurality of sets of the start timing, the end timing, and the pitch of the music, and is derived from the musical score information. The set may be referred to as pronunciation. For example, the time-series change in pitch is pronunciation # 1: “0:00 to 0:02, A (La) 3”, pronunciation # 2: “0:03 to”. 0:05, B (S) 3 ", pronunciation # 3:" 0: 05-0: 08, C (D) 4 ", etc. The default box shown in FIG. 5 expresses one pronunciation and has a plurality of channels. The first sample of each channel of the default box has a predicted onset of the pronunciation, a predicted offset, and a predicted probability of the pitch class of the default box, respectively. That is, there are a total of 2+ (the number of pitch classes) channels.

  The second model learning unit 130 searches for a default box that minimizes the sum of the onset and the offset for each pronunciation, and obtains a timing error and a reliability error for the detected default box and the pronunciation. Here, the timing error may be a sum of a deviation of the start timing in consideration of the predicted onset and a deviation of the end timing in consideration of the predicted offset. However, a relative value based on the length of the default box may be used as the expression of the difference. Further, the reliability error may be a cross entropy calculated from the pitch of the pronunciation and the predicted pitch. Note that a class representing silence may also be prepared as a teacher label, and in this case, a section having no sound can be predicted.

  The second model learning unit 130 may reduce the number of default boxes set for each point of each feature map in accordance with NMS (Non-Maximum Suppression), and use the remaining default boxes as predicted sounds. Specifically, the second model learning unit 130 first obtains a note presence prediction probability for each pitch class for each default box. Thereafter, the second model learning unit 130 may delete the default box whose prediction probability is equal to or less than a predetermined threshold (for example, 0.9). The second model learning unit sets a threshold value for the intersection set / union set among the remaining default boxes, deletes one of the default boxes equal to or larger than the threshold value, and eliminates a duplicated default box. The second model learning unit 130 sets the finally left default box as the predicted pronunciation.

  For example, when predetermined learning end conditions are satisfied, such as when the update processing is completed for a predetermined number of learning data, the error converges below a predetermined threshold, and the error improvement converges below a predetermined threshold. , The second model learning unit 130 sets the updated note existence probability prediction model as a learned model.

  In one embodiment, the first model learning unit 120 learns a feature map generation model for each of a plurality of types of audio components, and the second model learning unit 130 generates a transcription target sound source including a plurality of types of audio components. Note that a note existence probability prediction model may be learned so as to output a prediction probability that a note exists for each audio component type.

  For example, the feature map generation model and the note existence probability prediction model may be applied to music including monophonic vocals and accompaniment. In this case, the vocal feature map generation model and the accompaniment feature map generation model are composed of vocal learning data composed of a pair of a vocal single-tone sound source and pitch information, and a vocal single-tone sound source and pitch information. Using the learning data for accompaniment composed of pairs, learning is performed in the same manner as in the learning processing described above. On the other hand, the note existence probability prediction model for vocals and the note existence probability prediction model for accompaniment use a sound source for learning and score information to generate a feature map generation model for vocal and a feature map for learned accompaniment. The feature map generated by inputting the information to the model is input, and learning is performed in the same manner as in the above-described learning process.

  Alternatively, the feature map generation model and the note existence probability prediction model may be applied to a musical piece including a plurality of parts such as each musical instrument. This is the same as the above-described learning process for music including vocal and accompaniment, but in this case, the output of the note existence probability prediction model is the prediction probability that the specific note of the specific part exists in the fixed length section of the feature map. You may. For example, a prediction probability of the existence of a specific note of a specific part, such as “pitch of male A3” or “pitch of female A3” may be output.

  Alternatively, the present disclosure may be applied to songs having a time signature. In this case, the output of the note existence probability prediction model may be related to the onset and offset of the time signature, and for example, the prediction probability that the default box is one beat may be output.

  FIG. 6 is a flowchart illustrating a learning process of a feature map generation model according to an embodiment of the present disclosure. The learning process is realized by the learning device 100 or the processor of the learning device 100 described above.

  As shown in FIG. 6, in step S101, the learning data acquisition unit 110 acquires a pair of a single sound source and pitch information from the learning data storage 50. For example, there are 13 pitches of 12 pitches from “do” to “shi” and no sound, and a single tone sound source corresponding to the 13 pitches may be acquired.

  In step S102, the learning data acquisition unit 110 pre-processes the acquired single sound source. Specifically, the learning data acquisition unit 110 performs preprocessing (for example, short-time Fourier transform) on the waveform data of the single sound source, and acquires a spectrogram of the single sound source.

  In step S103, the first model learning unit 120 learns a feature map generation model based on a pair of a preprocessed single sound source and pitch information. For example, the feature map generation model is formed by a convolutional neural network, and converts an input sound source into a pitch prediction probability. Specifically, the first model learning unit 120 inputs the spectrogram of the single sound source to the feature map generation model, and performs the backpropagation so as to reduce the error between the output of the feature map generation model and the pitch information. Update the parameters of the map generation model.

  In step S104, the first model learning unit 120 determines whether a learning end condition has been satisfied. The predetermined learning end condition may be, for example, that the update processing has been completed for a predetermined number of learning data, the error has converged to a predetermined threshold or less, or the error improvement has converged to a predetermined threshold or less. Good. When the predetermined learning end condition is satisfied (S104: YES), the first model learning unit 120 may set the updated feature map generation model as a learned model. On the other hand, when the predetermined learning end condition is not satisfied (S104: NO), the process proceeds to step S101, and the above-described steps are repeated.

  FIG. 7 is a flowchart illustrating a learning process of a note existence probability prediction model according to an embodiment of the present disclosure. The learning process is realized by the learning device 100 or the processor of the learning device 100 described above.

  As shown in FIG. 7, in step S201, the learning data acquisition unit 110 acquires a pair of a monophonic sound source and musical score information from the learning data storage 50. For example, the monophonic sound source may be waveform data of a singing sound source, and the score information indicates the score of the monophonic sound source.

  In step S202, the learning data acquisition unit 110 pre-processes the acquired monophonic sound source. More specifically, the learning data acquisition unit 110 performs preprocessing (for example, short-time Fourier transform) on the waveform data of the monophonic sound source, and acquires a spectrogram of the monophonic sound source.

  In step S203, the second model learning unit 130 inputs the preprocessed monophonic sound source to the learned feature map generation model, and acquires a feature map generated by the learned feature map generation model. Specifically, the second model learning unit 130 acquires a feature map generated from each convolutional layer of the learned feature map generation model. The generated feature map is a feature map having a different temporal resolution depending on the degree of convolution of each convolutional layer.

  In step S204, the second model learning unit 130 learns a musical note existence probability prediction model based on a pair of the acquired feature map and musical score information. For example, the note existence probability prediction model is configured by a convolutional neural network, and converts an input feature map into a note existence prediction probability in which a note exists in a fixed length section of the feature map. Specifically, the second model learning unit 130 inputs each feature map to the note existence probability prediction model, and performs note propagation by back propagation so that an error between the output of the note existence probability prediction model and the score information becomes small. Update the parameters of the existence probability prediction model.

  In step S205, the second model learning unit 130 determines whether the learning end condition has been satisfied. The predetermined learning end condition may be, for example, that the update processing has been completed for a predetermined number of learning data, the error has converged to a predetermined threshold or less, or the error improvement has converged to a predetermined threshold or less. Good. If the predetermined learning end condition is satisfied (S205: YES), the second model learning unit 130 may set the updated note existence probability prediction model as a learned model. On the other hand, if the predetermined learning end condition is not satisfied (S205: NO), the process proceeds to step S201, and the above-described steps are repeated.

  Next, an automatic music transcription device according to an embodiment of the present disclosure will be described with reference to FIGS. FIG. 8 is a block diagram illustrating a functional configuration of the automatic transcription apparatus according to an embodiment of the present disclosure.

  As shown in FIG. 8, the automatic transcription apparatus 200 includes a model processing unit 210 and a musical score generation unit 220.

  The model processing unit 210 includes a learned feature map generation model that outputs a predicted probability of a pitch from a single sound source, and a learned note existence that outputs a predicted probability that a note exists in a fixed length section of the feature map from the feature map. Using the probability prediction model, the sound source to be transcribed is input to the learned feature map generation model, and the feature map generated by the learned feature map generation model is input to the learned note existence probability prediction model. The prediction probability that a note exists in the fixed-length section of is output.

  Specifically, the model processing unit 210 performs a preprocessing such as a short-time Fourier transform on the sound source to be transcribed, acquires a spectrogram of the sound source, and converts the acquired spectrogram into a learned feature map by the learning device 100. The feature map is input to the generation model and the feature map from each convolutional layer of the learned feature map generation model is obtained. Then, the model processing unit 210 inputs each of the acquired feature maps to the learned note existence probability prediction model by the learning device 100, and has a fixed length section starting from each point of the input feature map or the same length as the default box. The prediction probability that a note exists is obtained, and the obtained note presence prediction probability of each of the feature maps is passed to the score generation unit 220. For example, the note presence prediction probability is a predicted value of the probability of each pitch (for example, “do”, “re”,. It can be determined that the pitch having the probability corresponds to the note corresponding to the time length.

  The musical score generation unit 220 generates musical score information based on a predicted probability that a note exists. Specifically, the score generation unit 220 post-processes the output of the learned note existence probability prediction model according to the NMS (Non-Maximum Suppression) used for the SSD. Typically, many predicted note candidates are output from the learned note existence probability prediction model. It is necessary to specify a predicted note from these predicted note candidates, and the SSD often uses an NMS to narrow down the predicted note candidates.

  For example, first, the musical score generation unit 220 sets a note having the largest of the note existence prediction probabilities output for each point on the feature map input to the learned note existence probability prediction model as a predicted note at the time. . Then, the score generation unit 220 determines a predicted note for each point on the feature map, lists a data set of each point, the predicted note and the corresponding note existence prediction probability, and sorts the note existence prediction probability in the list in descending order with respect to the note existence prediction probability. Sort the dataset. Then, the score generating unit 220 applies a predetermined extraction condition and narrows down predicted note candidates from the list. For example, the score generating unit 220 may delete from the list a data set whose note existence prediction probability is equal to or less than a predetermined threshold value (for example, 0.9). In addition, the score generating unit 220 eliminates duplication of notes detected by duplication, so that the predicted notes are the same and the degree of duplication of the predicted notes is equal to or more than a predetermined threshold (for example, 80%). If the dataset is at the top of the list, only the top list may be left. The score generation unit 220 generates a score based on the data set in the final list.

  FIG. 9 is a flowchart illustrating an automatic music transcription process according to an embodiment of the present disclosure. The automatic music transcription processing is realized by the above-described automatic music transcription apparatus 200 or the processor of the automatic music transcription apparatus 200.

  As shown in FIG. 9, in step S301, the model processing unit 210 acquires a sound source to be transcribed. For example, the sound source may be a monophonic sound source or may include a plurality of types of audio components.

  In step S302, the model processing unit 210 pre-processes the acquired sound source. Specifically, the model processing unit 210 performs preprocessing such as short-time Fourier transform on the acquired sound source, and acquires a spectrogram of the sound source.

  In step S303, the model processing unit 210 inputs the preprocessed sound source to the learned feature map generation model to obtain a feature map, and inputs the obtained feature map to the learned note existence probability prediction model to input the feature. A prediction probability that a note having the same length as a fixed-length section or a default box starting from each point of the map is obtained.

  In step S304, the score generation unit 220 determines a predicted note based on the note presence prediction probability obtained for each point on the feature map. Specifically, the score generation unit 220 determines the note corresponding to the maximum note existence prediction probability among the note existence prediction probabilities acquired for each point as the predicted note for the point.

  In step S305, the score generating unit 220 performs post-processing on the predicted note at each point of the determined feature map. Specifically, the score generating unit 220 narrows down predicted notes at each point of the feature map according to the NMS in the SSD. For example, based on the predicted notes determined for each point on the feature map, the score generation unit 220 lists a data set of each point, the predicted note, and the corresponding note existence prediction probability, and lists the note existence prediction probability in descending order. Are deleted from the list, and a data set whose predicted note existence probability is equal to or less than a predetermined threshold value (for example, 0.9) is deleted from the list, and the predicted note is the same and the predicted note overlaps. When a data set whose degree is equal to or higher than a predetermined threshold (for example, 80%) is at the top of the list, only the top list may be left.

  In step S306, the score generation unit 220 generates a score based on the data set in the final list.

  For example, as shown in FIG. 10, the learning device 100 and the automatic transcription device 200 described above each include a CPU (Central Processing Unit) 101, a GPU (Graphics Processing Unit) 102, a RAM (Random Access Memory) 103, and a communication interface ( IF) 104, a hard disk 105, an input device 106, and an output device 107. The CPU 101 and the GPU 102 may be referred to as a processor or a processing circuit, and execute various processes of the learning device 100 and the automatic transcription device 200. In particular, the CPU 101 controls execution of various processes in the learning device 100 and the automatic transcription device 200. The GPU 102 executes various processes for learning and executing the machine learning model. The RAM 103 and the hard disk 105 function as a memory for storing various data and programs in the learning device 100 and the automatic transcription device 200. In particular, the RAM 103 functions as a working memory for storing work data in the CPU 101 and the GPU 102. , A control program for the CPU 101 and the GPU 102 and / or learning data. The communication IF 104 is a communication interface for acquiring learning data from the learning data storage 50. The input device 106 is various devices (for example, a display, a speaker, a keyboard, a touch screen, etc.) for inputting information and data, and the output device 107 displays various information such as the content, progress, and results of processing. Various devices (for example, a display, a printer, a speaker, and the like). However, the learning device 100 and the automatic transcription device 200 according to the present disclosure are not limited to the above-described hardware configuration, and may have any other appropriate hardware configuration.

In one aspect of the present disclosure,
A single sound source and pitch information are obtained as learning data of a first machine learning model, a sound source to be transcribed and musical score information are obtained as learning data of a second machine learning model, and the single sound source and the pitch information are obtained. A learning data acquisition unit that performs preprocessing on a sound source to be transcribed and acquires each spectrogram;
A first model learning unit that inputs a spectrogram of the single sound source as learning input data and learns a first machine learning model based on the pitch information so as to output a prediction probability of a pitch of the single sound source;
A feature map generated by inputting the spectrogram of the sound source to be transcribed to the learned first machine learning model is input as learning input data, and a note exists in a fixed length section of the feature map. A second model learning unit that learns a second machine learning model based on the score information so as to output a prediction probability;
Is provided.

In one embodiment,
The first machine learning model and the second machine learning model may be configured by a convolutional neural network.

In one embodiment,
The second model learning unit may input a plurality of feature maps having different time resolutions generated by the first machine learning model to the second machine learning model.

In one embodiment,
The second model learning unit may realize the first machine learning model and the second machine learning model as an SSD (Single Shot Detection).

In one embodiment,
The first model learning unit learns the first machine learning model for each of a plurality of types of audio components,
The second model learning unit may also learn the second machine learning model so as to output a prediction probability that a note exists for each audio component type for a sound source to be transcribed including a plurality of types of audio components. Good.

In one aspect of the present disclosure,
A first learned machine learning model that outputs a prediction probability of a pitch from a single sound source, and a second learned machine learning model that outputs a prediction probability that a note exists in a fixed length section of the feature map from a feature map The sound source to be transcribed is input to the first learned machine learning model, and the feature map generated by the first learned machine learning model is input to the second learned machine learning model. A model processing unit that outputs a prediction probability that a note exists in a fixed-length section of the feature map;
A score generation unit that generates score information based on the predicted probability that the note exists;
Is provided.

In one embodiment,
The model processing unit may obtain a spectrogram by performing preprocessing on the sound source to be transcribed, and may input the spectrogram to the first learned machine learning model.

In one embodiment,
The model processing unit may determine, as a predicted note, a note having the maximum prediction probability output from the second learned machine learning model for each point on the feature map.

In one embodiment,
The musical score generating unit may generate musical score information based on predicted notes extracted according to NMS (Non-Maximum Suppression).

In one aspect of the present disclosure,
A processor that obtains a single sound source and pitch information as learning data of a first machine learning model, obtains a sound source to be transcribed and music information as learning data of a second machine learning model, Performing preprocessing on the sound source and the sound source to be transcribed, and obtaining respective spectrograms;
The processor inputs a spectrogram of the monophonic sound source as learning input data, and learns a first machine learning model by the pitch information so as to output a prediction probability of a pitch of the monophonic sound source,
The processor inputs, as learning input data, a feature map generated by inputting a spectrogram of the sound source to be transcribed to the learned first machine learning model, and outputs the feature map to a fixed-length section of the feature map. Learning a second machine learning model based on the score information to output a predicted probability that a note is present;
Is provided.

In one aspect of the present disclosure,
A processor inputting a sound source to be transcribed to a first learned machine learning model that outputs a predicted probability of a pitch from a single sound source;
The processor outputs a feature map generated by the first learned machine learning model to a second learned machine learning model that outputs a prediction probability that a note exists in a fixed length section of the feature map from the feature map. Inputting,
The processor generating score information based on a predicted probability that the note is output from the second learned machine learning model,
Is provided.

In one aspect of the present disclosure,
A single sound source and pitch information are obtained as learning data of a first machine learning model, a sound source to be transcribed and musical score information are obtained as learning data of a second machine learning model, and the single sound source and the pitch information are obtained. Performing preprocessing on the sound source to be transcribed and obtaining spectrograms of each;
Inputting a spectrogram of the single-tone sound source as learning input data, and learning a first machine learning model by the pitch information so as to output a prediction probability of a pitch of the single-tone sound source;
A feature map generated by inputting the spectrogram of the sound source to be transcribed to the learned first machine learning model is input as learning input data, and a note exists in a fixed length section of the feature map. Learning a second machine learning model with the musical score information to output a prediction probability;
Is provided.

In one aspect of the present disclosure,
Inputting a sound source to be transcribed into a first learned machine learning model that outputs a prediction probability of a pitch from a single sound source;
Inputting the feature map generated by the first learned machine learning model to a second learned machine learning model that outputs a prediction probability that a note exists in a fixed length section of the feature map from the feature map; ,
Generating musical score information based on a predicted probability that the note is output from the second learned machine learning model;
Is provided.

In one aspect of the present disclosure,
A computer-readable storage medium storing the above-described program is provided.

  Although the embodiments of the present disclosure have been described above in detail, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made within the scope of the present disclosure described in the claims.・ Change is possible.

50 learning data storage 100 learning device 200 automatic transcription device

Claims (13)

A single sound source and pitch information are obtained as learning data of a first machine learning model, a sound source to be transcribed and musical score information are obtained as learning data of a second machine learning model, and the single sound source and the pitch information are obtained. A learning data acquisition unit that performs preprocessing on a sound source to be transcribed and acquires each spectrogram;
A first model learning unit that inputs a spectrogram of the single sound source as learning input data and learns a first machine learning model based on the pitch information so as to output a prediction probability of a pitch of the single sound source;
A feature map generated by inputting the spectrogram of the sound source to be transcribed to the learned first machine learning model is input as learning input data, and a note exists in a fixed length section of the feature map. A second model learning unit that learns a second machine learning model based on the score information so as to output a prediction probability;
A learning device having:   The learning device according to claim 1, wherein the first machine learning model and the second machine learning model are configured by a convolutional neural network.   The learning device according to claim 2, wherein the second model learning unit inputs a plurality of feature maps having different temporal resolutions generated by the first machine learning model to the second machine learning model.   4. The learning device according to claim 1, wherein the second model learning unit implements the first machine learning model and the second machine learning model as a single shot detection (SSD). 5. The first model learning unit learns the first machine learning model for each of a plurality of types of audio components,
The second model learning unit learns the second machine learning model so as to output a prediction probability that a note exists for each audio component type for a transcription target sound source including a plurality of types of audio components. The learning device according to any one of Items 1 to 4. A first learned machine learning model that outputs a prediction probability of a pitch from a single sound source, and a second learned machine learning model that outputs a prediction probability that a note exists in a fixed length section of the feature map from a feature map The sound source to be transcribed is input to the first learned machine learning model, and the feature map generated by the first learned machine learning model is input to the second learned machine learning model. A model processing unit that outputs a prediction probability that a note exists in a fixed-length section of the feature map;
A score generation unit that generates score information based on the predicted probability that the note exists;
Automatic transcription device with   The automatic transcription apparatus according to claim 6, wherein the model processing unit acquires a spectrogram by performing preprocessing on the sound source to be transcribed, and inputs the spectrogram to the first learned machine learning model. .   8. The automatic music transcription according to claim 6, wherein the model processing unit determines, as a predicted note, a note having a maximum prediction probability output from the second learned machine learning model for each point on the feature map. apparatus.   The automatic music transcription apparatus according to claim 8, wherein the musical score generating unit generates musical score information based on predicted notes extracted according to NMS (Non-Maximum Supplement). A processor that obtains a single sound source and pitch information as learning data of a first machine learning model, obtains a sound source to be transcribed and music information as learning data of a second machine learning model, Performing preprocessing on the sound source and the sound source to be transcribed, and obtaining respective spectrograms;
The processor inputs a spectrogram of the monophonic sound source as learning input data, and learns a first machine learning model by the pitch information so as to output a prediction probability of a pitch of the monophonic sound source,
The processor inputs, as learning input data, a feature map generated by inputting a spectrogram of the sound source to be transcribed to the learned first machine learning model, and outputs the feature map to a fixed-length section of the feature map. Learning a second machine learning model based on the score information to output a predicted probability that a note is present;
A learning method that has A processor inputting a sound source to be transcribed to a first learned machine learning model that outputs a predicted probability of a pitch from a single sound source;
The processor outputs a feature map generated by the first learned machine learning model to a second learned machine learning model that outputs a prediction probability that a note exists in a fixed length section of the feature map from the feature map. Inputting,
The processor generating score information based on a predicted probability that the note is output from the second learned machine learning model,
Automatic transcription method with A single sound source and pitch information are obtained as learning data of a first machine learning model, a sound source to be transcribed and musical score information are obtained as learning data of a second machine learning model, and the single sound source and the pitch information are obtained. Performing preprocessing on the sound source to be transcribed and obtaining spectrograms of each;
Inputting a spectrogram of the single-tone sound source as learning input data, and learning a first machine learning model by the pitch information so as to output a prediction probability of a pitch of the single-tone sound source;
A feature map generated by inputting the spectrogram of the sound source to be transcribed to the learned first machine learning model is input as learning input data, and a note exists in a fixed length section of the feature map. Learning a second machine learning model with the musical score information to output a prediction probability;
A program that causes a processor to execute Inputting a sound source to be transcribed into a first learned machine learning model that outputs a prediction probability of a pitch from a single sound source;
Inputting the feature map generated by the first learned machine learning model to a second learned machine learning model that outputs a prediction probability that a note exists in a fixed length section of the feature map from the feature map; ,
Generating musical score information based on a predicted probability that the note is output from the second learned machine learning model;
A program that causes a processor to execute