KR20170128060A - Melody extraction method from music signal - Google Patents

Abstract

The present invention relates to a method for extracting a melody by analyzing played music, which comprises the following steps. A computer device receives a continuous sound forming music. The computer device converts the sound into a frequency domain. The computer device determines a melody candidate group including a plurality of melodies by applying the sound of the frequency domain to a first machine learning model with regular time units. And the computer device determines one melody for each time unit by applying the melody candidate group to a second machine learning model previously learning with a melody group on a sound source.

Description

[0001] MELODY EXTRACTION METHOD FROM MUSIC SIGNAL [0002]

The technique described below relates to a technique for extracting a melody line from the music being reproduced.

A technique for analyzing music as an analog signal to provide a certain service has appeared. For example, there is a service for searching for information about music currently being played back using a smart device, or for classifying music to be played back regularly.

Korean Patent Publication No. 10-2011-0080554

The technique described below is intended to provide a method of automatically extracting a melody line by analyzing music being reproduced.

A method of analyzing music to be reproduced and extracting melody comprises the steps of: receiving a continuous sound constituting music by a computer device; converting the sound into a frequency domain; Applying a sound in the frequency domain to a first machine learning model to determine a melody candidate set including a plurality of melodies per unit of time; and determining, by the computer device, the melody candidate set as a melody set for the melody And determining one melody for each time unit by applying the second machine learning model to the second machine learning model.

The technique described below uses a two-step machine learning model to determine melody lines for music that is played at low complexity and cost.

FIG. 1 shows an example of a configuration of a device for extracting a melody by analyzing a reproduced sound source.
2 is another example showing a configuration of a device for extracting a melody by analyzing a reproduced sound source.
3 is an example of a flowchart of a method of extracting melody by analyzing a sound source to be reproduced.
FIG. 4 shows an example of a process of extracting melody by analyzing a reproduced sound source.
FIG. 5 shows an example of CNN used in the process of analyzing the reproduced sound source.

The following description is intended to illustrate and describe specific embodiments in the drawings, since various changes may be made and the embodiments may have various embodiments. However, it should be understood that the following description does not limit the specific embodiments, but includes all changes, equivalents, and alternatives falling within the spirit and scope of the following description.

The terms first, second, A, B, etc., may be used to describe various components, but the components are not limited by the terms, but may be used to distinguish one component from another . For example, without departing from the scope of the following description, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

As used herein, the singular " include "should be understood to include a plurality of representations unless the context clearly dictates otherwise, and the terms" comprises & , Parts or combinations thereof, and does not preclude the presence or addition of one or more other features, integers, steps, components, components, or combinations thereof.

Before describing the drawings in detail, it is to be clarified that the division of constituent parts in this specification is merely a division by main functions of each constituent part. That is, two or more constituent parts to be described below may be combined into one constituent part, or one constituent part may be divided into two or more functions according to functions that are more subdivided. In addition, each of the constituent units described below may additionally perform some or all of the functions of other constituent units in addition to the main functions of the constituent units themselves, and that some of the main functions, And may be carried out in a dedicated manner.

Also, in performing a method or an operation method, each of the processes constituting the method may take place differently from the stated order unless clearly specified in the context. That is, each process may occur in the same order as described, may be performed substantially concurrently, or may be performed in the opposite order.

When a CD, an MP3, or the like is reproduced by a reproducing apparatus, the reproducing apparatus outputs an analog signal through a speaker. Or, in the field, the player plays the instrument directly, or even if the singer makes a song, consecutive notes are delivered to the listener as analog signals. Basically, a human recognizes an analog signal as an ear and identifies a specific sound. The technique described below relates to a technique for estimating a melody for a music based on an analog signal such as a music to be reproduced.

Describe music-related terms in a relaxed manner. The pitch depends on the frequency of the generated sound. The pitch of the note is usually 7 degrees, such as A, B, C, D, E, F and G. The height of a note is expressed in the scale, and the height of the note varies with the position of the note. For example, the A4 note has a size of about 440 Hz and the A5 has a size of 880 Hz.

The melody corresponds to the arrangement of the pitch of the sound over time. Also, the height of each note has a constant duration. Arrangement of the pitch of the sound with a constant occurrence time corresponds to the melody rhythm. The rhythm is determined by how long the note lasts when it occurs. The technique described below is a technique for extracting a melody for a music to be reproduced.

Tonality refers to a system in which the pitch of a tone or chord is arranged so that stability and attraction are perceived. In general, if the pitch or note is in a stable sense, the composition is consonant, and if the pitch or note is not stable, the composition is dissonant. Dissonance includes tension against the sense of stability. Resolution refers to the process in which a chord or note changes from discordance to harmony. Composition includes information that distinguishes between harmonious composition and incompatible composition. As will be described later, the machine learning model is used to estimate melody by learning information about compositing.

FIG. 1 shows an example of a configuration of a device for extracting a melody by analyzing a reproduced sound source.

1 (a) is an example of extracting a melody using a smart device 50 such as a smart phone. The smart device 50 acquires an embedded micro analog music signal. The smart device 50 converts the analog signal into the frequency domain and applies the frequency domain signal to the machine learning model to determine the melody for the input music. The machine learning model will be described later. On the other hand, the smart device 50 may simultaneously determine a melody by receiving a micro signal while reproducing a sound source with a speaker.

1 (a), the smart device 50 includes a mark 51, a storage device 52, a computing device 53, and an output device 54. The microphone 51 acquires an analog music signal. The computing device 53 applies the analog music signal to the machine learning model to determine the melody. The storage device 52 may store data for a machine learning model, training data for machine learning, and the like. The storage device 52 may temporarily store the acquired analog music signal. The storage device 52 may store information on the melody corresponding to the analog music signal. The output device 54 can output information on the determined melody.

1 (b) is an example of extracting melody using a device such as a PC. The user acquires an analog music signal with the microphone 81 connected to the computer 85. [ The computer 85 converts the analog signal into the frequency domain and applies the frequency domain signal to the machine learning model to determine the melody for the input music.

1 (c) is an example in which the server 98 at a remote location determines melody. The computer 85 acquires an analog music signal through the microphone 81. [ The computer 85 may communicate the analog music signal to the server 98. [ In this case, the server 98 converts the analog signal transmitted from the computer 95 into a frequency domain, and applies a frequency domain signal to the machine learning model to determine a melody for the input music.

Or the analog music signal obtained by the computer 85 into a frequency domain signal and transmit the signal in the frequency domain to the server 98. [ In this case, the server 98 applies the received signal in the frequency domain to the machine learning model to determine the melody for the input music.

2 is another example showing a configuration of a device for extracting a melody by analyzing a reproduced sound source. FIG. 2 is an example of a process of extracting melody through the apparatus illustrated in FIG. 1 (c). The computer 95 transmits the analog music signal or the signal converted into the frequency domain to the server 98 via the network as described above. The server 98 applies a frequency domain signal to the machine learning model to extract a certain melody. The model DB 99 can store data for the machine learning model or data for learning. The server 98 may transmit the melody information corresponding to the music to the computer 95. [ The melody information can be expressed in various ways, but in the following description, it is assumed that the melody information is represented by a text indicating a sound height. FIG. 2 shows an example (No. 1) of transmitting a character indicating a sound height with respect to a 7-degree tone regardless of an octave, and an example (No. 2) of transmitting a character expressed with octave to a 7-degree tone. In the second example, the octave is numbered. Although not shown in FIG. 2, the computer 95 may output the received melody information to the screen or store the received melody information.

For convenience of explanation, it is explained below that the computer device extracts or decides the melody. The computer device may be the above-described smart device 50, computer 85, server 98, and the like.

FIG. 3 is an example of a flowchart for a method 100 for extracting a melody by analyzing a reproduced sound source. The computer device uses a two-step machine learning model to cool the melody of the music being played (sound source). First, the computer device receives a continuous sound constituting music (110). The computer device converts the received sound into a frequency domain (120). For example, a computer device can convert an analog signal into a frequency domain by applying STFT (Short-Time Fourier Transform). Alternatively, the computer device may convert the analog signal into a frequency-domain signal such as a Mel-spectrogram. Further, the computer device may use other conversion techniques to transform the analog signal into a frequency domain signal. A detailed description of a specific process of converting an analog signal into a frequency domain signal will be omitted. The computer device processes consecutive notes in a constant time unit. The constant time unit may be, for example, a few seconds or a few tens of seconds. To do this, a computer device can use a transform such as STFT.

The computer device applies a frequency domain sound signal to the first machine learning model to determine all available melodies per unit of time (130). The first machine learning model determines possible melody candidates for each time unit. Thus, the computer device obtains a set containing a plurality (or at least one) of melody lines per time unit through a first machine learning model. A melody set obtained through the first machine learning model is hereinafter referred to as a melody candidate set.

The computer device applies the melody candidate set to the second machine learning model previously learned by the melody set for the sound source, and determines one melody per unit of time (140). The computer device uses a melody candidate set as input to extract one melody line for each time unit. Finally, the computer device can extract the entire melody by combining the extracted melodies in units of time. Meanwhile, the computer device can output the extracted melody to the screen (150). The computer device may store text information about the extracted melody. The computer device may transmit text information about the extracted melody to a specific client device via the network.

Hereinafter, the melody extraction process will be described in detail with reference to the above-described machine learning model. FIG. 4 shows an example of a process of extracting melody by analyzing a reproduced sound source. The processes shown in FIG. 4 are all performed in a computer device.

 As described above, the computer device extracts the melody by a predetermined time unit. Although not shown in FIG. 4, the computer apparatus parses the analog music signal by a predetermined time unit, or parses the frequency-domain-converted signal by a predetermined time unit. The whole process is then performed on a constant time basis. For example, a computer device may apply a signal belonging to a first time unit to a first machine learning model to determine a first set of melody candidates, apply a first set of melody candidates to a second machine learning model to generate a first melody . The computer device then repeats the same process for other time units. The computer device can estimate the melody in a sequential time unit order. Or the computer device may perform operations on a plurality of time units in parallel. Finally, the computer device combines the melody lines for each time unit in time to extract the entire melody line.

4, the first machine learning model is referred to as a first AI model 210 and the second machine learning model is referred to as a second AI model 220. [ It means that a computer device extracts melody from an analog music signal using artificial intelligence (AI).

The first AI model 210 receives a frequency domain signal and generates a melody candidate set. The first training data 215 is used for learning by the first AI model 210 to determine a melody candidate set.

The sound waves converted into the frequency domain can be expressed on the Hz axis and the Db axis. Considering time, music becomes three-dimensional data. The first AI model 210 corresponds to a kind of classifier that classifies the three-dimensional data into a set of harmonics that can be compressed in time units.

The first AI model 210 is prepared using a machine learning model. For example, a KNN (K Nearest Neighbor) algorithm, a DNN (Deep Neural Network), or a CNN (Convolutional Neural Network) may be used as the machine learning model. Each model is briefly described.

It is desirable to use harmonic music in training data. The music as the training data is tagged with the melody for the music. The melody can be represented by a symbol (C, D, C7, etc.) representing the code and by information (e.g., a number) representing the octave. The training data may include sound waves played on one or more musical instruments. Several algorithms are described. Of course, the first AI model 210 may be generated based on another algorithm.

Model based on KNN

KNN consists of the k closest training data in the feature space. KNN is the item assigned to the most common items among the k nearest neighbors to yield the final result. On the other hand, there are a variety of algorithms that can build classified artificial intelligence by learning a large amount of labeled data such as support vector machines, artificial neural networks, and Naive Bay classification methods.

The first AI model 210 classifies n clusters by learning data with a total of n kinds of labels. Then, the first AI model 210 determines which cluster among the previously classified clusters the input data should be allocated to. The decision / prediction methodology belongs to the area of instructional learning within the machine learning domain. The first AI model 210, which has learned the training data through the map learning process, can perform a classification operation for indicating to what kind of label the additional input data can be attached.

KNN is a nonparametric technique that finds samples that are closest to the sample when a new sample arrives, and groups the sample based on the dominant sample. The nonparametric method can be useful for processing nonlinear discrete data when combined with map learning. After clustering, KNN calculates the silhouette values by evaluating clusters inferred by the nonparametric method based on the actual attached label. At this time, if each cluster is well clustered based on the label, the silhouette value will be evaluated close to 1. The first AI model 210 executes the KNN and places the difference between the silhouette value and 1 as a cost function.

  A large amount of music data is high-dimensional data. For data compression, it is possible to perform dimension reduction by Principal Component Analysis (PCA) as a preprocessing process. In this case, the first AI model 210 determines whether the cost function is minimized when clustering based on a combination of principal components (PC), and stores the principal component combination that minimizes the cost function. When input data is received, the first AI model 210 linearly converts the input data to correspond to the combination of the principal components, and classifies the converted data on the basis of the Euclidean distance corresponding to the existing cluster. The first AI model 210 extracts clusters having a high probability of being input test data, and outputs a cluster list having a p-value of 0.05 or less.

Model based on DNN

DNN is an artificial neural network composed of several hidden layers between the input layer and the output layer. DNN can model complex nonlinear relationships as well as general neural networks.

If DNN is used, octave information of the attached label is ignored, and it is possible to construct a one-hot vector by taking only code information. Here, the code information means information that excludes the octave from the melody and represents only the pitch of the sound. One layer of the DNN is implemented equal to the number of dimensions of the input data, and the dimension of the last layer is configured to be equal to the label type n. The back propagation method is implemented using the slope descent method.

Model based on CNN

CNN is a kind of multilayer perceptrons designed to use minimal preprocessing. CNN is composed of one or several layers of convolutional hierarchy and general artificial neural networks layered on top of it, and utilizes additional weighting and pooling layers. This structure allows CNN to fully utilize the input data of the two-dimensional structure.

  Ignoring octave information is a necessary task for CNN to avoid the overfitting problem. The label is mapped to an n-dimensional vector, the sum of which is 1. When the input data is mapped to one label, the parameter value of the corresponding cluster becomes 1. When the data is mapped to several labels at the same time, the cumulative number in the learning data is assigned to the parameter corresponding to the label, When finished, level the total sum of all labels to 1.

For example, when the input data is 625-dimensional, convolution layer and pooling layer are overlapped three times, and the output result is connected to a fully connected neural network, Lt; / RTI > In this case, the pooling layer takes the input data as a rectangle rather than a square. The reason is that the sound wave data is a virtual value shifted to the frequency domain through the Fourier transform, unlike the image data in which the shape, shape, or color or combination thereof is important, and the comparison of the db value that increases along the hz- This is the key. Therefore, a 1 × 4 pulling layer is arranged to extract the melody from the frequency domain signal. Note that a 2x2 pooling layer is placed every time a dimension is pooled in a quarter of normal image data.

FIG. 5 shows an example of CNN used in the process of analyzing the reproduced sound source. A signal obtained by converting an analog music signal into a frequency domain signal has an input value. FIG. 5 shows all N convolution layers and N pooling layers. Meanwhile, as described above, the pulling layer has a 1x4 form.

Returning to the description of FIG. The first AI model 210 determines a melody candidate set by inputting a frequency domain signal for a unit time. FIG. 4 shows an example in which all three melody candidates are derived.

The second AI model 220 receives the melody candidate set and extracts one melody. The second training data 225 is used for learning by the second AI model 220 to determine the melody candidate set. The second training data 225 is data in which the music wave is divided by a predetermined time unit. In FIG. 4, the second AI model 210 finally determines one melody for the unit time.

The second AI model 212 may also be provided with various algorithms, such as the first AI model 210. For the sake of convenience of description, the description will be made on the basis of a hidden Markov model (HMM). That is, a model such as CNN. The second AI model 212 is a model that performs learning about harmony through the training data. The second AI model 212 receives a string corresponding to the melody as an input. The second AI model 212 is an artificial intelligence that learns the type and probability of the next character based on the rank n characters configured according to the user's purpose. The rank n has a constant of 1 or more as a default, and it goes through a process of increasing n by 1 for each iteration and finding n where the cost function is minimized. The cost function is defined as the difference between the result produced after the input of the training data of the model as the test data and the existing data. Depending on the genre of music, a separate second artificial intelligence can provide the most predictive accuracy.

The melody candidate set is input to the second AI model 212, and sequentially selects one of the options matching the probabilistic automata in the second AI among various codes existing every time unit. The second AI model 212 finally outputs only one code progress string listed in the course of time.

The present embodiment and drawings attached hereto are only a part of the technical idea included in the above-described technology, and it is easy for a person skilled in the art to easily understand the technical idea included in the description of the above- It will be appreciated that variations that may be deduced and specific embodiments are included within the scope of the foregoing description.

50: Smart devices
51: microphone
52: Storage device
53:
54: Output device
81: Microphone
85: Computer
91: microphone
95: Computer
98: Server

Claims (7)

Receiving a continuous sound constituting music by a computer device;
The computer device converting the sound into a frequency domain;
Applying the sound of the frequency domain to the first machine learning model at a predetermined time interval to determine a melody candidate set including a plurality of melodies per unit of time; And
And the computer device applies the melody candidate set to a second machine learning model previously learned by a melody set for a sound source to determine one melody for each time unit to extract a melody How to. The method according to claim 1,
Wherein the first machine learning model is a model based on KNN (K Nearest Neighbor) algorithm, DNN (Deep Neural Network) or CNN (Convolutional Neural Network). The method according to claim 1,
Wherein the second machine learning model is a model based on DNN (Deep Neural Network) or CNN (Convolutional Neural Network). The method according to claim 1,
Wherein the first machine learning model extracts a melody by analyzing music to be reconstructed in advance using data obtained by converting a sound generated by reproducing a plurality of commercial sound sources into a frequency domain. The method according to claim 1,
Wherein the computer device applies the melody candidate set for the currently input time unit and the melody candidate set for the previously input time unit to the second machine learning model that has learned the Mars using the melody line for the commercial sound source, Wherein the melody is extracted by analyzing music to be played back which determines the one melody for the current time unit. The method according to claim 1,
Wherein the melody candidate set is constituted by information indicating a pitch of the sound, and the computer device analyzes the music to be reproduced which determines the one melody represented by the information on the time series in succession with respect to the note, How to. The method according to claim 1,
The first machine learning model is based on a CNN (Convolutional Neural Network), and the CNN is generated on the basis of a sound in the frequency domain. The CNN analyzes melody reproduced by analyzing the music having a 1x4 pooling layer Way.

Images