KR102031282B1 - Method and system for generating playlist using sound source content and meta information - Google Patents

Abstract

Disclosed are a method and a system for automatically generating a playlist using sound source contents and meta information. The playlist generating method may further include generating unique acoustic features of the corresponding sound source from the sound source data of the sound source content for each sound source content; And calculating a similarity between the sound source contents using the sound feature to generate a playlist based on the similarity between the sound source contents.

Description

Playlist automatic generation method and system using sound source content and meta information {METHOD AND SYSTEM FOR GENERATING PLAYLIST USING SOUND SOURCE CONTENT AND META INFORMATION}

The description below relates to a technique for automatically generating a playlist for a sound source.

Recently, people are using media content such as music pieces, video or audio tracks on various forms of media on a daily basis. Playlists provide users with the ability to organize music or other content based on preferences, themes, genres, artists or occasions.

As an example of a playlist generation technique, Korean Patent No. 10-0810276 (Registration Date February 27, 2008) discloses a technique for creating a user playlist for a plurality of sound source data stored in a sound source data reproducing apparatus. .

Conventionally, playlists are generated from a sound source having a play history or a preview history, or a similarity measurement method using a collaborative filtering feature is generated. Since the existing playlist generation method does not use the characteristics of the sound source itself as a factor, there is a high probability that a song having a different characteristic from the start song (seed song) will appear during automatic playback. In addition, when a playlist is generated using the metadata of the sound source as a strong filter condition, there is a problem that the diversity of the playlist is reduced.

The present invention provides a method and system for automatically generating a playlist by selecting a song similar to a user's favorite song or a song having a playing history using a model that has learned the sound source and meta information of the sound source.

A playlist generating method performed in a computer system, the method comprising: generating unique acoustic features of a corresponding sound source from sound source data of the sound source content for each sound source content; And calculating a similarity between the sound source contents using the sound feature to generate a playlist based on the similarity between the sound source contents, and generating the playlist includes: similarity between the sound source contents A playlist is created by selecting a next play song similar to the current play song based on the current playback song, and selecting a next play song by checking the similarity with the seed song. It provides a playlist generating method characterized in that the generation.

There is provided a computer-readable recording medium, in which a program for executing the playlist generating method through a computer is recorded.

A playlist generation system implemented in a computer, comprising: a unique feature processor for generating a unique sound feature of a corresponding sound source from sound source data of the sound source content for each sound source content; And a playlist generator configured to calculate a similarity between the sound source contents using the sound feature to generate a playlist based on the similarity between the sound source contents, and wherein the playlist generator is based on the similarity between the sound source contents. A playlist is generated by selecting a next playback song similar to the playback song, and generating a chain-type playlist with sound source content similar to the seed song by checking similarity with the seed song when the next playback song is selected. Provide a list generation system.

According to an exemplary embodiment of the present invention, similarity between songs can be more accurately calculated by using an eigen-value of a sound source itself as a factor for learning using deep learning.

According to an embodiment of the present invention, the similarity between songs can be calculated using not only the characteristics of the sound source but also meta information of the sound source as additional information, and a logic for continuously checking the similarity with the seed songs is applied. By doing so, it is possible to prevent the inclusion of songs that are too different from the seed songs while maintaining the diversity of the playlist.

1 is a block diagram illustrating an example of an internal configuration of a computer system according to an embodiment of the present invention.
2 illustrates an example of components that may be included in a processor of a computer system according to an exemplary embodiment of the present invention.
3 is a flowchart illustrating an example of a playlist generation method that may be performed by a computer system according to an exemplary embodiment of the present invention.
4 is a diagram illustrating an example of an automatic playlist generation process according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The present embodiments relate to methods and systems for creating a playlist, and more particularly, to a playlist generation technique capable of automatically generating a playlist using a model that has learned a sound source and meta information of the sound source. It is about.

Embodiments, including those specifically disclosed herein, achieve the creation of playlists for sound recording content and thereby achieve significant advantages in terms of cost savings, efficiency and accuracy, and the like.

The playlist generated for the sound source content may be utilized for recommendation, search, classification, management, etc. of the sound source content. For example, in order to recommend a user-customized sound source, a sorting operation for songs similar to a user's favorite song is required, and as a sorting operation, a playlist may be automatically generated by using meta information of the sound source and the sound source.

1 is a block diagram illustrating an example of an internal configuration of a computer system according to an embodiment of the present invention. For example, a playlist generation system according to embodiments of the present invention may be implemented through the computer system 100 of FIG. 1. As shown in FIG. 1, the computer system 100 is a component for executing a playlist generation method, the processor 110, the memory 120, the persistent storage device 130, the bus 140, and an input / output interface ( 150 and network interface 160.

Processor 110 may include or be part of any apparatus capable of processing any sequence of instructions. Processor 110 may include, for example, a processor within a computer processor, mobile device or other electronic device, and / or a digital processor. The processor 110 may be included in, for example, a server computing device, a server computer, a series of server computers, a server farm, a cloud computer, a content platform, a mobile computing device, a smartphone, a tablet, a set top box, a media player, and the like. The processor 110 may be connected to the memory 120 through the bus 140.

Memory 120 may include volatile memory, permanent, virtual, or other memory for storing information used by or output by computer system 100. The memory 120 may include, for example, random access memory (RAM) and / or dynamic RAM (DRAM). Memory 120 may be used to store any information, such as status information of computer system 100. Memory 120 may also be used to store instructions of computer system 100, including, for example, instructions for creating playlists for sound source content. Computer system 100 may include one or more processors 110 as needed or where appropriate.

Bus 140 may include a communication infrastructure that enables interaction between various components of computer system 100. Bus 140 may carry data, for example, between components of computer system 100, for example, between processor 110 and memory 120. Bus 140 may include wireless and / or wired communication media between components of computer system 100 and may include parallel, serial, or other topology arrangements.

Persistent storage 130 is a component, such as a memory or other persistent storage, such as used by computer system 100 to store data for some extended period of time (eg, relative to memory 120). Can include them. Persistent storage 130 may include non-volatile main memory as used by processor 110 in computer system 100. Persistent storage 130 may include, for example, flash memory, hard disk, optical disk, or other computer readable medium.

The input / output interface 150 may include interfaces for a keyboard, mouse, voice command input, display, or other input or output device. Input for configuration commands and / or playlist generation, such as sound source data and text information, may be received via the input / output interface 150.

Network interface 160 may include one or more interfaces to networks such as a local area network or the Internet. Network interface 160 may include interfaces for wired or wireless connections. Sound source data and text information of the sound source content for configuration commands and / or playlist generation may be received via the network interface 160.

In addition, in other embodiments, computer system 100 may include more components than the components of FIG. 1. However, it is not necessary to clearly show most of the prior art components. For example, the computer system 100 may be implemented to include at least some of the input / output devices connected to the input / output interface 150 described above, or may include a transceiver, a global positioning system (GPS) module, a camera, various sensors, It may further include other components such as a database. More specifically, when the computer system 100 is implemented in the form of a mobile device such as a smartphone, the acceleration sensor or gyro sensor, a camera, various physical buttons, a button using a touch panel, Various components, such as an input / output port and a vibrator for vibration, may be implemented to be further included in the computer system 100.

2 is a diagram illustrating an example of components that a processor of a computer system according to an embodiment of the present invention may include, and FIG. 3 is a playlist that may be performed by the computer system according to an embodiment of the present invention. It is a flowchart which shows the example of a generation method.

As shown in FIG. 2, the processor 110 may include a sound source input controller 210, a sound source preprocessor 220, a unique feature processor 230, and a playlist generator 240. The components of such a processor 110 may be representations of different functions performed by the processor 110 in accordance with a control instruction provided by at least one program code. For example, the sound source input controller 210 may be used as a functional representation that operates to control the computer system 100 so that the processor 110 receives sound source data. The processor 110 and the components of the processor 110 may perform steps S310 to S350 included in the playlist generating method of FIG. 3. For example, the processor 110 and the components of the processor 110 may be implemented to execute instructions according to the code of the operating system included in the memory 120 and the at least one program code described above. Here, at least one program code may correspond to a code of a program implemented to process the playlist generation method.

The playlist creation method may not occur in the order shown, and some of the steps may be omitted or additional processes may be included.

In operation S310, the processor 110 may load program code stored in a program file for a playlist generation method, into the memory 120. For example, the program file for the playlist generation method may be stored in the persistent storage device 130 described with reference to FIG. 1, and the processor 110 may store the program file stored in the persistent storage device 130 through the bus. Computer system 110 may be controlled such that program code is loaded into memory 120.

At this time, each of the processor 110 and the sound source input controller 210, the sound source preprocessor 220, the unique feature processor 230, and the playlist generator 240 included in the processor 110 may be stored in the memory 120. It may be different functional representations of the processor 110 for executing instructions of a corresponding portion of the loaded program code to execute subsequent steps S320 to S350. In order to execute the steps S320 to S350, the processor 110 and the components of the processor 110 may directly process an operation according to a control command or control the computer system 100.

In operation S320, the sound source input controller 210 may control the computer system 100 to receive sound source data of the sound source content with respect to the sound source content. In this case, the sound source content may mean all digital data having an audio file format, and may include, for example, MP3 (MPEG Audio Layer-3), WAVE (Waveform Audio Format), and FLAC (Free Lossless Audio Codec). have. The sound source input controller 210 may control the computer system 100 to receive text information related to sound source content. The text information related to the sound recording content may include lyrics or meta information such as a singer, genre, title, album name, etc., and hashtags or queries input in relation to the classification or search of the sound recording content. May contain information. The sound source data and text information may be input or received by the computer system 100 through the input / output interface 150 or the network interface 160, and may be stored and managed in the memory 120 or the permanent storage device 130.

In operation S330, the sound source preprocessor 220 may process the sound source data received in operation S320 in the form of learning data. In this case, the sound source preprocessor 220 may express the sound source data in time-frequency through preprocessing. For example, the sound source preprocessor 220 may convert the sound source data into time-frequency-size data such as Mel-spectrogram or Mel Frequency Cepstral Coefficient (MFCC). The sound source preprocessor 220 may preprocess the text information when the text information is input together with respect to the sound source content. For example, the sound source preprocessor 220 may filter meaningless texts from input text information using a language preprocessor such as a morpheme analyzer and an index word extractor. In other words, the sound source preprocessor 220 removes words and special symbols (!,?, /, Etc., etc.) of unnecessary parts of speech, such as surveys and quiet verbs, which are included in text information, and extracts words corresponding to spoken words or roots. can do.

In operation S340, the unique feature processor 230 may generate the unique feature by learning the preprocessed training data on the sound source data through the learning model, and then store the generated unique feature in a database (not shown). The unique feature processor 230 may generate unique acoustic features of the sound source data itself using deep learning. For example, the unique feature processor 230 may use a learning model based on a convolutional neural network (CNN). The unique feature processor 230 may express sound source data as a multidimensional real vector using a CNN learning model. The CNN learning model may include a sound source data learning layer, and steps 1 to 3 below may be examples of a process of generating a real vector corresponding to sound source data in the sound source data learning layer.

In process 1, the sound source data (eg, mp3 file) included in the sound source content may be converted into data in a time-frequency-size form such as mel-spectrogram or MFCC through preprocessing. For example, the sound source preprocessor 220 may express sound source data input by the sound source input controller 210 in time-frequency.

A plurality of frequency frames for one or more short time intervals (1 second to 10 seconds) may be sampled from the sound source data converted in step 2 and used as input data for the learning model of the sound source data. For example, the unique feature processor 230 may sample a plurality of frames and use it as an input of a CNN model presented as an example of a sound source model in the sound source data learning layer. Therefore, the CNN model for learning sound source data may be a model having the same number of channels as the number of sampled frames. Alternatively, each generated frame can be used as a single channel, and after convolutional / pooling of each frame, the generated feature vectors for each frame can be bonded to each other to be used as an input of a fully connected layer.

In step 3, an abstracted feature may be generated from the sound source frame by repeatedly configuring a plurality of convolutional and pooling layers that the sound source data learning layer may include. In convolution, the patch size can be configured in various ways. Among the pooling techniques such as the pooling technique using the maximum pooling degree (max), the pooling technique using the average value, and the hybrid pooling technique combining the two pooling techniques, At least one may be used.

Above the multiple convolutional and pooling layers is a fully-connected layer for generating acoustic features, with each layer function comprising the sigmoid function, the hyperbolic tangent (Than) function, and the ReLU ( Various functions such as Rectified Linear Unit) function can be used. As a result, one multidimensional real vector may be generated for the sound source data. For example, given the first sound source data m0, x0 = {0.2, -0.1, 0.3,... In the output layer of the sound source learning model. } Can be expressed as a multidimensional real vector. The number of dimensions is an integer greater than zero and can empirically have a value from 50 to 500, in general. This process 3 may be processed by the unique feature processor 230 described above.

In addition, when the unique feature processor 230 receives text information with respect to the sound source content, the text information may also be expressed as a multidimensional real vector. For example, the unique feature processor 230 may generate texts filtered through preprocessing of the text information into a word vector using a previously trained learning model. For example, the word vector may be expressed in the form of a numerical multidimensional vector. A word appearance frequency histogram, a term frequency (TF) / inverse document frequency (IDF), a language learning model (eg, word2vec, phrase2vec, document2vec, etc.) may be used to generate the word vector. For example, singer / genre / title / year is one n-dimensional real vector v = {0.3, -1.2, 1.2,... } Can be expressed as In addition, the order of the text information fields (singer, genre, title, year, etc.) for the language learning model is not fixed and can be changed to suit the purpose.

The unique feature processor 230 may store and maintain a feature vector for sound source data and a word vector for text information in a database for sound source content. The feature vector for the sound source data and the word vector for the text information may be constructed in separate databases, or in one database. Such a database may be implemented as a component included in the computer system 100 or may exist as an external database built on a separate system that can interoperate with the computer system 100.

In operation S350, the playlist generator 240 may calculate the similarity between sound sources using the inherent characteristics of the sound source data stored in the database with respect to the sound source content, and the playlist for the sound source content based on the similarity between the sound sources. Can be generated automatically. For example, the playlist generating unit 240 may calculate the similarity between sound sources using the feature vector of the sound source data. In another example, the playlist generator 240 may use the word vector for the text information together with the feature vector for the sound source data. Similarity between sound sources can be calculated. In this case, when the user selects a specific song, the playlist generator 240 may generate the playlist in the form of a chain of songs similar to the seed song by using the corresponding song as a seed song. The playlist generator 240 may generate a playlist of songs that the user prefers based on the user's recent log history of the sound source content. It is also possible to create playlists.

Therefore, the present invention can automatically generate playlists with similar songs based on the similarity between songs using a model that has learned text information such as sound source and meta information of the sound source.

4 is a diagram illustrating an example of an automatic playlist generation process according to an embodiment of the present invention. Steps S401 to S407 of FIG. 4 may be performed by being included in step S350 of FIG. 3.

In operation S401, when the seed song is selected, the playlist generator 240 may calculate the similarity between the seed song and the other song by using unique features for each sound source stored in the database. For example, when a user selects and plays a specific song, the song may be selected as a seed song. As another example, a user's favorite song may be selected as a seed song based on a user's recent log history of the sound source content. For example, when the seed song is m0, the processor 110 converts the seed song m0 into x0, which is a d-dimensional real vector representing the inherent characteristics of the sound source, using the CNN model when the seed song m0 is a new song that does not exist in the database. If the seed song m0 is not a new song, the d-dimensional real vector corresponding to the seed song m0 is retrieved from the database and defined as x0. The sound source feature vector may be used to calculate the similarity between songs. As another example, the text feature vector v representing text information such as meta information of each song in a vector form may also be used to calculate the similarity between songs along with the sound source feature vector. have. The playlist generator 240 may calculate a similarity between the sound source feature vector of the songs stored in the database and the sound source feature vector x0 of the seed song m0. In one example, a cosine distance that calculates the distance between vectors may be used. The cosine distance CD between two vectors a and b may be defined as in Equation 1. That is, it may be determined that the smaller the distance between the vectors, the higher the similarity of the songs, and the larger the distance between the vectors, the lower the similarity of the songs.

In addition to the cosine distance, the similarity between songs can be calculated using various methods such as Euclidean distance and Hamming distance. In this case, the sound source feature vector and the text feature vector may be combined and calculated, or may be represented as weighted sums after calculating the distance between the vectors.

In operation S402, the playlist generator 240 may select a plurality of songs having similarities with the seed songs m0 among the songs in which the unique features are stored in the database and generate a separate candidate set M *. For example, the playlist generator 240 may select a song having a similarity level or more from other songs sung by the singer of the seed song m0, or select a predetermined number of songs in order of high similarity to generate a candidate set M *. can do. As another example, the playlist generator 240 may generate a candidate set M * by querying a database only for songs whose text feature vector distance between songs is equal to or less than a predetermined value.

In operation S403, the playlist generator 240 may select a song having a high similarity to the current playback song as the next playback candidate song based on the similarity between the songs whose unique features are stored in the database. When the song to be played is called mt, a song having a high similarity with respect to the songs in the database may be selected as mt + 1, which is a song to be played next time. In this case, the playlist generator 240 may determine a song having the greatest similarity with the current playback song mt or a song randomly selected from among a plurality of songs selected based on the similarity as the next playback candidate song mt + 1. The already reproduced song may be excluded from the selection target of the next reproduction candidate song.

In operation S404, the playlist generating unit 240 calculates a similarity with at least one reference song of a seed song or a previous playback song with respect to the candidate song selected in operation S403, and defines a predetermined threshold. Can be compared with At this time, the threshold serves to prevent a song that is too different from the seed song or the reference song from being played while maintaining the diversity of the playlist. In other words, the present invention may include logic for continuously checking similarity with the seed song or the reference song through the threshold value in selecting the next play candidate song. For example, the threshold value can be arbitrarily selected from the threshold distribution among the whole songs, and can usually be set within 0.2 to 0.3. Thresholds can be used to balance or weight the diversity and similarity of playlists. The reference song may be the first song or a song having a history of being reproduced within a predetermined range (for example, within the fifth time and within one hour) after the first playback song.

In steps S405 and S406, the playlist generator 240 determines that the candidate song selected in step S403 has a similarity with the seed song or the reference song higher than the threshold value (that is, the distance between the vectors is smaller than the threshold distance). If the candidate song selected in step S403 is determined as the final next playback song, it can be played after the current playback song.

In steps S405 and S407, the playlist generator 240 determines that the candidate song selected in step S403 has a similarity with the seed song or the reference song below the threshold value (that is, the distance between the vectors is larger than the threshold distance). In this case, any one of the songs included in the candidate set M * generated in step S402 may be selected as the final next playback song, and may be played after the current playback song. For example, if the candidate song selected in step S403 has a similarity with the seed song or the reference song below the threshold value, the playlist generator 240 may select any one from the candidate set M * which is a set of songs similar to the seed song. You can select and play songs.

The playlist generator 240 may repeat steps S403 to S407 in the above process, and may infinitely generate a playlist in a chain form based on the similarity between songs. Accordingly, the present invention can secure the diversity of the playlist by probabilistically selecting the next song to be played continuously based on the similarity between songs, and at the same time, the similarity with the seed song (or reference song) when selecting the next play song. When lowered below a certain value, the similarity of the playlist with the seed song (or the reference song) may be replaced with a song having a high ranking.

As described above, according to embodiments of the present invention, similarity between songs can be more accurately calculated by using the unique value of the sound source itself as a factor for learning using deep learning. Furthermore, according to embodiments of the present invention, similarity between songs can be calculated using not only the characteristics of the sound source but also the meta information of the sound source as additional information, and the speed of calculating the similarity between songs can be dramatically reduced by using the meta information in the form of a filter. By applying logic to continuously check similarity with seed songs, it is possible to prevent playlists from being too different from the seed songs while maintaining the diversity of playlists.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable PLU (programmable). It can be implemented using one or more general purpose or special purpose computers, such as logic units, microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims (3)

In a playlist generation method performed in a computer system,
Generating, for each sound source content, unique acoustic features of the sound source from sound source data of the sound source content; And
Calculating a similarity between the sound source contents using the sound feature to generate a playlist based on the similarity between the sound source contents;
Including,
Generating the playlist,
Based on the similarity between the sound source contents, a candidate song similar to the current playback song is selected as the next playback song to generate a playlist.
The candidate song is selected as the next playback song according to the similarity between the candidate song and the seed song by checking the similarity between the candidate song and the seed song when the next playback song is selected, or another similar to the seed song. Creating the playlist in chain form by selecting a song as the next playback song
Playlist generation method characterized in that. A computer-readable recording medium in which a program for executing the playlist generating method of claim 1 is executed by a computer. In the playlist generation system implemented by computer,
A unique feature processor for generating a unique sound characteristic of the corresponding sound source from the sound source data of the sound source content for each of the sound source contents; And
A playlist generator for generating a playlist based on the similarity between the sound source content by calculating the similarity between the sound source content using the sound feature
Including,
The playlist generator,
Based on the similarity between the sound source contents, a candidate song similar to the current playback song is selected as the next playback song to generate a playlist.
When selecting the next playback song, the similarity between the candidate song and the seed song is checked to select the candidate song as the next playback song or another song similar to the seed song according to the similarity between the candidate song and the seed song. Creating the playlist in chain form by selecting as a playback song
Playlist creation system, characterized in that.

Images