KR20100007102A - Online digital contents management system - Google Patents

Abstract

PURPOSE: An online digital content management system is provided to manage personal digital contents online by using an audio matching technique. CONSTITUTION: A feature extracting unit extracts feature values from audio data of digital contents selected by a user, and a feature matching unit(27) compares the extracted feature values with the audio feature values stored at a feature database. The feature matching unit searches the matched digital contents, and a content management unit(29) provides a user terminal with the matched digital contents. The contents management unit includes the detailed information to a personal possession list of the user.

Description

ONLINE DIGITAL CONTENTS MANAGEMENT SYSTEM

The present invention relates to a digital content management system through online.

Recently, due to the development of data communication such as the Internet or a mobile communication network, many people can easily access various multimedia data through the Internet or a mobile communication network. For example, there are a number of sites that sell, provide, or share music, movies, and the like online, and users can easily access the sites to obtain their desired multimedia data. In addition, it is possible to purchase CDs, DVDs and the like offline or online, convert the data, and store the data in a portable terminal or a personal PC.

However, in order for a user to play or use his or her own multimedia data on a plurality of devices, the corresponding data must be stored for each device. For example, if you want to listen to music that you own using your home computer, your company's computer, or your MP3 device at work, you should store it on all of those devices. This is a very cumbersome task.

In addition, in the case of reproducing the sound sources in the user's terminal, the user often cannot know the detailed information of the data held by the user. This is because in general, music files that are traded online are not distributed through normal transactions, so individual sound sources obtained through such a trading path do not have file names, artist information, album information, song names, copyright information, etc. Even if the product is purchased through a normal transaction path, the attribute information may be damaged due to an individual's file name change or neglect in management.

Accordingly, an object of the present invention is to provide an online digital content management system that can manage digital content held by an individual online using audio matching technology. In addition, it is to provide a digital content management system and digital content management server through the online that can automatically organize and show the detailed information of the digital content owned by the individual, and can be settled based on the copyright information. In addition, when an individual downloads his or her digital content again, it is to provide a digital content management system through online that can clean up the file name of the corresponding content and input additional meta information.

According to the present invention, there is provided a digital content management system online, comprising: a media storage unit for storing digital content; A feature database storing audio feature values for the digital content; A personal database storing a personal possession list of digital contents held by each user; A feature extractor for extracting feature values from audio data of the digital content uploaded and selected by the user; A feature matching unit which searches the matched digital content by comparing the extracted feature values with the audio feature values stored in the feature database; And a content management unit which provides detailed information on the matched digital content to the user terminal and includes the content information in the personal possession list of the user. have.

And the user terminal, wherein the media storage unit, the feature database, the personal database, and the content manager are provided in the digital content management server, and the feature extractor and the feature matcher are connected to the user terminal. It may be provided in any one of the digital content management server.

The detailed information of the digital content provided to the user terminal may include at least two of a title, an artist name, a producer, an album name, a playing time, a copyright holder, an album image, and lyrics.

The content manager may extract or stream digital content corresponding to the request from the feature database in response to a user's request to play or download the digital content included in the private list, and transmit or stream the digital content corresponding to the request. The digital content transmitted in response may normally include a file name and attribute information.

When the matching fails, the content manager may receive the digital content uploaded and selected by the user and construct a database.

Meanwhile, the feature extracting unit decodes some data corresponding to a certain reproduction time from the start time of reproduction to a predetermined reproduction time in the audio data, decodes the data, converts the decoded data into a mono format, and converts the mono data into the mono format. After the normalized data is processed, a starting point at which a sound starts may be searched to extract a feature vector from the data after the searched starting point.

In addition, the feature extracting unit checks whether the minimum waveform model constituting the sound exists in the unit of the normalized data on a frame basis, and when a predetermined number or more frames of the minimum waveform model are continuously detected, Determine, as a starting point, a position where the first minimum waveform model of the detected frames is detected; The minimum waveform model may include a predetermined number or more of the number of samples having a predetermined value or more energy.

In addition, the feature extractor may add summed feature vectors for each predetermined number of frames to calculate sum data, calculate difference values between the sum data, and output the sum values.

As described above, according to the present invention, there is provided an online digital content management system that can manage digital content held by an individual online using audio matching technology. In addition, a digital content management system and a digital content management server are provided online that can automatically organize and show detailed information of a digital content owned by an individual and make a settlement based on copyright information. In addition, when an individual downloads his or her own digital content again, an online digital content management system is provided that can clean up the file name of the corresponding content and input additional meta information.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings.

1 is a schematic diagram of a digital content management system via online according to a first embodiment of the present invention. As shown in FIG. 1, the digital content management system online according to the present invention includes a user terminal 10a, 10b, 10c, 10, a digital content management server 20, and a network 30 for interworking therebetween. ).

The user terminal 10 is a personal terminal for receiving a digital content management service. For example, various devices such as a personal PC, a notebook computer, a PDA, and an MP3 device may be used. In addition, any device that can be connected to the digital content management server 20 through the Internet may be used as the user terminal 10 of the present invention.

The user terminal 10 includes a module for accessing the digital content management server 20 and a module for uploading data.

The user accesses the digital content management server 20 through the user terminal 10 and uploads his or her own digital content, or stores, plays back, or stores his or her own content in the digital content management server 20. Can be deleted or modified from the list. Here, the digital content is meant to include multimedia data in various digital forms such as music, video, flash, etc. However, a music file will be described as an example for convenience of description.

The system for managing digital content online according to the present invention includes a feature extractor (not shown) for extracting audio feature values from a digital content uploaded and selected by a user and a feature matching unit 27 for performing matching between feature values. Each may be provided in the user terminal 10 or the digital content management server 20. In this embodiment, the feature extraction unit is installed in the user terminal 10, the feature matching unit 27 is an example that is installed in the digital content management server.

The feature extraction unit according to the present embodiment may be a web installation program, for example, an active access program. In detail, the feature extraction unit installed in the user terminal 10 uploads digital content stored in a recording medium that the user holds in his terminal by connecting to the digital content management server 20 or is connected to or inserted into the terminal. When the function is selected, feature values are extracted from audio data of the digital content uploaded and selected by the user and transmitted to the digital content management server 20 through the network 30.

Digital content management server 20 is to provide a digital content providing service of the present invention in conjunction with the user terminal 10 through the network 30, so that the corresponding function is made according to a request transmitted from the user terminal 10 It works.

As shown in FIG. 1, the digital content management server 20 includes a feature matching unit 21, a media storage unit 21, a feature database 23, a personal database 25, and a content management unit 29. do.

The media storage unit 21 is constructed of various digital contents as a database, and may include not only the content itself but also various content attribute information such as title, album name, play time, author, copyright holder, performer, producer, and the like. In addition, it may further include detailed information such as album images and lyrics.

The feature database 23 stores audio feature values for digital content stored in the media storage unit 21 and is used as comparison data that matches the audio feature values transmitted from the user terminal 10. The audio feature values stored in the feature database 23 are extracted and stored in a similar manner to the extraction method used by the feature extractor installed in the user terminal 10 described above.

The personal database 25 stores a personal possession list of contents held by each user. For example, various user information such as user personal information, a list of reserved contents, and history information may be stored. Such a private list is provided to the user as a web page, and the digital content management server may provide the user with various management functions for sorting, grouping, and editing the content list owned by the user.

The feature matching unit 27 matches the audio feature values transmitted from the user terminal 10 with the feature values stored in the feature database 23 to find matching content.

The content management unit 29 is for interworking with the user terminal 10 through the network 30 to manage the digital content of the present invention, and operates to perform a corresponding function according to a request transmitted from the user terminal 10. do.

In detail, the content manager 29 provides the user terminal 10 with detailed information about the matched digital content as a result of the matching of the feature matching unit 27, and corresponds to the user-owned content list of the personal database 25. Include information about the content.

Here, the detailed information about the content provided to the user may include attribute information such as title, album name, playing time, author, copyright holder, performer, producer, and the like. These details are summarized at a glance and provided to the user in the form of a list, through which the user can obtain various detailed information about the contents held by the user. For example, even if the content possessed by the user is a file whose attribute information or the like is damaged, the attribute information of the corresponding content can be easily known using the system of the present invention.

As described above, the user may access the digital content management server 20 to check a list of contents owned by the user, detailed information of the contents, and the like. The content manager 29 provides a management function according to the user's request. For example, the content manager 29 may provide various services such as playback, content download, album image download, and lyrics display.

For example, when a user accesses the content management unit 29 and selects a function of viewing a content list owned by the user, the content management unit 29 extracts a content retention list of the user stored in the personal database 25. A list of retained contents and detailed information thereof are provided to the user terminal 10. In addition, when the user selects a function of playing or downloading the contents included in the personal possession list, the selected contents may be extracted from the media storage unit 21 to be streamed or transmitted to the user terminal.

In this case, the data transmitted according to the download request preferably includes the file name, meta information, etc. of the corresponding content. For example, in the case of a music file type that can additionally store meta information of a file, such as an MP3 file, a song name, album name, artist name, issue year, track number, etc. are correctly input and downloaded.

As such, when using this system, the user can see the detailed information on the contents uploaded by him at a glance, and can receive streaming service anytime and anywhere if the user can access the digital content management server for the uploaded contents. Can be. In addition, for example, the music stored in the home PC is uploaded to the digital content management server 20, and then connected to the digital content management server 20 through another terminal, for example, an MP3, PDA, or a mobile phone. It can be downloaded to the file, and it is very beneficial to the user because it can receive the data including the file name and file property information when downloading.

For example, even if the digital content owned by the user is randomly changed in the file name or the attribute information is damaged, the system allows the user to download and play the digital content with the correct attribute information. This same sound source has the advantage of being able to download and play services.

In addition, the system of the present invention can provide the information of the copyright holder, it is also possible to settle the transaction between individuals based on such copyright information.

The above-described digital content management server 20 may be configured as a single physical device, each component may be configured as an independent module and these modules may operate in conjunction with a communication connection. For example, the content management unit 29 is implemented by a computer on which a web server program, a management program, or the like is installed, and the media storage unit 21, the feature database 23, and the personal database 25 are each a content management unit 29 and a personal computer 25. Can be implemented with interlocking DB server. In addition, the feature matching unit 27 may be implemented as a software program, and may be installed and executed on a server computer.

Hereinafter, the overall operation of the digital content management system of the present invention according to FIG. 1 will be described. FIG. 2 is a control flowchart of the digital content management system of FIG. 1.

 2, the user subscribes to the web site of the digital content management server 20 as a member in order to receive the service (S10). In addition, the digital content management server 20 installs a feature extraction unit, which is a program necessary for providing a service, to the user terminal 10 through the web (S11).

The user accesses the digital content management server 20 and selects an upload function provided by a web page, and is stored in a CD, DVD, memory device, etc., which is currently being stored in or connected to or inserted into his terminal. Upload of the digital content may be selected (S12).

In this way, when the user selects the upload function of the digital content, the feature extraction unit installed in the user terminal 10 is executed to extract feature values from the audio data of the uploaded digital content (S13). Meanwhile, the feature extracting unit may support a CD ripping function, and in this case, feature values are extracted after converting data stored in the CD inserted into the user terminal 10 into a file such as MP3.

The extracted feature values are transmitted to the digital content management server 20 through the network 30 (S14).

The feature matching unit 27 compares audio feature values transmitted from the user terminal 10 with feature values constructed in the feature database 23 (S15), and searches for matching digital content.

If the matching is successful (S16), the content manager 29 provides the detailed information of the matched digital content to the user terminal 10, and adds the corresponding content information to the user's own content list (S17).

As described above, detailed information (eg, song name, album name, artist, etc.) of the digital content that has been successfully matched may be formed as a list and provided to the user. In addition, the content list provided to the user may include various information such as title, author, performer, execution time, album jacket, and the like of the content.

However, if matching fails (S16), the content manager 29 requests the user terminal 10 to upload the entire data (S18), and the user terminal 10 sends the entire requested content to the content manager 29. The content manager 29 constructs the transmitted content as a database (S19). For example, if the uploaded data properly includes attribute information such as meta information, the digital content is included in the media storage unit 21, and the feature database 22 is extracted by extracting feature values from audio data of the digital content. ), And also add the information of the digital content to the personal database (23).

As described above, according to the present invention, in the case of the content that is not built into the database in the media storage unit 21, it can be built into the database using the content uploaded by the user, so that the number of users increases and the amount of content uploaded increases. This has the advantage of increasing the amount of DB.

3 is a schematic diagram of a method of providing a streaming service of digital content in the digital content management system of FIG. 1.

Referring to FIG. 3, when a user accesses the digital content management server 20 through the network 30 and logs in a user (S20), the content manager 29 provides a page of the logged in user (S21). A user may upload content stored in his or her user terminal 10 on his page, or play, store, delete, or share the uploaded contents with another user.

3 illustrates a method of playing content posted by the user. When the user selects a list of retained content in the web page (S22), the content manager 29 provides the corresponding content list to the user (S23). Here, the content list provided to the user may include various information such as title, author, performer, execution time, album jacket, and the like of the content.

The user may select playback of the desired digital content from the content list (S24), and when selected, the content manager 29 extracts the selected digital content from the media storage unit 21 and streams it to the user terminal 10 ( S25).

In addition, when the user selects the storage (download) of the desired content from the content list, the content manager 29 retrieves the content from the media storage unit 21 and downloads the content to the user terminal 10.

For example, if a user accesses the digital content management server 20 and uploads an 'A' music file, when the user later selects to download the 'A' music file from the content retention list, the file that he originally owned is Instead, the file held by the digital content management server 20 is downloaded. At this time, the downloaded file includes the attribute information of the content correctly, and the file name may also be downloaded under the normal name. For example, if a user uploads a music file that does not include such content information and then downloads the corresponding music file from the content retention list, the user can download a music file that normally contains content attribute information.

As another example, the digital content management server 20 according to an embodiment of the present invention may provide a content sharing service between users. When the user allows the user to share the content that he owns to other users, the digital content management server 20 provides the other users who want to share the detailed information of the corresponding content selected in the list, and the other user If the user wants to play or download the media storage unit 21 to search for the content to the streaming service or download to other users.

As described above, the digital content management system according to the present invention extracts the feature values from the content possessed by the user by using audio feature value extraction and matching technology of the content, and the user through the pre-built DB for matching successful content It shows the detailed information of the uploaded content to users so that they can stream or download the content at any time. In order to provide such a service, it is important to extract and match feature values of audio data quickly and accurately.

Hereinafter, in the digital content management system of the present invention, an audio feature value extraction method used in the feature extraction unit installed in the user terminal 10 or the digital content management server 20, and a feature database of the digital content management server 20 (23) An audio feature value extraction method used in construction will be described in detail.

4 is a flowchart illustrating a feature vector extraction method of audio data according to an embodiment of the present invention, and FIG. 5 illustrates an example of a waveform according to the processing of FIG. 4.

According to an exemplary embodiment of the present invention, a feature vector is extracted using only some data without using the entire audio data of the digital content. The reason for using some data is to increase the speed, which is possible due to the use of highly reliable sound source matching algorithm. In some cases, of course, you can use the entire data.

Referring to FIG. 4, one embodiment of the present invention uses data of about 20 seconds at the beginning of audio data (music file) (S30). Of course, the amount of data used can be appropriately changed depending on the embodiment. For example, it uses up to 20 seconds of data from an MP3 file compressed in time units.

Referring to Fig. 5, (a) shows the waveform of the input MP3 file, and (b) shows the waveform of the audio data after cutting the beginning 20 seconds in this MP3 file.

In an embodiment of the present invention, before extracting the feature vector, a preprocessing process of converting the start data of the acquired audio data into a predetermined format is performed. Even if the same sound source has different formats according to the encoding environment, it is possible to further improve the sound source matching performance by changing the various sound sources to a certain format before extracting the feature vector. The preprocessing process according to an embodiment of the present invention includes a series of processing processes such as constant bit conversion, decoding, normalization, starting point extraction and removal, and the like.

Specifically, referring to FIG. 5, first, the bit rate is changed so that audio data has a predetermined bit (S31). Since the number of feature outputs may be different when the bit rate is different even with the same sound source, the present invention is preprocessed to have a predetermined bit rate.

For example, the encoding bitrate of the audio data may vary from 32 Kbps, 64 Kbps, 128 Kbps, 192 Kbps, 329 Kbps, etc. In one embodiment of the present invention, the uniform change to 64 Kbps in consideration of the speed As an example. In FIG. 6, (c) shows a waveform after changing the bit rate to 64 Kbps.

Thereafter, the changed audio data to have a predetermined bit rate is decoded (S32). The format of the music file may vary, for example, WAVE, MP3, WMA, ASF, OGC, etc. In an embodiment of the present invention, the original PCM data is extracted by decoding the compressed file and based on the feature vector Extract In FIG. 5, (d) shows a waveform after decoding the MP3 file. At this time, the format of the file is changed from MP3 to WAV.

In this way, by preprocessing the audio data to have a predetermined format before extracting the feature vector, the feature vector extracted therefrom does not depend on the format or bitrate of the file, further improving the matching success rate.

The decoded audio data is normalized with respect to volume so that the feature can be extracted better (S33). FIG. 5E illustrates normalization of the volume of FIG. 5D. As shown in FIG. 5E, the overall waveform is increased in size.

After normalization, the format of the audio data is changed to mono (S34). If the audio data has multiple channels, the sound may be different for each channel, so change it to a mono channel. Fig. 5 (f) shows the waveform after changing the stereo format (e) to the mono format.

When the above process is completed, the start point where the sound starts from the audio data having a certain format is searched for (S35). Even in the same sound source, since the starting point of the music may be different according to the encoding environment, the sound source matching may be misaligned in this case. For example, in the case of digital music data, a fine but waveform change occurs every time the encoding process is performed. Such a deformation causes a shift in the time code, causing a shift in the time axis between the comparison sound sources, which causes difficulty in retrieving the same data.

Therefore, in the present invention, the matching success rate can be improved by searching for a starting point of music and then removing it (S36). FIG. 5G illustrates a search of a starting point in (e), and (h) illustrates a waveform of audio data after removing data before the searched starting point.

Hereinafter, a method of detecting a starting point of specific audio data according to an embodiment of the present invention will be described with reference to FIGS. 6 to 10.

The method for detecting a starting point according to an embodiment of the present invention divides PCM samples of preprocessed audio data into frame units, and examines whether a minimum waveform model exists for each frame to find a starting frame constituting a sound. At this time, in order not to be affected by the noise, the starting point can be found with an exception condition according to the characteristic of the noise.

Hereinafter, a minimum waveform model used for starting point detection will be described. Humans can hear sounds because the air vibrates and stimulates the auditory organs, and the vibration of the air forms waves. A wave is a phenomenon in which a vibration state of a medium at a point in the medium is periodically spread through the medium, and energy is cyclically converted to each other at a point in space. The period of the wave is generally spread like a sine wave, but the wave of music has an irregular period of wave because the sound and voice of various instruments are synthesized, and the energy of the wave represents the intensity of sound.

The minimum waveform model represents a half-period waveform having an energy value of one sign in a wave having an irregular period of music, and the energy value of a sign means negative and positive values based on zero points in the period of the waveform. The half-period waveform cannot be the minimum waveform model, and the minimum waveform model is required only when the number of samples forming the half-period waveform is larger than a certain number. Since the minimum number of samples that can form the minimum waveform model is represented by N.

6 is a graph of a minimum waveform model according to an embodiment of the present invention. Referring to FIG. 6, it can be seen that seven models having a half period larger than N are detected (N is set to 5 in this embodiment). The sample unit for measuring the number of waveforms is a sample of energy over time in order to convert a wave of analog music into a digital sound source. In the case of a CD sound source, it has 44100 samples per second. The minimum waveform model formed by the half-period waveform is used as the smallest unit of sound.

In one embodiment of the present invention, in order to increase the probability that the minimum waveform model is negative, the minimum waveform model inspection is not performed in units of samples but in units of frames that are bundles of samples. If it is executed in the sample unit, the starting point is determined by looking at only the first minimum waveform model. However, if it is performed in the frame unit, the probability of being negative is increased because all the minimum waveform models belonging to the frame are determined. In one embodiment of the present invention, the frame unit setting is applied to 20ms (880sample), which is widely used in the speech recognition field, as a unit capable of recognizing a sound to a minimum.

On the other hand, in the detection of the starting point, when noise is included in the digital sound source, the noise also has waves and energy constituting the sound, and thus there is a minimum waveform model. Therefore, a digital sound source containing noise has a problem of finding noise as a starting point.

In the present invention, in order to detect the starting point of music without being affected by noise, the characteristics of the waveform including the noise are classified and an exception condition using the classification is used to minimize the error.

7A to 7D show the characteristics of the noise waveform present in the digital sound source.

As shown in FIG. 7A, the noise waveform of FIG. 7A is impossible to identify noise. However, when the existence of the minimum waveform model is examined to detect the starting point, the starting point is detected at the silent portion before the sound comes out. This is because there is a low energy waveform in the silence that is hard to perceive by the ear. When measuring N, which determines the minimum waveform model, to avoid the impact of low-energy noise, do not include those samples in N counts if the samples are low-energy. The low sample criterion is less than 0.5% of total energy. The maximum energy of PCM data is 32767, and a reference value of less than about 0.5% can be set to 128 (2 ⁷ ).

Figure 7b is noise that occurs at a single point, with energy recognizable by the ear and not evenly distributed on the time axis. The noise having this characteristic is determined as a starting point only when the minimum waveform model is continuously detected by inspecting a plurality of frames in succession, instead of performing only one frame of the minimum waveform model. For example, in an embodiment of the present invention, when the minimum waveform model exists in three consecutive frames, the first frame of the three frames is determined as the start frame.

FIG. 7C illustrates a characteristic in which the ear has a perceivable energy, shows a feature that is elongated on the time axis, and the value of energy is weighted to a specific sign value. It can be seen that the noise waveform is constant without changing the sign value of the sample. Therefore, when the minimum waveform model is detected in consideration of this point, for example, if the number of model samples whose sign does not change is 1/3 or more of the frame length, it can be regarded as a noise waveform.

FIG. 7D has energy recognizable to the ear and is evenly distributed over the time axis. Noise with this characteristic is difficult to process because it is similar to the model of the waveform with the actual sound. However, this noise is often inserted as a part of the beginning depending on the music, in which case it will not be a problem because it can be viewed as a sound.

Hereinafter, an algorithm for detecting a starting point in a music file according to the starting point detecting principle described above will be described in detail. 8 is a flowchart illustrating a method of detecting a starting point according to an embodiment of the present invention. Table 1 below illustrates a frame inspection table.

Referring to FIG. 8 and Table 1, pre-processed PCM data is received (S40), and sequentially checked from the first frame (S41). If the minimum waveform model exists in the frame (S42), it writes to the frame inspection table (S43). When filling in, enter the minimum number of waveform models according to the location and sign of the sample. As described above, the minimum waveform model counts only samples having energy above a predetermined value. In the present embodiment, if five or more samples are counted by counting samples having energy of 128 or more, the minimum waveform model is detected as the minimum waveform model and entered into the frame inspection table. (I.e., noise processing in FIG. 7A).

If the minimum waveform model is not detected in the frame (S42), the frame inspection table is initialized (S44), and the next frame is examined (S45, S41).

On the other hand, the frame is continuously examined until the number of frames counted as the minimum waveform model exists in the frame inspection table is 3 (S46), that is, until the number of frames in which the minimum waveform model exists continuously is 3 (S46). (S45, S41) (ie, noise processing in FIG. 7B).

If 3 is satisfied, a ratio test is performed to determine whether the number of model samples whose sign does not change is less than 1/3 of the frame length (S47) (that is, the noise processing of FIG. 7C).

When the rate check is satisfied, the first frame position value of the frame check table becomes a music starting point (S48).

If the number of model samples whose sign does not change in the ratio check is 1/3 or more of the frame length even if the number of counted frames satisfies 3, the frame check table is initialized (S44), and the frame is checked again from the next frame ( S41).

As such, according to an embodiment of the present invention, since noise is considered together with the detection of a negative starting point, it is possible to minimize the possibility of error and have good performance.

In order to find out the performance of the aforementioned starting point detection algorithm by the number of samples per music genre, an experiment was performed, and the experimental results are shown in FIGS. 9 and 10. 9 is a performance measurement graph according to the number of N, Figure 10 is a performance measurement graph according to the music genre according to the present experiment.

The digital sound sources used in the experiments were sampled at 44100 Hz, 16 bits, and music was classified into 15 clusters as shown in <Table 2> to see if they affect the starting point detection according to genre and rhythm. In addition, 40 songs per group were used for a total of 600 songs.

In this experiment, the number of N to determine the minimum waveform model of music was set by number and the starting position measured for each number was detected in units of frames. The detected results were compared with the directly selected detection frame by creating a program that can represent music as a waveform picture and select each frame. In order to verify the detection performance of the starting point, the error range was one frame before and after, and if it was out of the error range, it was regarded as failure.

In FIG. 9, the horizontal axis represents the number N of samples that the minimum waveform model can have, and the vertical axis represents the coincidence rate compared with a directly selected detection frame. As shown in FIG. 9, the waveform consisting of one N number may not be regarded as a minimum waveform model, and a half-period waveform composed of at least two samples may be regarded as a minimum waveform model. The minimum waveform model with five samples was about 86%, which showed the best performance in the experiment, and after 6 the performance was getting worse. From the results of these experiments, it is preferable that the number of N that determines the model of the minimum waveform is determined between two and five.

10 is based on the performance measurement results according to the number of N was the best performance was measured by genre with N = 5. Genres that start smoothly, such as ballads and jazz, have a low match between 70% and 80%, while genres that consist of strong or bounced notes show high matches, but overall It has a relatively good performance.

The feature vector is extracted from the audio data from which the data before the starting point is removed through the above process. The feature vector includes a fast Fourier transform (FFT) of the data from which data before the start point is removed, a MEL filter bank of the fast Fourier transformed data, and a log conversion of the data performed by the mel filter bank. After performing the step, and the discrete cosine transform (DCT) of the log-transformed data, the feature vector is extracted. Each of these conversion processes is well known, and detailed descriptions thereof will be omitted.

However, the feature vector extraction process according to the present invention can extract the MFCC feature vector without any significant difference in performance without undergoing other processes such as windowing. In an embodiment of the present invention, a thirteenth order feature vector is used.

Hereinafter, a post-processing process of the extracted feature vector will be described with reference to FIG. 11.

11 is a flowchart illustrating a process of post-processing a feature vector according to an embodiment of the present invention. Referring to FIG. 11, the extracted 13 th order MFCC feature vectors are summed by 50 frames (S50). Since a 13 th order feature vector exists in one frame, a total of 13 * 50 = 650 feature vectors exist in 50 frames, and all of these feature vectors are summed to calculate a total of 81 sum data. For example, if the data before the starting point is deleted from the starting data of about 20 seconds, approximately 50 * 81 frames are generally left, and after combining feature vectors by 50 frames, a total of 81 sum data are calculated and used. do.

The difference value of the 81 sum data is calculated (S51), and the feature values of the sound source are determined. For example, when 81 sum data is Sum ₁ , Sum ₂ , Sum ₃ ,..., Sum ₈₁ , the difference between them may be defined as Sum _{i +1} − Sum _i . Since 81 aggregated data are used, a total of 80 difference values (feature values or feature data) are calculated. Here, in some cases, when much data before the starting point is cut off, there may be less than 81 aggregate data, and in this case, the difference value is set to zero.

The reason why the difference value is used as the feature value is that if the sum total of 81 pieces of data is used as it is, even if it is normalized, a difference in volume may occur. Therefore, the slope data, which is the difference value, is used as the final feature data.

The feature extractor installed in the user terminal 10 extracts feature values from some data of the music file uploaded and selected by the user through the above-described process, and transmits the feature values to the content manager 29 through the network 30. Meanwhile, feature values stored in the feature database 23 are also extracted through the above-described process.

The feature matching unit 27 tries to match audio feature values of the music file with a plurality of feature data stored in the feature database 23 through the network 30.

12 is a flowchart illustrating a process of performing sound source matching in the feature matching unit 27 according to an embodiment of the present invention. First, when metadata exists in a music file to be matched, the feature extractor transmits the metadata to the digital content management server 20, and the feature matcher 27 attempts to match using the metadata. . On the other hand, if the metadata does not exist in the music file, the feature values of the corresponding sound source are extracted through the above-described process and transmitted to the digital content management server 20 through the network 30.

Referring to FIG. 12, the feature matching unit 27 compares the extracted feature data with feature data stored in a database (S60) and adds difference values between the feature data (S61). For example, 80 feature data of the currently extracted sound source and 80 feature data of a plurality of sound sources stored in the database are compared and the difference values are summed.

Table 3 below is an exemplary table for explaining a result of determination of sound source matching.

In this example, the user has uploaded a "visible" of buzz on his terminal. The feature extracting unit extracts feature values from some data at the beginning of the "visible" of the buzz selected by the user, and transmits the feature values to the digital content management server 20. The feature matching unit 27 shows a result of attempting matching with the feature vectors of the plurality of sound sources stored in the feature database 23 based on this. In the table above, three songs with the minimum difference are listed. Next to the song name, 457.22, 2029.27, and 2162.09 are the sums of the difference values of the feature data between the song and the corresponding song. When matching sound sources, the above values are used rounded off to one decimal place.

The difference between the feature value of the song to be matched and the feature value of "visible" stored in the feature database 23 is the difference between Value1 = 457 and the difference between "lie" Value2 = 2029 and the "consideration" Assume that the value is Value3 = 2162, DIS1 = (Value2-Value1), DIS2 = (Value3-Value2), and DIS3 = abs (DIS2-DIS1).

Referring to FIG. 12, when DIS1 is larger than 5000 (S62), it is considered that matching fails (S63). In this case, since the difference is too large, it is determined that no music is matched.

On the other hand, when DIS1 is 100 or less (S64), it is considered that the matching is successful (S65). If the difference is 100 or less, it can be considered that almost matched, it is determined that the song is a match.

If DIS1 exceeds 100, but DIS3 is larger than 200 (S66), it is considered that the matching is successful (S65).

On the other hand, even when DIS1 is 0 (S67), the matching is considered to be successful (S65). This can often occur when a song is remade so that the same song exists in multiple song databases.

If this criterion is applied in Table 3, it corresponds to the case where DIS3 is greater than 200, and thus the case of successful matching. In other words, the song that attempted matching is determined as the visual of the buzz.

Accordingly, the content manager 29 provides detailed information regarding the 'visible' of the buzz to the user terminal 10, and adds the 'visible' of the buzz to the list of contents held by the user of the personal database 25. Subsequently, when the user selects a 'bounce' of the buzz from the content list to play or download the content, the content manager 29 streams or downloads the 'bounce' of the buzz stored in the media storage unit 23.

Hereinafter, a conventional example in which sound source matching is attempted without detecting a starting point and an example in which sound source matching is attempted after detecting a starting point according to the present invention will be described.

13A to 13C are examples of attempting sound source matching without detecting a starting point, where 13a is a waveform of a file (song name: happy giver, mantissa: sunflower) as a reference, and 13b is a waveform (file name: happiness) of a file to be compared. Giver, singer: sunflower), 13c shows a matching graph.

As can be seen intuitively through FIGS. 13A and 13B, it can be seen that the starting points of the two files are different. In addition, even in the matching result of FIG. 13C, the graphs are different from each other because the starting points of the two sound sources are different, and the sum of the difference between the reference feature values and the comparison feature values is 2576.56. As such, when a matching is attempted without detecting a starting point, even if the same sound source is used, the matching may be failed in many cases.

14A to 14C illustrate examples of attempting sound source matching after detecting a starting point according to the present invention. The music name is the same as that of Figs. 13A to 13C.

As shown in FIGS. 14A and 14B, it can be seen that a starting point is detected in two files to be matched. The result of attempting matching after removing the data before the starting point after detecting the starting point is shown in FIG. 14C.

As can be seen in FIG. 14C, the difference between the feature values is lowered to 488.58 since the starting point is detected to eliminate the staggering of the time axis.

As described above, according to the present invention, matching performance can be improved by converting a bit rate, a format, etc. of music into a predetermined format, detecting a starting point, and extracting feature vectors.

Table 4 shows the experimental data about the time required for the feature vector extraction and matching process according to an embodiment of the present invention.

In Table 4, time was represented to two decimal places. As can be seen from the above table, bit conversion, decoding, and feature vector extraction take only a little time, and other processes take little time, and it can be seen that a short time of 1.21 seconds is required to extract the feature vector. In addition, the experimental results show that feature vector extraction and matching is completed within an average of 2-3 seconds. This processing time is about a hundred times shorter than the conventional DTW method. In addition, it was confirmed that the matching success rate reached 97%.

As described above, the above-described feature extraction and matching technique has an effect of dramatically shortening the processing time and has a high matching success rate, which is very useful for the digital content management system of the present invention requiring real-time sound source matching.

Hereinafter, a description will be given of a system for managing digital content through online according to a second embodiment of the present invention.

In the above-described embodiment, the feature extraction unit has been described as being installed in the user terminal 10. However, in the second embodiment of the present invention, the feature extraction unit is installed in the digital content management server 20. FIG. That is, only a module for uploading data is installed in the user terminal 10, and when the user selects upload of his or her content stored in the terminal using the module, the corresponding content is transmitted to the digital content management server 20. do.

The feature extractor installed in the digital content management server 20 extracts the feature value, and the feature matcher performs feature value matching. Other parts are similar to the above-described embodiment, so a detailed description thereof will be omitted.

Hereinafter, a description will be given of a system for managing digital content through online according to a third embodiment of the present invention. In the third embodiment of the present invention, the feature extraction unit and the feature matching unit are installed in the user terminal 10. That is, when the user selects the upload of his or her content, the feature extractor installed in the user terminal 10 extracts the feature values of the corresponding content, and the feature matching unit works with the digital content management server 20 to perform feature value matching. Perform. The matching result is transmitted to the digital content management server 20 via the network 30. Other parts are similar to the above-described embodiment, so a detailed description thereof will be omitted.

The above-described preprocessing, feature value extraction and sound source matching processing can be implemented by a software algorithm for performing the above-described calculation and / or processing, and a plurality of processing steps are mentioned as the preprocessing process, but some of these steps will be omitted. Can be. In addition, although the specific criteria of sound source matching have been presented in the above-described embodiments, the application criteria may be changed.

In addition, in the above-described embodiment, the music file has been described as an example, but the present invention can be applied to video content in which audio data is inserted. In addition, when the present invention is applied to the image content, it is obvious to those skilled in the art that different standards from the above-described embodiments may be applied.

Although some embodiments of the invention have been shown and described, it will be apparent to those skilled in the art that modifications may be made to the embodiment without departing from the spirit or spirit of the invention. . It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

1 is a schematic diagram of a digital content management system via online according to a first embodiment of the present invention;

2 is a control flow diagram of the system of FIG.

3 is a schematic diagram of a method for providing a streaming service of digital content in the system of FIG. 1;

4 is a flowchart of a method for extracting feature vectors of audio data from digital content according to an embodiment of the present invention;

5 shows an example of a waveform according to the process of FIG. 4;

6 is a diagram for explaining a minimum waveform model used for starting point detection according to an embodiment of the present invention;

7A-7D illustrate features of noise waveforms included in audio data of digital content;

8 is a flowchart of a method for detecting a starting point according to an embodiment of the present invention;

9 is an example of a performance measurement graph according to the number of samples forming a minimum waveform model;

10 is an example of a performance measurement graph according to a music genre;

11 is a flowchart of a method for calculating feature data in accordance with an embodiment of the present invention;

12 is a flowchart of a method of matching a sound source according to an embodiment of the present invention;

13A to 13C are examples of sound source matching attempted without detecting a starting point;

14A to 14C illustrate examples of attempting sound source matching after detecting a starting point according to the present invention.

Claims (8)

In the digital content management system online, A media storage unit storing digital content; A feature database storing audio feature values for the digital content; A personal database storing a personal possession list of digital contents held by each user; A feature extractor for extracting feature values from audio data of the digital content uploaded and selected by the user;  A feature matching unit which searches the matched digital content by comparing the extracted feature values with the audio feature values stored in the feature database; And And a content manager to provide detailed information on the matched digital content to a user terminal and to include it in the personal possession list of the user. The method of claim 1, And a digital content management server interworking with the user terminal through the network. The media storage unit, the feature database, the personal database, and the content management unit is provided in the digital content management server, The feature extractor and the feature matcher are provided in any one of the user terminal and the digital content management server. The method according to claim 1 or 2, The detailed information of the digital content provided to the user terminal includes at least two of a title, artist name, producer, album name, play time, copyright holder, album image, lyrics. The method according to claim 1 or 2, The content management unit extracts and streams or transmits digital content corresponding to the request from the feature database in response to a user's request to play or download the digital content included in the personal possession list. And digital content transmitted in response to the download request normally includes a file name and attribute information. The method of claim 4, wherein When the matching fails, the content management unit receives the digital content uploaded and selected by the user and constructs a database into a database. The method of claim 1, The feature extracting unit changes and decodes some data corresponding to a predetermined playback time from the start time of reproduction to the predetermined playback time in the audio data, converts the decoded data into a mono format, and converts the data into the mono format. And processing a starting point at which a sound starts after normalizing the A, and extracting a feature vector from the data after the searched starting point. The method of claim 6, The feature extractor checks whether or not the minimum waveform model constituting the sound exists in the unit of normalized data in units of frames, and detects a predetermined number or more of the frames in which the minimum waveform model exists continuously. The first minimum waveform model of the extracted frames determines the detected position as a starting point; The minimum waveform model is a digital content management system through the on-line, characterized in that the number of samples having a predetermined value or more energy. The method of claim 7, wherein The feature extractor adds the extracted feature vector to a predetermined number of frames to calculate sum data, calculates difference values between the sum data, and outputs the sum values as the feature values. .

Images