KR20100007108A - System and method for online relaying digital contents sales - Google Patents

Abstract

PURPOSE: A system and a method for online relaying digital contents sales are provided to increase the efficiency by relaying the transaction between individuals with a DB that is constructed through an audio matching technique. CONSTITUTION: A sales relay management unit(26) stores matched digital contents with sales condition information in a sales database(24). The sales relay management unit provides a purchase with the sales information of the digital contents stored at the sales database. The sales relay management unit performs settlement depending on the purchase determination of the purchaser. The sales relay management unit extracts the determined digital contents from a media storage unit, and transmits the extracted digital contents to the purchase terminal.

Description

Sales relay system and method of digital contents online [SYSTEM AND METHOD FOR ONLINE RELAYING DIGITAL CONTENTS SALES}

The present invention relates to a sales relay system and method of digital content online.

Recently, due to the development of data communication such as the Internet or a mobile communication network, many people can easily access various multimedia data through the Internet or a mobile communication network. For example, there are a number of sites that sell, provide, or share music, movies, and the like online, and users can easily access the sites to obtain their desired multimedia data. The user may enjoy playing the multimedia content acquired online through the terminal.

The sale of digital content online is usually done by online operators, but transactions between individuals are also active. However, such transactions between individuals are usually made free of charge through sharing and the like, and illegal and indiscriminate copying is abundant, and the damages are being returned to copyright holders and record companies.

Therefore, in such a transaction between individuals, in order for the copyright holder to return a legitimate profit to the record business, etc., the transactions between individuals need to be made more transparent. For transparency in interpersonal transactions, online operators may consider relaying transactions between individuals.

On the other hand, in relaying the transactions of digital content between such individuals, the mediator has to store all the files that the individual wants to sell, and thus there is a burden on the storage space, especially when the file size is large or user access is large. The burden on transmission traffic can be further increased, which can be a business constraint.

Accordingly, an object of the present invention is to improve the efficiency by relaying transactions between individuals using a DB previously built through audio matching technology, and also to accurately know copyright information through matching of audio data of the contents, thereby to sell the contents. The present invention provides a system and method for relaying sales of digital content online that can secure the basis for the settlement of a royalty fee.

According to the present invention, there is provided a sales relay system for digital content online, comprising: a media storage unit for storing digital content; A feature database storing audio feature values for the digital content; A sales database in which sales information about digital content registered for sale by a user is stored; A feature extractor for extracting feature values from audio data of digital content uploaded and selected by the user for sale; A feature matching unit which searches the matched digital content by comparing the extracted feature values with the audio feature values stored in the feature database; And store the information on the matched digital content in the sales database together with the sales condition input by the user, provide the buyer with the sales information of the digital contents stored in the sales database, and pay according to the purchase decision of the buyer. And a sales relay management unit for extracting the determined digital content from the media storage unit and transmitting the purchased digital content to the purchaser's terminal. .

On the other hand, the above object, according to the present invention, the system for relaying sales of digital content online, comprising: a media storage unit for storing digital content; A feature database storing audio feature values for the digital content; A personal database storing a personal possession list of digital contents held by each user; A sales database in which sales information about digital content registered for sale by the user is stored; A feature extractor for extracting feature values from audio data of the digital content uploaded and selected by the user; A feature matching unit which searches the matched digital content by comparing the extracted feature values with the audio feature values stored in the feature database; And providing detailed information about the matched digital content to the user terminal while including the information in the personal possession list of the user and selecting the sale of the digital content included in the personal possession list. Information is stored in the sales database together with the sales conditions entered by the user, the sales information of the digital content stored in the sales database is provided to the buyer, and the payment is made according to the purchase decision of the buyer. It can be achieved by a sales relay system for digital content via online, characterized in that it comprises a sales relay management unit for including the determined digital content in the personal possession list of the buyer.

And a sales relay server interworking with the user terminal through the network, wherein the feature extraction unit and the feature matching unit are installed at any one of the user terminal and the sales relay server. The relay server may include the media storage unit, the feature database, the sales database, and the sales relay management unit.

Here, the feature extracting unit changes and decodes some data corresponding to a certain playback time from the start time of reproduction of the audio data of the digital content to have a predetermined bit, converts the decoded data into a mono format, and the mono After normalizing the data converted into a format, a starting point at which a sound starts may be searched, and the feature values may be extracted based on feature vectors extracted from the data after the searched starting point.

In addition, the feature extracting unit checks whether the minimum waveform model constituting the sound exists in the unit of the normalized data on a frame basis, and when a predetermined number or more frames of the minimum waveform model are continuously detected, A position where the first minimum waveform model is detected among the detected frames may be determined as the starting point, and the minimum waveform model may include a predetermined number of samples having an energy of a predetermined value or more.

The feature extractor may add summed feature vectors to a predetermined number of frames to calculate sum data, and determine difference values between the sum data as the feature value.

On the other hand, the above object is, according to the present invention, in the method for relaying sales of digital content online, the method comprising: building a database for the digital content; Constructing a feature database for feature values extracted from audio data of the digital content; Extracting feature values from audio data of the digital content uploaded and selected by the user for sale; Searching the matched digital content by comparing the extracted feature values with the audio feature values stored in the feature database; Storing the information on the matched digital content in a sales database together with the sales condition input by the user; Providing the buyer with sales information of the digital content stored in the sales database; And performing the payment according to the purchaser's purchase decision and extracting the purchased digital content from the database and transmitting the extracted digital content to the purchaser's terminal. Can be.

On the other hand, the above object is, according to the present invention, in the method for relaying sales of digital content online, the method comprising: building a database for the digital content; Constructing a feature database for feature values extracted from audio data of the digital content; Extracting feature values from audio data of the digital content uploaded and selected by the user; Searching the matched digital content by comparing the extracted feature values with the audio feature values stored in the feature database; Providing detailed information about the matched digital content to the user's terminal and adding the same to the user's personal list; When the user selects the sale of the digital content included in the personal possession list, storing the sale condition input by the user together with the information of the sale-selected digital content in a sales database; Providing the buyer with sales information of the digital content stored in the sales database; And performing payment according to the purchaser's purchase decision and including the digital content determined in the purchaser's personal possession list to be sold by the method for relaying digital content online. Can be.

The method may further include calculating a royalty fee to the copyright holder of the purchased digital content.

As described above, according to the present invention, it is possible to increase efficiency by relaying transactions between individuals using a DB previously built through audio matching technology, and also to accurately know copyright information through matching of audio data of contents. It is possible to secure the basis for the normal settlement of copyright fees for the sale of digital goods, and to secure the transparency in the transactions of digital contents online among individuals, and to relay the sales of digital contents through online which can lead a healthy trading culture. A system and method are provided.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings.

1 is a schematic control block diagram of a system for relaying sales of digital content online according to a first embodiment of the present invention. Referring to FIG. 1, a sales relay system for digital content online according to a first embodiment of the present invention includes a user terminal (10a, 10b, hereinafter referred to as 10), a sales relay server 20, and interworking between them. Network 30.

The user terminal 10 is a personal terminal for receiving a sales relay service of digital content between individuals according to the present invention. For example, various devices such as a personal PC, a notebook computer, a PDA, and an MP3 device may be used. In addition, any device that can be connected to the sales relay server 20 through the Internet can be used as the user terminal 10 of the present invention.

The user terminal 10 has a module that can access the sales relay server 20 and a module that can upload data.

A user may access the sales relay server 20 through the user terminal 10 and register to sell digital content owned by the user, or purchase content that others are selling. Here, the digital content is used to include multimedia data in various digital forms such as music, video, and flash. However, in the embodiment of the present invention, a music file will be described as an example for convenience of description.

According to the present invention, a system for relaying digital content online includes a feature extractor (not shown) for extracting audio feature values from a digital content uploaded and selected by a user, and a feature matcher 25 for performing matching between feature values. Includes, may be provided in the user terminal 10 or the sales relay server 20, respectively. In this embodiment, the feature extraction unit is installed in the user terminal 10, the feature matching unit 25 is an example that is installed in the sales relay server.

The feature extraction unit according to the present embodiment may be a web installation program, for example, an active access program.

Specifically, when the user accesses the sales relay server 20 and selects the upload function for the sale of the digital content, the feature extractor installed in the user terminal 10 extracts feature values from the audio data of the digital content uploaded and selected by the user. To the sales relay server 20 through the network (30).

As shown in FIG. 1, the sales relay server 20 includes a media storage unit 21, a feature database 22, a personal database 23, a sales database 24, a feature matching unit 25, and It includes a sales relay management unit 26.

The media storage unit 21 is constructed of various digital contents as a database, and may include not only the content itself but also various content attribute information such as title, album name, play time, author, copyright holder, performer, producer, and the like. In addition, it may further include detailed information such as album images and lyrics.

The feature database 22 stores audio feature values of digital content stored in the media storage unit 21 and is used as comparison data that matches the audio feature values transmitted from the user terminal 10. Audio feature values stored in the feature database 22 are extracted and stored in a similar manner to the extraction method used by the feature extractor installed in the user terminal 10 described above.

The personal database 23 stores content information of each user, and for example, various user information such as user personal information subscribed to the service, a list of contents held, sales and purchase history information, and the like may be stored.

The sales database 24 stores sales information on digital content registered by sellers for sale. The sales database 24 includes content information including sales information, titles, versions, authors, and playback time, and information on sales. And various other information.

The feature matching unit 25 searches for matching content by matching the audio feature values transmitted from the user terminal 10 with the feature values stored in the feature database 23.

The sales relay management unit 26 is for providing a sales relay service for the digital content of the present invention by interworking with the user terminal 10 through the network 30, and a corresponding function is made according to a request transmitted from the user terminal 10. It works to lose.

In detail, the sales relay manager 26 registers the content manager 29 in the sales database together with information on the matching result of the feature matching unit 27 and sales conditions in which the user inputs detailed information on the matched digital content. do. In this case, detailed information of the digital content matched to the user terminal may be transmitted, and the user may check whether the content is correct.

On the other hand, the user can access the online sales relay server of the digital content to obtain information about the content to be sold. For example, a user may input and search for a content name that he / she wants to purchase from among the sale products, and select and purchase a product having a desired condition from sellers who sell the corresponding content. The sales relay manager 26 supports the payment according to the purchase and the download function of the corresponding content. When the payment is completed, the sales relay manager 26 extracts the purchased digital content from the media storage 21 and transmits the purchased digital content to the user terminal 10 of the buyer. In this case, the transmitted data may include the attribute information of the corresponding content and be transmitted normally.

Such sales and purchase details are recorded in the sales database and the personal database 23, and information on the author or the copyright holder, etc., regarding the content in which the transaction is made also remains, and may be based on the settlement of the copyright fee.

As described above, since the present invention accurately knows copyright information through matching of audio data of contents, it is possible to secure a basis on which the settlement of copyright fees for the sale of contents can be normally performed, and thus the digital contents of online contents between individuals can be secured. We can secure transparency in trading and lead a healthy trading culture.

In addition, the sales agency according to the present invention is very advantageous in terms of cost as it can provide a relay service using a database built by itself without uploading and downloading the user's data using audio matching technology.

The aforementioned sales relay server 20 may be configured as a single physical device, each component may be configured as an independent module and these modules may operate in conjunction with a communication connection. For example, the sales relay management unit 26 is implemented by a computer on which a web program, a search program, a management program, etc. are installed, and the media storage unit 21, the feature database 22, the personal database 23, and the sales database 24 are provided. ) May be implemented as DB servers interworking with the sales relay management unit 26. In addition, the feature matching unit 25 may be implemented as a matching program.

Hereinafter, a method of relaying a sales relay server for digital content through online of FIG. 1 will be described. 2A and 2B are flowcharts illustrating a method of relaying a sales relay system for digital content online through FIG. 1.

 Referring to FIG. 2A, an on-line operator who wants to provide the system builds a media storage unit 21 for digital content and a feature database 22 for feature values extracted from audio data of digital content in advance (S10). , S11).

In order to receive the service, the user first subscribes as a member to the web site of the sales relay server 20 (as another example, it may be provided as a non-member-based service instead of a member-based service). The sales relay server 20 installs a user application program corresponding to the feature extraction unit required to provide a service to the user terminal 10 through the web (S12).

If the user wants to sell his own digital content, he selects the upload function provided on the web page of the online sales relay server 30, and is currently storing or connecting to or inserting the CD into his terminal. The upload of the digital content stored in the DVD, the memory device, or the like can be selected (S13).

When the user selects the upload function for the sale of the digital content, a feature extractor program installed in the user terminal 10 is executed to extract feature values from the audio data of the uploaded digital content (S14).

The extracted feature values are transmitted to the sales relay server 30 through the network 30 (S14).

The feature matching unit 25 compares audio feature values transmitted from the user terminal 10 with feature values constructed in the feature database 22 and searches for matching digital content (S15).

If the match is successful (S16), the sales relay management unit 26 receives the sales information such as the selling price from the user (S17). When the input is completed, the sales relay management unit 26 registers the information of the matched digital content in the sales database 24 together with the sales conditions input by the seller (S178. For example, the sales relay management unit 26). The user may provide a window for the user terminal 10 to input the sales conditions and the like. When the user inputs the sales amount and the like to prepare for the sale, the inputted sales information may be matched with the information of the matched digital content. The user registers with the sales database 24. The user can check whether the digital content uploaded by the sales relay server 30 is properly registered in the sales list.

In this case, even when the user uploaded digital content does not include attribute information such as metadata, the system of the present invention may provide the user with various attribute information of the corresponding content by using audio matching. In general, given that it is essential that product details should be provided to buyers at the time of product sale and purchase, the system of the present invention utilizes an audio matching technique to build a pre-built database. It is very useful in that it can provide attribute information of various contents.

However, if the matching fails (S16), the sales relay management unit 26 requests the user terminal 10 to upload the entire data (S19), and the user terminal 10 sends the entire requested content to the sales relay server 30. To send). The sales relay management unit 26, which has received the entire digital content data, constructs this as a database (S20).

For example, if the digital content includes attribute information such as meta information properly, the digital content is included in the media storage unit 21, and the feature database 22 is extracted by extracting feature values from audio data of the digital content. In addition to this, the sales information of the digital content is added to the sales database 24 (S21).

In addition, even if there is no attribute information in the digital content, if the correct attribute information is found later, the above database construction work is performed. However, if the information on the content can not be found, even if the effort is not added to the sales database 24 to inform the user of the fact that the sale is not possible.

For example, the user may be prevented from posting to the sales list for the content that has not been matched. If the matching is successful, the user may be notified to the sales list.

FIG. 2B describes the purchasing process of digital content following FIG. 2A.

When the user (buyer) accesses the website provided by the sales relay server 20 and selects digital content to purchase from the sales list (S21), the sales relay manager 26 proceeds with the payment process (S23). When the payment is completed (S24), the selected digital content is extracted from the media storage unit 21 and transmitted to the user terminal 10 of the purchaser (S25). In this case, the transmitted data may include attribute information of the corresponding content in addition to the purchased content. For example, in the case of a music file type that can additionally store meta information of a file, such as an MP3 file, it is possible to download data properly inputted with a song name, album name, artist name, issue year, track number, and the like.

On the other hand, the sales relay server 20 according to the present embodiment can provide a search service for the digital content that the buyer wants to purchase, and if you click on an item in the sales list displayed as a search result, the detailed information of the item is checked. You can provide services to help.

For example, if a buyer wants to purchase a "Sohot" music file of Wonder Girls, and enters "Sohot" into the search field, the sales relay management unit 26 displays the sales information registered in the sales database 24 for "Sohot". The search is made available to the buyer as a sales list, which allows the buyer to purchase content that meets the requirements.

Such a purchase fact is left as a record in the sales relay server 20, and the online operator may deposit the remaining amount after paying the copyright fee and the relay fee from the profit earned by the seller.

As described above, the method and system for relaying sales of digital content through online according to the present invention can guarantee a transparent transaction by relaying transactions between individuals, and can properly pay copyright fees to copyright holders or record companies. It has the advantage of establishing a trading order.

 Hereinafter, a sales relay system and method for digital content through online according to a second embodiment of the present invention will be described.

The system and method for relaying digital content through online according to the second embodiment of the present invention is similar to the above-described first embodiment, except that it additionally provides a content management service in addition to the sales relay.

The user uploads his or her digital contents to the sales relay server 20 through the network 30, and accesses the sales relay server 20 through the network 30 at any time to download and stream the contents. , You can get sales service. According to the second embodiment of the present invention, information on the (uploaded) digital content possessed by the user is constructed in the personal database 23 of FIG.

3A and 3B are flowcharts illustrating a method for relaying sales of digital content online according to a second embodiment of the present invention. Referring to FIG. 3A, the sales relay server 20 constructs a media storage unit 21 for digital content and a feature database 22 for feature values extracted from audio data of the digital content (S30 and S31). ).

When the user accesses the sales relay server 20 and selects the upload function provided by the web page, the user can store the CD, DVD, memory device, etc. currently stored in his terminal or connected or inserted into his terminal. The upload of the digital content can be selected (S32).

As such, when the user selects the upload function of the digital content (S33), the feature extractor program installed in the user terminal 10 is executed to extract feature values from the audio data of the uploaded digital content (S34). The extracted feature values are transmitted to the sales relay server 20 through the network 30 (S34).

The feature matching unit 25 compares audio feature values transmitted from the user terminal 10 with feature values constructed in the feature database 22 and searches for matching digital content (S35).

If the matching is successful (S36), the detailed information of the matched digital content is added to the user-owned content list and the detailed information is provided to the user terminal 10 (S37). Here, the content list provided to the user may include various information such as a file name, a title of the content, an author, a copyright holder, a performer, an execution time, an album image, and lyrics.

However, if matching fails (S36), the sales relay manager 26 requests the user terminal 10 to upload the entire data (S38), and the user terminal 10 sends the entire requested content to the sales relay server 20. To send). The sales relay management unit 26 builds the transmitted content into a database (S39).

With respect to the contents that have been matched through the above-described process and added to the user's retention list, the user can not only view detailed information about these contents but also play, download or sell the contents at any time. Can be.

Specifically, referring to FIG. 3B, when a user selects content to be sold from his or her content possession list (S40) and inputs a sale condition (S41), the sales relay management unit 26 selects the user to sell and sell. The information about the contents and the sales information are added to the sales database 24 (S42). Here, the information about the digital content registered in the sales database 24 and the selling information may be detailed information of the contents, information of a seller, a selling price, and the like, as described above.

On the other hand, when there is content that the user wants to purchase, the user may search for the corresponding content in the sales list or click the sale information of the corresponding content displayed on the web page to select the digital content to purchase (S43).

The sales relay management unit 26 proceeds with the payment procedure according to the purchase (S44), and when the payment is completed (S45), the detailed information of the content selected by the user is added to the purchaser's own content list (S46). Of course, this involves the settlement of copyright and relay fees according to the sales record and the payment of sales fees.

The purchaser can play, download and sell at any time since the content he purchased has been added to his or her own inventory.

For example, if the buyer selects the download of the content (S47), the sales relay manager 26 retrieves the content from the media storage unit 21 and transmits the content to the buyer's terminal (S48). In this case, the transmitted data may include various attribute information of the content.

  As described above, the sales relay system for digital content online according to the present invention extracts the feature values from the content possessed by the user by using the audio feature value extraction and matching technology of the content and builds in advance for the successful content. Enables download service of contents stored in one DB. In order to provide such a service, it is important to extract and match feature values of audio data quickly and accurately.

Hereinafter, in the digital content management system of the present invention, the audio feature value extraction method used in the feature extraction unit installed in the user terminal 10 or the sales relay server 20, and the feature database 22 of the sales relay server 20 The audio feature extraction method used in the construction will be described in detail.

4 is a flowchart illustrating a feature vector extraction method of audio data according to an embodiment of the present invention, and FIG. 5 illustrates an example of a waveform according to the processing of FIG. 4.

According to an exemplary embodiment of the present invention, a feature vector is extracted using only some data without using the entire audio data of the digital content. The reason for using some data is to increase the speed, which is possible due to the use of highly reliable sound source matching algorithm. In some cases, of course, you can use the entire data.

Referring to FIG. 4, one embodiment of the present invention uses data of about 20 seconds at the beginning of audio data (music file) (S50). Of course, the amount of data used can be appropriately changed depending on the embodiment. For example, it uses up to 20 seconds of data from an MP3 file compressed in time units.

Referring to Fig. 5, (a) shows the waveform of the input MP3 file, and (b) shows the waveform of the audio data after cutting the beginning 20 seconds in this MP3 file.

In an embodiment of the present invention, before extracting the feature vector, a preprocessing process of converting the start data of the acquired audio data into a predetermined format is performed. Even if the same sound source has different formats according to the encoding environment, it is possible to further improve the sound source matching performance by changing the various sound sources to a certain format before extracting the feature vector. The preprocessing process according to an embodiment of the present invention includes a series of processing processes such as constant bit conversion, decoding, normalization, starting point extraction and removal, and the like.

Specifically, referring to FIG. 4, first, the bit rate is changed so that the audio data has a predetermined bit (S51). Since the number of feature outputs may be different when the bit rate is different even with the same sound source, the present invention is preprocessed to have a predetermined bit rate.

For example, the encoding bitrate of the audio data may vary from 32 Kbps, 64 Kbps, 128 Kbps, 192 Kbps, 329 Kbps, etc. In one embodiment of the present invention, the uniform change to 64 Kbps in consideration of the speed As an example. In FIG. 5, (c) shows a waveform after changing the bit rate to 64 Kbps.

Thereafter, audio data changed to have a predetermined bitrate is decoded (S52). The format of the music file may vary, for example, WAVE, MP3, WMA, ASF, OGC, etc. In an embodiment of the present invention, the original PCM data is extracted by decoding the compressed file and based on the feature vector Extract In FIG. 5, (d) shows a waveform after decoding the MP3 file. At this time, the format of the file is changed from MP3 to WAV.

In this way, by preprocessing the audio data to have a predetermined format before extracting the feature vector, the feature vector extracted therefrom does not depend on the format or bitrate of the file, further improving the matching success rate.

In order to allow the feature to be extracted better, the decoded audio data is normalized with respect to the volume (S53). FIG. 5E illustrates normalization of the volume of FIG. 5D. As shown in FIG. 5E, the overall waveform is increased in size.

After normalization, the format of the audio data is changed to mono (S54). If the audio data has multiple channels, the sound may be different for each channel, so change it to a mono channel. Fig. 5 (f) shows the waveform after changing the stereo format (e) to the mono format.

When the above process is completed, the start point of the sound is searched for in the audio data having a certain format (S55). Even in the same sound source, since the starting point of the music may be different according to the encoding environment, the sound source matching may be misaligned in this case. For example, in the case of digital music data, a fine but waveform change occurs every time the encoding process is performed. Such a deformation causes a shift in the time code, causing a shift in the time axis between the comparison sound sources, which causes difficulty in retrieving the same data.

Therefore, in the present invention, by searching for the starting point of the music and then removing it (S56), it is possible to improve the matching success rate. FIG. 5G illustrates a search of a starting point in (e), and (h) illustrates a waveform of audio data after removing data before the searched starting point.

Hereinafter, a method of detecting a starting point of specific audio data according to an embodiment of the present invention will be described with reference to FIGS. 6 to 10.

The method for detecting a starting point according to an embodiment of the present invention divides PCM samples of preprocessed audio data into frame units, and examines whether a minimum waveform model exists for each frame to find a starting frame constituting a sound. At this time, in order not to be affected by the noise, the starting point can be found with an exception condition according to the characteristic of the noise.

Hereinafter, a minimum waveform model used for starting point detection will be described. Humans can hear sounds because the air vibrates and stimulates the auditory organs, and the vibration of the air forms waves. A wave is a phenomenon in which a vibration state of a medium at a point in the medium is periodically spread through the medium, and energy is cyclically converted to each other at a point in space. The period of the wave is generally spread like a sine wave, but the wave of music has an irregular period of wave because the sound and voice of various instruments are synthesized, and the energy of the wave represents the intensity of sound.

The minimum waveform model represents a half-period waveform having an energy value of one sign in a wave having an irregular period of music, and the energy value of a sign means negative and positive values based on zero points in the period of the waveform. The half-period waveform cannot be the minimum waveform model, and the minimum waveform model is required only when the number of samples forming the half-period waveform is larger than a certain number. Since the minimum number of samples that can form the minimum waveform model is represented by N.

6 is a graph of a minimum waveform model according to an embodiment of the present invention. Referring to FIG. 3, it can be seen that seven models having a half period larger than N are detected (in this embodiment, N is set to 5). The sample unit for measuring the number of waveforms is a sample of energy over time in order to convert a wave of analog music into a digital sound source. In the case of a CD sound source, it has 44100 samples per second. The minimum waveform model formed by the half-period waveform is used as the smallest unit of sound.

In one embodiment of the present invention, in order to increase the probability that the minimum waveform model is negative, the minimum waveform model inspection is not performed in units of samples but in units of frames that are bundles of samples. If it is executed in the sample unit, the starting point is determined by looking at the first one minimum waveform model. However, if it is performed in the unit of frame, the probability of being negative is increased because all the minimum waveform models belonging to the frame are determined. In one embodiment of the present invention, the frame unit setting is applied to 20ms (880sample), which is widely used in the speech recognition field, as a unit capable of recognizing a sound to a minimum.

On the other hand, in the detection of the starting point, when noise is included in the digital sound source, since the noise also has waves and energy constituting sound, there exists a minimum waveform model. Therefore, a digital sound source containing noise has a problem of finding noise as a starting point.

In the present invention, in order to detect the starting point of music without being affected by noise, the characteristics of the waveform including the noise are classified and an exception condition using the classification is used to minimize the error.

7A to 8D show the characteristics of the noise waveform present in the digital sound source.

As shown in FIG. 7A, the noise waveform of FIG. 7A is impossible to identify noise. However, when the existence of the minimum waveform model is examined to detect the starting point, the starting point is detected at the silent portion before the sound comes out. This is because there is a low energy waveform in the silence that is hard to perceive by the ear. When measuring N, which determines the minimum waveform model, to avoid the impact of low-energy noise, do not include those samples in N counts if the samples are low-energy. The low sample criterion is less than 0.5% of total energy. The peak energy of the PCM data is 32767 and a reference value of less than about 0.5% can be set to 128 (27).

Figure 7b is noise that occurs at a single point, with energy recognizable by the ear and not evenly distributed on the time axis. The noise having this characteristic is determined as a starting point only when the minimum waveform model is continuously detected by inspecting a plurality of frames in succession, instead of performing only one frame of the minimum waveform model. For example, in an embodiment of the present invention, when the minimum waveform model exists in three consecutive frames, the first frame of the three frames is determined as the start frame.

FIG. 7C illustrates a characteristic in which the ear has a perceivable energy, shows a feature that is elongated on the time axis, and the value of energy is weighted to a specific sign value. It can be seen that the noise waveform is constant without changing the sign value of the sample. Therefore, when the minimum waveform model is detected in consideration of this point, for example, if the number of model samples whose sign does not change is 1/3 or more of the frame length, it can be regarded as a noise waveform.

FIG. 7D has energy recognizable to the ear and is evenly distributed over the time axis. Noise with this characteristic is difficult to process because it is similar to the model of the waveform with the actual sound. However, this noise is often inserted as a part of the beginning according to the music, in which case it will not be a problem because it can be viewed as a sound.

Hereinafter, an algorithm for detecting a starting point in a music file according to the starting point detecting principle described above will be described in detail. 8 is a flowchart illustrating a method of detecting a starting point according to an embodiment of the present invention. Table 1 below illustrates a frame check table.

Referring to FIG. 8 and Table 1, pre-processed PCM data is received (S60) and sequentially inspected from the first frame (S61). If the minimum waveform model exists in the frame (S62), it writes to the frame inspection table (S63). When filling in, enter the minimum number of waveform models according to the location and sign of the sample. As described above, the minimum waveform model counts only samples having energy above a predetermined value. In the present embodiment, if five or more samples are counted by counting samples having energy of 128 or more, the minimum waveform model is detected as the minimum waveform model and entered into the frame inspection table. (I.e., noise processing in FIG. 7A).

If the minimum waveform model is not detected in the frame (S62), the frame inspection table is initialized (S64), and the next frame is examined (S65 and S61).

On the other hand, the frame is continuously examined until the number of frames counted as the minimum waveform model exists in the frame inspection table is 3 (S66), that is, until the number of frames in which the minimum waveform model exists continuously is 3 (S66). (S65, S61) (i.e., noise processing in Fig. 7B).

If 3 is satisfied, a ratio test is performed to determine whether the number of model samples whose sign does not change is less than 1/3 of the frame length (S67) (that is, the noise processing of FIG. 7C).

If the rate check is satisfied, the first frame position value of the frame check table becomes a music starting point (S68).

If the number of framed samples does not change in the ratio check even if the number of frames counted is 3 or more than 1/3 of the frame length, the frame check table is initialized (S64), and the frame is checked again from the next frame ( S61).

As such, according to an embodiment of the present invention, since noise is considered together with the detection of a negative starting point, it is possible to minimize the possibility of error and have good performance.

In order to find out the performance of the aforementioned starting point detection algorithm by the number of samples per music genre, an experiment was performed, and the experimental results are shown in FIGS. 9 and 10. 9 is a performance measurement graph according to the number of N, Figure 10 is a performance measurement graph according to the music genre according to the present experiment.

The digital sound sources used in the experiments were sampled at 44100 Hz, 16 bits, and music was classified into 15 clusters as shown in <Table 2> to see if they affect the starting point detection according to genre and rhythm. In addition, 40 songs per group were used for a total of 600 songs.

In this experiment, the number of N to determine the minimum waveform model of music was set by number and the starting position measured for each number was detected in units of frames. The detected results were compared with the directly selected detection frame by creating a program that can represent music as a waveform picture and select each frame. In order to verify the detection performance of the starting point, the error range was one frame before and after, and if it was out of the error range, it was regarded as failure.

In FIG. 9, the horizontal axis represents the number N of samples that the minimum waveform model can have, and the vertical axis represents the coincidence rate compared with a directly selected detection frame. As shown in FIG. 9, the waveform consisting of one N number may not be regarded as a minimum waveform model, and a half-period waveform composed of at least two samples may be regarded as a minimum waveform model. The minimum waveform model with five samples was about 86%, which showed the best performance in the experiment, and after 6 the performance was getting worse. From the results of these experiments, it is preferable that the number of N that determines the model of the minimum waveform is determined between two and five.

10 is based on the performance measurement results according to the number of N was the best performance was measured by genre with N = 5. Genres that start smoothly, such as ballads and jazz, have a low match between 70% and 80%, while genres that consist of strong or bounced notes show high matches, but overall It has a relatively good performance.

The feature vector is extracted from the audio data from which the data before the starting point is removed through the above process. The feature vector includes a fast Fourier transform (FFT) of the data from which data before the start point is removed, a MEL filter bank of the fast Fourier transformed data, and a log conversion of the data performed by the mel filter bank. After performing the step, and the discrete cosine transform (DCT) of the log-transformed data, the feature vector is extracted. Each of these conversion processes is well known, and detailed descriptions thereof will be omitted.

However, the feature vector extraction process according to the present invention can extract the MFCC feature vector without any significant difference in performance without undergoing other processes such as windowing. In an embodiment of the present invention, a thirteenth order feature vector is used.

Hereinafter, a post-processing process of the extracted feature vector will be described with reference to FIG. 11.

11 is a flowchart illustrating a process of post-processing a feature vector according to an embodiment of the present invention. Referring to FIG. 11, the extracted 13 th order MFCC feature vectors are summed by 50 frames (S70). Since a 13 th order feature vector exists in one frame, a total of 13 * 50 = 650 feature vectors exist in 50 frames, and all of these feature vectors are summed to calculate a total of 81 sum data. For example, if the data before the starting point is deleted from the starting data of about 20 seconds, approximately 50 * 81 frames are generally left, and after combining feature vectors by 50 frames, a total of 81 sum data are calculated and used. do.

The difference value of the 81 sum data is calculated (S71), and the feature values of the sound source are determined. For example, when 81 sum data is Sum ₁ , Sum ₂ , Sum ₃ ,..., Sum ₈₁ , the difference between them may be defined as Sum _{i +1} − Sum _i . Since 81 aggregated data are used, a total of 80 difference values (feature values or feature data) are calculated. Here, in some cases, when much data before the starting point is cut off, there may be less than 81 aggregate data, and in this case, the difference value is set to zero.

The reason why the difference value is used as the feature value is that if the sum total of 81 pieces of data is used as it is, even if it is normalized, a difference in volume may occur. Therefore, the slope data, which is the difference value, is used as the final feature data.

The feature extraction unit installed in the user terminal 10 extracts feature values from some data of the music file uploaded and selected by the user through the above-described process, and transmits the feature values to the sales relay server 20 through the network 30. Meanwhile, feature values stored in the feature database 22 are also extracted through the above-described process.

The feature matching unit 25 attempts to match audio feature values of the music file with a plurality of feature data stored in the feature database 22 through the network 30.

12 is a flowchart illustrating a process of performing sound source matching in the feature matching unit 25 according to an embodiment of the present invention. First, when metadata exists in a music file to be matched, the feature extracting unit transmits the corresponding metadata to the sales relay server 20, and the feature matching unit 25 attempts matching using the metadata. On the other hand, if no metadata exists in the music file, the feature values of the corresponding sound source are extracted through the above-described process and transmitted to the sales relay server 20 through the network 30.

Referring to FIG. 12, the extracted feature data is compared with feature data stored in the feature database 22 (S80), and a difference value between the feature data is added (S81). For example, 80 feature data of the currently extracted sound source and 80 feature data of a plurality of sound sources stored in the database are compared and the difference values are summed.

Table 3 below is an exemplary table for explaining a result of determination of sound source matching.

In this example, the user has uploaded a "visible" of buzz on his terminal. The feature extraction unit extracts feature values from some data at the beginning of the "visible" of the buzz selected by the user, and transmits the feature values to the sales relay server 20. Based on this, the feature matching unit 25 attempts to match feature vectors of a plurality of sound sources stored in the feature database 22 as a table. In the table above, three songs with the minimum difference are listed. Next to the song name, 457.22, 2029.27, and 2162.09 are the sums of the difference values of the feature data between the song and the corresponding song. When matching sound sources, the above values are used rounded off to one decimal place.

The difference between the feature value of the song to be matched and the feature value of "visible" stored in the feature database 22 is the difference between Value1 = 457 and the difference between "lie" Value2 = 2029 and the "consideration" Assume that the value is Value3 = 2162, DIS1 = (Value2-Value1), DIS2 = (Value3-Value2), and DIS3 = abs (DIS2-DIS1).

Referring to FIG. 12, when DIS1 is greater than 5000 (S82), it is considered that matching fails (S83). In this case, since the difference is too large, it is determined that no music is matched.

On the other hand, when DIS1 is 100 or less (S84), it is considered that the matching is successful (S85). If the difference is 100 or less, it can be considered that almost matched, it is determined that the song is a match.

If DIS1 exceeds 100, but DIS3 is larger than 200 (S86), it is considered that the matching is successful (S85).

On the other hand, even when DIS1 is 0 (S87), the matching is considered to be successful (S85). This can often occur when a song is remade so that the same song exists in multiple song databases.

If this criterion is applied in Table 3, it corresponds to the case where DIS3 is greater than 200, and thus the case of successful matching. In other words, the song that attempted matching is determined as the visual of the buzz.

Accordingly, the sales relay management unit 26 provides the user terminal 10 with detailed information regarding the 'visible' of the buzz, and adds the 'visible' of the buzz to the list of contents held by the user of the personal database 23. . Subsequently, when the user selects a 'bounce' of the content from the list of contents to be sold, played or downloaded, the sales relay manager 26 downloads or streams the 'bounce' of the buzz stored in the media storage 21. Service.

Hereinafter, a conventional example in which sound source matching is attempted without detecting a starting point and an example in which sound source matching is attempted after detecting a starting point according to the present invention will be described.

13A to 14C are examples of attempting a sound source matching without detecting a starting point, where 14a is a waveform of a file (a title: happiness giver, a mantissa: sunflower) as a reference, and 14b is a waveform of a file to be compared (song: happiness) Giver, singer: sunflower), 14c shows a matching graph.

As can be seen intuitively through FIGS. 13A and 14B, the starting points of the two files are different. In addition, even in the matching result of FIG. 13C, the graphs are different from each other because the starting points of the two sound sources are different, and the sum of the difference between the reference feature values and the comparison feature values is 2576.56. As such, when a matching is attempted without detecting a starting point, even if the same sound source is used, the matching may be failed in many cases.

14A to 15C illustrate examples of attempting sound source matching after detecting a starting point according to the present invention. The music name is the same as in Figs. 13A to 14C.

As shown in FIGS. 14A and 15B, it can be seen that a starting point is detected in two files to be matched. The result of attempting matching after removing the data before the starting point after detecting the starting point is shown in FIG. 14C.

As can be seen in FIG. 14C, the difference between the feature values is lowered to 488.58 since the starting point is detected to eliminate the staggering of the time axis.

As described above, according to the present invention, matching performance can be improved by converting a bit rate, a format, etc. of music into a predetermined format, detecting a starting point, and extracting feature vectors.

Table 4 shows the experimental data about the time required for the feature vector extraction and matching process according to an embodiment of the present invention.

In Table 4, time was represented to two decimal places. As can be seen from the above table, bit conversion, decoding, and feature vector extraction take only a little time, and other processes take little time, and it can be seen that a short time of 1.21 seconds is required to extract the feature vector. In addition, the experimental results show that feature vector extraction and matching is completed within an average of 2-3 seconds. This processing time is about a hundred times shorter than the conventional DTW method. In addition, it was confirmed that the matching success rate reached 97%.

As described above, the above-described feature extraction and matching technique has an effect of dramatically shortening the processing time and has a high matching success rate, which is very useful for the digital content management system of the present invention requiring real-time sound source matching.

Hereinafter, a description will be given of a system for managing digital content through online according to a third embodiment of the present invention.

In the above-described embodiment, the feature extraction unit has been described as being installed in the user terminal 10, but in the third embodiment of the present invention, it is installed in the sales relay server 20. That is, only a module for uploading data is installed in the user terminal 10. When the user selects upload of his or her content stored in the terminal using the module, the corresponding content is transmitted to the sales relay server 20. .

Then, the feature value is extracted from the feature extraction unit installed in the sales relay server 20, and the feature matching unit performs feature value matching. Other parts are similar to the above-described embodiment, so a detailed description thereof will be omitted.

Hereinafter, a description will be given of a system for managing digital content via online according to a fourth embodiment of the present invention. In the fourth embodiment of the present invention, the feature extraction unit and the feature matching unit are installed in the user terminal 10. That is, when the user selects the upload of his or her content, the feature extractor installed in the user terminal 10 extracts the feature values of the corresponding content, and the feature matching unit performs feature value matching in conjunction with the sales relay server 20. do. The matching result is transmitted to the sales relay server 20 through the network 30. Other parts are similar to the above-described embodiment, so a detailed description thereof will be omitted.

The above-described preprocessing, feature value extraction and sound source matching processing can be implemented by a software algorithm for performing the above-described calculation and / or processing, and a plurality of processing steps are mentioned as the preprocessing process, but some of these steps will be omitted. Can be. In addition, although the specific criteria of sound source matching have been presented in the above-described embodiments, the application criteria may be changed.

In addition, in the above-described embodiment, the music file has been described as an example, but the present invention can be applied to video content in which audio data is inserted. However, when the present invention is applied to the image content, it is obvious to those skilled in the art that different standards from the above-described embodiments may be applied.

Although some embodiments of the invention have been shown and described, it will be apparent to those skilled in the art that modifications may be made to the embodiment without departing from the spirit or spirit of the invention. . It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

1 is a schematic control block diagram of a system for relaying sales of digital content online according to a first embodiment of the present invention.

2A and 2B are flowcharts illustrating a method of relaying a sales relay system for digital content online through FIG. 1.

3A and 3B are flowcharts illustrating a method for relaying sales of digital content online according to a second embodiment of the present invention.

4 is a flowchart of a method of extracting a feature vector of audio data from digital content according to an embodiment of the present invention.

5 illustrates an example of a waveform according to the process of FIG. 4.

6 is a diagram for explaining a minimum waveform model used for detecting a starting point according to an embodiment of the present invention.

7A to 7D illustrate characteristics of noise waveforms included in audio data of digital content.

8 is a flowchart illustrating a method of detecting a starting point according to an embodiment of the present invention.

9 is an example of a performance measurement graph according to the number of samples forming a minimum waveform model.

10 is an example of a performance measurement graph according to a music genre.

11 is a flowchart of a method of calculating feature data according to an embodiment of the present invention.

12 is a flowchart of a method of matching sound sources according to an embodiment of the present invention.

13A to 14C illustrate examples of sound source matching without detecting a starting point.

14A to 15C illustrate examples of attempting sound source matching after detecting a starting point according to the present invention.

Claims (9)

In the sales relay system for digital content online, A media storage unit storing digital content; A feature database storing audio feature values for the digital content; A sales database in which sales information about digital content registered for sale by a user is stored; A feature extractor for extracting feature values from audio data of digital content uploaded and selected by the user for sale;  A feature matching unit which searches the matched digital content by comparing the extracted feature values with the audio feature values stored in the feature database; And The information about the matched digital content is stored in the sales database together with the sales condition input by the user, the sales information of the digital contents stored in the sales database is provided to the buyer, and the payment is made according to the purchase decision of the buyer. And a sales relay management unit which extracts the determined digital contents from the media storage unit and transmits the purchased digital contents to the terminal of the buyer. In the sales relay system for digital content online, A media storage unit storing digital content; A feature database storing audio feature values for the digital content; A personal database storing a personal possession list of digital contents held by each user; A sales database in which sales information about digital content registered for sale by the user is stored; A feature extractor for extracting feature values from audio data of the digital content uploaded and selected by the user;  A feature matching unit which searches the matched digital content by comparing the extracted feature values with the audio feature values stored in the feature database; And Providing detailed information about the matched digital content to the user terminal while including it in the personal possession list of the user, and when the user selects the sale of the digital content included in the personal possession list, the sale of the selected digital content is performed. Information is stored in the sales database together with the sales conditions entered by the user, the sales information of the digital content stored in the sales database is provided to the buyer, payment is made according to the purchase decision of the buyer, and the purchase decision is made. And a sales relay management unit for including the digital contents in the personal possession list of the buyer. The method according to claim 1 or 2, And a sales relay server interworking with the user terminal and the user terminal through a network. The feature extraction unit and the feature matching unit are installed in any one of the user terminal and the sales relay server, The sales relay server includes a media storage unit, the feature database, the sales database, the sales relay management unit, characterized in that the online sales of digital content. The method according to claim 1 or 2, The feature extracting unit changes and decodes some data corresponding to a predetermined playback time from the start time of reproduction of the audio data of the digital content to have a predetermined bit, converts the decoded data into a mono format, and converts the data into the mono format. After normalizing the converted data, search for a starting point where sound starts, and extract the feature values based on feature vectors extracted from the data after the searched starting point. Sales relay system. The method of claim 4, wherein The feature extractor checks whether or not the minimum waveform model constituting the sound exists in the unit of normalized data in units of frames, and detects a predetermined number or more of the frames in which the minimum waveform model exists continuously. The first minimum waveform model of the extracted frames is determined as the starting point, The minimum waveform model is a sales relay system for digital content online, characterized in that the number of samples having a predetermined value or more energy. The method of claim 5, And the feature extracting unit calculates summed data by adding the extracted feature vectors for a predetermined number of frames, and determines difference values between the summed data as the feature values. In the method of relaying sales of digital content online, Building a database for digital content; Constructing a feature database for feature values extracted from audio data of the digital content; Extracting feature values from audio data of the digital content uploaded and selected by the user for sale;  Searching the matched digital content by comparing the extracted feature values with the audio feature values stored in the feature database; Storing the information on the matched digital content in a sales database together with the sales condition input by the user; Providing the buyer with sales information of the digital content stored in the sales database; And And performing payment according to the purchaser's purchase decision and extracting the purchased decision digital content from the database and transmitting the extracted digital content to the purchaser's terminal. In the method of relaying sales of digital content online, Building a database for digital content; Constructing a feature database for feature values extracted from audio data of the digital content; Extracting feature values from audio data of the digital content uploaded and selected by the user;  Searching the matched digital content by comparing the extracted feature values with the audio feature values stored in the feature database; Providing detailed information about the matched digital content to the user's terminal and adding the same to the user's personal list; When the user selects the sale of the digital content included in the personal possession list, storing the sale condition input by the user together with the information of the sale-selected digital content in a sales database; Providing the buyer with sales information of the digital content stored in the sales database; And And performing payment according to the purchaser's purchase decision and including the purchase-determined digital content in the purchaser's personal possession list. The method according to claim 7 or 8, The method of claim 1, further comprising the step of calculating a royalty fee to the copyright holder of the purchased digital content.

Images