KR100702059B1 - Ubiquitous music information retrieval system and method based on query pool with feedback of customer characteristics - Google Patents

Abstract

The present invention operates a query pool for a customer's query input through a customer terminal frame connected to a wired or wireless network, so that the query pool is searched first, and the customer is searched for from a plurality of candidate music presented as a search result. The purpose of the present invention is to improve the efficiency of the search for new customer queries by performing a matching analysis on the original query input music and then applying feedback to the music identified as the correct answer. By analyzing the error existing between the music entered by the customer and the music identified by the correct answer, the customer is asked to identify the characteristics of the query so that the search can be made in consideration of the characteristics of the customer for the same customer's query. In addition, it is possible to perform the search more efficiently by considering the exposure environment of the customer according to the location information of the customer provided by the location-based service in the music information search. In addition, you can search music by embedding both search engine and database by using the terminal itself, or when there is no separate space, equipment or facility for music only, you can compose or play music, practice listening to music, and practice using only your terminal. This suggests a possible way.

Query pool, weights, matching, humming, speech recognition, melody, music information retrieval, location-based, ubiquitous, composition

Description

Ubiquitous music information retrieval system and method based on query pool with feedback of customer characteristics}

1 is a block diagram of a music information retrieval system according to the present invention;

2 is a flowchart of a music information retrieval system and method according to the present invention;

3 is a flow chart of matching analysis and weight adjustment feedback.

[Description of Symbols for Main Parts of Drawing]

100. Customer's terminal

200. Music Search Server

300. query pool

400. Music File Database

500. Customer Database

600. Instrument Database

Recently, with the development of internet and communication technology, the increase of digital music contents, network speed, and the development of music content compression technology have explosively expanded the use and retrieval of music information. It is true. Most digital music contents are systematically cataloged by title, genre, lyricist, composer and lyrics. However, if the user cannot know the contents of the list in detail, it is not easy to search for the desired music, and if the user mistakenly inputs the contents of the list by mistake, the user cannot accurately search. Preferably, search through randomly heard music, user's hum, song, etc. is not very dependent on user's knowledge, but only through music melody or part of music. This is a required skill. Some of the technologies that have been put to practical use are simply pronounced by voice, or recognized by listening to music stored on a CD. However, there is still a general solution for the case where mixed musical elements are mixed. It is not.

The present invention proposes a new music information retrieval system and method to solve the above problems. Queries input through the terminal of the customer may take various forms. The customer may simply be humming, a song with lyrics, or an orchestral accompaniment. In the present invention, this type of query input is divided into speech and melody, and the melody is divided into main melody and sub melody and used for information retrieval. do. In addition, once the music that the customer has queried is searched and confirmed by the customer, it analyzes the mutual similarity between the music of the query and the music corresponding to the previously stored correct answer and assigns a weighted value to the feedback to the query pool. It suggests how to feedback. In addition, by using the customer's location information to improve the efficiency of the search by using the customer's query environment for the search. In the present invention, the music search order is to search the query pool first, and then search the entire music file database if necessary.

The present invention has been conceived that when a customer searches for music, most of the time, the most characteristic part of the song he or she is looking for is repeatedly inputted in a similar way and that the customer's query generally has a certain pattern or characteristic. It has a pool that stores the customer's query every time, and matches the music first asked by the customer for the music selected or finalized by the customer for the singular or plural candidate results finally presented as a search result. . As a result, a matching weight is changed to reflect the query characteristics of the currently input music, and the feedback is fed back to the query pool. That is, since the song selected by the customer corresponds to the correct answer, it is assumed that the current customer question input is the sample closest to the correct answer. It is to maintain the popular preference for the query pool by analyzing the music characteristics of the corresponding pitch, tone, and beat by customer, and reflecting the matching weight in each search from the results obtained from the matching analysis. .

Hereinafter, with reference to the drawings in more detail, the drawings are used for illustration only, the scope of the present invention is not limited to these drawings.

1 is a system configuration diagram of a music search system and method according to the present invention.

Various types of music are input by a query through the terminal 100 of the customer. At this time, the form of music is analog sound waves of all kinds, such as human development, voice, musical instrument performance, song, orchestra accompaniment. The customer's terminal refers to any type of terminal that can be connected to wired / wireless internet network, mobile communication network, digital broadcasting network such as PDA, PSP, MP3P, PMP, notebook, PC, DTV as well as mobile phone. The customer terminal is connected to the music search server 200 through a network, and the music search server is connected to the query pool 300, the music file database 400, the customer database 500, and the instrument database 600. The instrument database 600 classifies and stores information related to natural frequencies and tones of various instruments. In addition, recently, the performance of the terminal has been rapidly developed, so that the search engine can be installed in the form of a virtual machine (VM), applet, or software program on the terminal itself, and all database data related to the search are also mounted in the terminal. It is also possible to operate the music information retrieval system proposed in the present invention in the form of a terminal stand-alone server (through alone) without going through.

2 is a flowchart illustrating a music search system and method according to the present invention.

First, after accessing the music search server 200 through the terminal 100 of the customer, after the location information of the customer is confirmed by the server, various types of music are input through the customer terminal 100. . The various forms of music refer to all forms of comprehensive music such as voice, humming, song, performance, orchestra, and the like. That is, any music can be input by query. Since the input query is an analog sound wave, it is processed as a digital signal. The original original digital data signal-processed is stored in the original sound database DB of the query pool 300. The query input signal is input to the speech recognition module, and only a speech is separated and signal processed, and stored in the voice DB of the query pool. The signal passing through the speech recognition module is input to the melody separation module and divided into a main melody and a sub melody and stored in the main melody DB and the subordinate melody DB of the query pool, respectively. Each database stored in the query pool contains music information such as title, lyrics, songwriter, composer, and genre, and music characteristic information such as tone, pitch, beat, and melody in the format of MIDI. . Therefore, a desired song can be retrieved using any one or a plurality of pieces of information such as voice information, main melody information, or mixed original sound information. Whenever a query is entered, the query pool 300 always accumulates the query, and searches this information first to terminate the search immediately when a song to be found by the customer is found. In this case, the entire music file database 400 is additionally searched. For example, if a customer sings "Ari La Ang" for the first measure as a humming song with the title "Arirang", it is immediately stored in the query pool, which is then the top priority for future searches. It will be the target DB, and if a new customer later sings the title “Arirang” again with a different tone or beat, he will probably call “Ari La Ang” again. In that case, the second customer's input is likewise stored in the query pool to form a new database. In this case, only some weights related to the tone or the beat are adjusted. If the second customer enters a query that says, “Bake me ~~~”, this is the data from another query pool for the title “Arirang” song. The reason why the query pool is operated is that the query entered by the customer tends to be limited to a few popular well-known phrases. By excluding them, the search speed is greatly improved. In addition, the singing error existing between the correct answer and the query can be controlled as a weight as the data is accumulated eventually, so that a quick search for similar query input can be performed even if the song is not the correct song. However, if the customer's query is a new input that does not exist in the query pool, it is inevitable that the entire music file DB 400 must be searched for.

Here, the query pool or various databases do not have to exist on the network, but it is also possible to be included in the customer's terminal and operate as a stand alone server of the customer's own terminal. In addition, the list of results retrieved from the search server is presented on the client's terminal in the form of a short message (SMS) or multimedia message (MMS), so that the customer selects a desired song from the list and returns it to the server. It is also possible to do so.

3 is a flow chart for matching analysis and weighted adjustment feedback in accordance with the present invention.

The music that the customer has entered into the query is called function F, and the music that the customer completes after confirming the answer is called the function G. Functions F and G are composed of a combination of N musical characteristic values, and each musical characteristic value may be a tone, a pitch, a beat, a pitch, and the like. In this case, it is not important what function the musical characteristic values make up each music, and the relationship does not necessarily need to be mathematical, but it can be viewed as a function in the concept of correlation that individual musical characteristic values are gathered together to form a complete music. have. For most searches, F and G cannot be exactly the same and to some extent have similar values. For example, if a customer calls a measure of a song and enters a query, and the search system finds the correct answer based on that input, the error in the matching between the query input and the correct answer is one or more times. Accurate calculation can be performed by confirming the correct answer song from a plurality of candidate search lists. If the pitch is almost identical between the song entered by the customer and the song with the correct answer, but it is assumed that the beat has many differences, it can be inferred that the pitch value should be reflected in the search with a higher weight than the beat value. . In other words, the customer inputs the pitch relatively accurately, but the beat is generally incorrectly input. Therefore, when the same customer inputs a new song, the correspondence of the pitch of the song is greater than that of the beat. Just search for it. Such a method is not always applied to all types of query music input to the terminal of the customer, but at least the weight may be applied to the input form such as the development or humming of the customer. As a result, the weight of the pitch is higher for new query inputs, and the search efficiency can be improved by analyzing and reflecting the propensity of the same customer. In addition, according to the present invention, the environment in which the customer inputs a query can always be grasped by the music search server, so that it is possible to classify and apply various elements related to the query input such as noise characteristics according to the customer environment for each customer. In other words, if a query is repeatedly input at the same place as a query input in the past, the environmental part of noise factors considered when searching for music information can be removed in advance, thereby further improving the speed of the search. Can be imported.

FIG. 4 exemplarily illustrates a matching error between the functions described with reference to FIG. 3.

In the figure, the pitch, beat, tone, emotion, and pronunciation of the music entered by the customer are converted into music characteristic values. There is an error for each characteristic between the query music F and the correct answer music G. In this example, the pronunciation part is largely different, and thus the weighting factor of the pronunciation may be relatively low when searching for the customer.

The invention described so far can be deduced that the rights are recognized to a far greater extent which is technically applicable.

As described above, the present invention introduces a ubiquitous music information retrieval system and method for retrieving music whenever and wherever a customer wants with a terminal possessed by a customer. According to the present invention, by analyzing the degree of matching between the music input by the customer and the final search result music is grasped the taste or characteristics of the customer's music query, and based on this weighted or updated for each music characteristic value is fed back In the future, the query of new customer is reflected to customer's characteristics, so that the search and update of customer's query pool is updated first, thereby improving the efficiency and speed of the search. In addition, the terminal location information of the customer is transmitted to the search server to further consider the customer's query input environment, so that the search can be much easier in the signal processing or music search for additional query input. Application of this, in the case of ubiquitous karaoke or listening to music using a mobile terminal, it will be very useful when a customer searches for a song through the terminal. In addition, if there is no separate space for exclusive music, equipment, or facilities, it is possible to perform compositions, performances by listening to music, practice listening to music, and practice music using only the terminal of the customer. The ripple effect in the edutainment industry is expected to be even greater.

Claims (16)

Connecting the terminal of the customer connected to the network to the music search server; Checking, by the server, location information of the customer; Inputting a query of music to be searched into the terminal of the customer; Signal processing the query music input to the terminal of the customer; The signal processing is stored in an original database of a query pool; Inputting the query input to a speech recognition module; Separating voice data from the voice recognition module and inputting the voice data into a voice database of a query pool; Inputting to the melody separation module by the speech recognition module; Separating the main melody in the melody separation module and inputting the main melody in the query pool; Inputting to the auxiliary melody database of the query pool in the melody separation module; Searching after the query pool is completed; Further searching the query pool and music file database; Outputting the search result to the terminal of the customer; Confirming a final answer among the search results by the customer and completing a search; Analyzing a match between the result confirmed by the customer as a correct answer and the query input; Updating weights according to the matching analysis and feeding feedback to the query pool; Query pool based ubiquitous music information retrieval method that reflects the customer characteristics characterized in that consisting of. The method of claim 1, Network means, A query pool-based ubiquitous music information retrieval method in which customer characteristics are reflected in feedback, including wired / wireless internet network, mobile communication network, wired / wireless communication network, analog broadcasting network or digital broadcasting network. The method of claim 1, A query pool is a A database space in which a query input to a terminal of a customer is signaled and stored, It consists of original query DB, speech DB, main melody DB, sub melody DB, The file format of each DB further comprises a SMAF (.mmf) or MIDI (MIDI) format, etc. The query pool based ubiquitous music information retrieval method that reflects customer characteristics. The method according to claim 1 or 3, The main melody is In the sound of the analog sound wave type that the human voice and the machine playing sound are mixed and input into the query, A query pool based ubiquitous music information retrieval method in which a customer characteristic is reflected and reflected as a representative and fundamental melody of the music, which can be distinguished from other music only by the melody alone. The method of claim 1, Speech recognition module means This module is responsible for recognizing and separating human voices from the input music. A query pool based ubiquitous music information retrieval method that reflects customer characteristics characterized by speaker-independent recognition, continuous word recognition, dialogue recognition, key word recognition, or spontaneous speech recognition. The method of claim 1, Matching analysis means If one of your candidates outputs as a result of a search, and the customer identified the correct song as the correct answer, This is a comparative analysis of the similarity, or similarity, between the songs identified by the correct answer and the songs input by the first customer. More specifically, the similarities of individual components of the characteristics of the music, such as tone, pitch, beat, and pitch. Query pool based ubiquitous music information retrieval method that reflects the customer characteristics, characterized in that the comparative analysis. The method of claim 1, The weighting value means: There is a functional relationship between the music identified by the correct answer or the music inputted by the first customer and the individual component of the music characteristic, so that a numerical value, that is, a weight, is used to quantify the interrelationship between the music and the music characteristic, so that the weight is used for the music search. Query pool based ubiquitous music information retrieval method that reflects customer characteristics characterized in that the feedback. The method of claim 1, The terminal of the customer, Customer characteristics, which include both wireless terminals such as mobile phones, PDAs, PMPs, PMCs, PSPs, MP3Ps, laptops, Internet video phones, game machines such as X-Box, and wired terminals such as audio and video related electronic appliances. Query pool based ubiquitous music information retrieval method reflected in this feedback. The method of claim 1, The step of the server confirms the location information of the customer, The location of the terminal of the customer is automatically recognized by the server by the location-based service (LBS) or through the method of sending the information related to the location of the user directly to the server through the terminal, the query input for music search ( A query pool-based ubiquitous music information retrieval method further comprising reflecting customer characteristics, wherein the music search server further includes considering a customer's environment variable by the customer's location information. The method of claim 1, The customer's terminal is equipped with a virtual machine (VM) or applet or software or program for music search and has some or all of a query pool for storing the customer's query in the terminal itself. It can be equipped with a customer characteristic of the customer characterized in that it further comprises the first search for the terminal of the customer for the customer's query, and then send only the search results to the music search server, the final search of the music file database on the server A query pool based ubiquitous music information retrieval method. The method of claim 1, The step of outputting the search result to the terminal of the customer, Presenting the search result list in the form of a short message (SMS) or a multimedia message (MMS) on a terminal of a customer; Selecting a desired song from the list by the customer to return or call back to the server; Query pool-based ubiquitous music information retrieval method that reflects the customer characteristics characterized in that it further comprises. If the customer's terminal is not connected to the network, The virtual machine (VM) or applet (Applet) or software or program for music search is installed in the terminal itself, The terminal itself may be provided with both a query pool, a music file database and a customer database, Inputting a query of music to be searched into the terminal of the customer; Signal processing the query music input to the terminal of the customer; The signal processing is stored in an original database of a query pool; Inputting the query input to a speech recognition module; Separating voice data from the voice recognition module and inputting the voice data into a voice database of a query pool; Inputting to the melody separation module by the speech recognition module; Separating the main melody in the melody separation module and inputting the main melody in the query pool; Inputting to the auxiliary melody database of the query pool in the melody separation module; Searching after the query pool is completed; Further searching the query pool and music file database; Outputting the search result to the terminal of the customer; Confirming a final answer among the search results by the customer and completing a search; Analyzing a match between the result confirmed by the customer as a correct answer and the query input; Updating weights according to the matching analysis and feeding feedback to the query pool; Query pool based ubiquitous music information retrieval method that reflects the customer characteristics characterized in that consisting of. The method of claim 1 or 12, Extracting a query in the form of a sound wave and a main melody and converting it into a format such as MIDI or SMAF (.mmf); Comparing whether or not the main tune converted to the MIDI or SMAF (.mmf) and the like match the data with the MIDI or SMAF (.mmf) or the like on an existing database; Presenting the found song on the terminal screen after matching comparison; Selecting and selecting a plurality of songs according to matching accuracy or similarity order after the matching comparison; And Reproducing the song selected by the customer; Query pool based ubiquitous music information retrieval method that reflects the customer characteristics characterized in that it comprises a further comprising. The method of claim 1 or 12, Extracting a main melody by a query of a customer; Outputting the extracted main melody as a main melody score in the form of a musical note and a wrong paper on the terminal; Selecting the output main tune score and the musical instrument to be played by a customer from a musical instrument database; Playing and reproducing the main melody score with the selected musical instrument; Selecting by the customer from among a plurality of music candidates searched by the customer's query; Comparing the music selected by the customer with the music queried by the customer to show the similarity; and further comprising: enabling to perform the composition or the composition according to the composition. Pool based ubiquitous music information retrieval method. The method of claim 1 or 12, At the same time or at the same time as listening to music played through the customer's terminal, The music score is displayed directly in the form of musical notes on the screen of the terminal, which is processed graphically, and the notes and beats are displayed by the terminal button, or the musical score data on the displayed score is corrected or changed, or the listening function is performed. Query pool based ubiquitous music information retrieval method that reflects the customer characteristics characterized in that it further comprises a possible. The method of claim 1 or 12, The sheet music formed in the form of lyrics or notes drawn on the line paper can be output to the terminal of the customer, When the music score is shown to the customer, the query pool based ubiquitous music information retrieval method reflecting customer characteristics, characterized in that it is configured to be viewed by the customer according to the pitch and beat.

Images