KR100378240B1 - Method for re-adjusting ranking of document to use user's profile and entropy - Google Patents

Abstract

The present invention relates to a document ranking method using an entropy and a user profile to improve the accuracy of a search in an information retrieval system by using an entropy value and a user profile according to probability information between a user query word and subject words of each document. .

Meanwhile, the present invention uses a mutual information amount using air information between words to calculate an entropy value, and combines a weighting method using Bayesian learning with a weight calculation method based on inverse literature frequency when calculating an entropy value. The document ranking method is applied by applying entropy and user profile to achieve the improvement of accuracy.

On the other hand, the present invention is designed to entropy by designing a user profile consisting of the keywords used in the conventional search and the frequency of use thereof in order to reflect the recent interest, search behavior and semantic information of the user query words of users using the information retrieval system In order to reflect the calculation, and in the actual entropy calculation, only a set number of queries having the highest frequency among previous queries in the user profile participate in the calculation in order to suppress the delay of the search time due to the complexity of time. The ranking of documents relates to a document ranking adjustment method using entropy and a user profile that causes the total entropy to be rearranged in increasing order.

Description

Method for re-adjusting ranking of document to use user's profile and entropy}

The present invention relates to a document ranking adjustment method using an entropy and a user profile, and more particularly, by using an entropy value and a user profile according to probability information between a user query word and subject words of each document, the accuracy of a search in an information retrieval system is improved. It's about document ranking with entropy and user profiles to improve it.

Meanwhile, the present invention uses a mutual information amount using air information between words to calculate an entropy value, and combines a weighting method using Bayesian learning with a weight calculation method based on inverse literature frequency when calculating an entropy value. By applying the entropy and user profile to achieve improved accuracy of search documents by applying the document ranking method.

On the other hand, the present invention is designed to entropy by designing a user profile consisting of the keywords used in the conventional search and the frequency of use thereof in order to reflect the recent interest, search behavior and semantic information of the user query words of users using the information retrieval system In order to reflect the calculation, and in the actual entropy calculation, only a set number of queries having the highest frequency among previous queries in the user profile participate in the calculation in order to suppress the delay of the search time due to the complexity of time. The ranking of documents relates to a document ranking adjustment method using entropy and a user profile that causes the total entropy to be rearranged in increasing order.

Recently, due to the high penetration rate of computers and the development of the Internet, a vast amount of information exists in the form of web documents on the Internet. These web documents are distributed in various sites, and the information presented is also dynamically changing. Therefore, it is not easy to find the data the user wants among the scattered information.

In general, an information retrieval system is a system that collects the necessary information, analyzes the content, processes it into an easy-to-search form, and provides the corresponding information as a search result when a user's information request occurs. One of the important functions of this information retrieval system is not simply to search the collection of documents that satisfy the user's query, but to rank the searched documents according to the degree of satisfying the query. Minimize.

Among the various types of information retrieval models, conceptual models can be divided into an exact match method and an inexact match method according to the retrieval method. Matching methods include text pattern search and Boolean models, and non-matching methods include probability models, vector space models, and clustering models. These classifications are not mutually exclusive, so you can design two or more models together.

Among the various conceptual models of IR, there have been many researches on content retrieval, including full text scanning, inverted index files, signature files, and clustering methods. Can be classified as

1 is a form of a general web information retrieval system in which a document identifier is first assigned to each web document collected by a web robot. Thereafter, nominal phrase analysis is performed on the collected documents to extract index word candidates. Each index word candidate of the extracted documents is given a term weight based on the inverse document frequency and based on this, an inverse index file is constructed.

In most commercial IR systems designed on the basis of Boolean models, each document is represented by a list of index terms consisting of subjects. The information request of the user using the information is expressed by a query, which means an expression form for searching for the presence or absence of the main words representing the document in each document.

In the Boolean model, the evaluation function and the selection criteria for documents satisfying the user's query use a common standard for most systems. That is, most of the queries expressed by the user's information request are expressed by Boolean logic, which determines whether the document is a conforming document according to whether or not index documents included in the Boolean query exist in each document. Is done.

Most boolean models use inverted index files. An IR model uses an inverted index file to construct and generate an inverted index file list that contains a list identifier of keywords and documents for all documents collected by a web robot. The searched list of inverted index files is searched using files sorted according to alphabets of main words or alphabetical order of Hangul. The search result may be obtained depending on whether or not the query word reflecting the user's information request exists in the corresponding file.

The boolean model using the inverted index file is difficult to accurately express and reflect the user's information request, and the number of documents for the search result is determined by the number of documents including the user query. In general, such systems do not take into account weights that reflect the importance of user queries and index terms for each document. In addition, the search results according to the query can obtain the search results according to the order of inverted index files designed by the system designer in advance regardless of the user's intention, and do not accurately reflect the semantic information of the user query. Therefore, the Boolean model can adjust the document targets to be searched only by using the limited methods provided by the system. Most of the search results do not satisfy the user's query intent and the search results are in the order of documents that are not related to the query intention. to provide. This boolean model provides very powerful online navigation for librarians and professional users who are used to using the system, but provides an unprecedented level of service for most simple users who do not use the system frequently.

In general, most simple users are familiar with the terms in the datasets they want to explore, but lack the training and practice, and are unfamiliar with using complex queries required by Boolean systems. it's difficult.

According to the above description, the information request of the user using the information search engine on the web is accurately reflected in the user query, and after searching the corresponding web documents including the query, the search results are ranked in the order of accurately reflecting the user's intention. It must be mad. However, when using most web search engines, there are many cases in which results that are completely different from the user's intentions appear at the top. This is judged to be insufficient efforts to accurately reflect the user's information request. Therefore, the system must be designed and implemented more intelligently to design a web search engine that accurately reflects user's information request.

The present invention has been created to solve the above problems, by using an entropy value and a user profile according to probability information between user query words and subject words of each document, entropy to improve the accuracy of the search in the information retrieval system The purpose of this document is to provide a document ranking method using a user profile and a user profile.

Another object of the present invention is to search for the entropy value by using the mutual information amount using the air information between words, and by applying the weighting method using Bayesian learning to the weight calculation method by the inverse literature frequency when calculating the entropy value It is to provide a document rank adjustment method using entropy and user profile to achieve the improved accuracy of the document.

Another object of the present invention is to entropy by designing a user profile consisting of the keywords used in the conventional search and the frequency of use thereof to reflect the recent interest, search behavior and semantic information of the user query words of users using the information retrieval system In order to reflect the calculation, and in the actual entropy calculation, only a set number of queries having the highest frequency among previous queries in the user profile participate in the calculation in order to suppress the delay of the search time due to the complexity of time. The ranking of documents provides a method of adjusting document rankings by applying entropy and user profile, which causes the total entropy to be re-adjusted in increasing order.

1 is a diagram of one embodiment of a general web information retrieval system;

2 is a flowchart of a document rank adjustment method applying entropy and a user profile according to the present invention;

3 is a schematic diagram of a web information retrieval system to which the present invention is applied;

4 is a graph of one embodiment of the present invention entropy function.

Figure 5 is an embodiment of a mutual information module according to the present invention.

Figure 6 is an embodiment of a mutual information amount calculation module according to the present invention.

Figure 7 is an embodiment of a user profile reflecting the frequency of use of the user query of the present invention.

8 is an embodiment of entropy-based document ranking adjustment according to the present invention.

Figure 9 is an embodiment of a morpheme analyzer module for extracting the index word of the present invention.

Figure 10 is a graph of one embodiment of the discriminating power of the present invention index words and documents.

11 is an embodiment of a word frequency and document frequency module in accordance with the present invention.

12 is an embodiment of an inverted index file generation module according to the present invention.

Figure 13 is an embodiment of the present invention file structure.

14 is a comparison table diagram of the present invention index word data structure.

15 is an embodiment of an extended inverse file data structure including semantic information according to the present invention.

16 is a conceptual diagram of entropy calculation between a user query and a user profile and index words of each document according to the present invention.

Figure 17 illustrates one embodiment of an algorithm for calculating entropy values of the present invention.

In order to achieve the above object, a document ranking adjustment method using entropy and a user profile according to the present invention is provided.

In the document ranking adjustment method using entropy and user profile,

Recording a query that the user wants to search for;

Reflecting the semantic information of the user query word through a predetermined information theory and calculating a probability vector information amount of each word of the query word based on air information between the words of the query word;

Designing a user profile consisting of keywords used in the most recent search and their frequency of use to reflect the user's preferences in document ranking adjustment;

Performing entropy calculation with index terms of respective documents extracted from the user profile and query terms in a search result according to a user main term;

And re-ordering and outputting the documents in order of decreasing entropy value calculated in the step.

The user profile is in the form of an annular linked list consisting of existing keywords according to each user's ID and password and the frequency of use thereof, and are arranged in order of use frequency at the same time as the user login.

The entropy calculation is characterized in that it calculates and applies the inverse literature frequency and Bayesian learning weight value of each subject of the user in order to increase the accuracy, and performs the calculation based on this.

In a preferred embodiment of the present invention, the amount of information in the predetermined information theory indicates the degree of uncertainty of the message to be delivered, and the greater the degree of uncertainty, the greater the amount of information, and the uncertainty is inversely proportional to the probability of occurrence of the message. It is expressed as the average amount of information obtained by weighting the probability of occurrence of each message by taking the logarithm of this value.

In a preferred embodiment of the present invention, the entropy calculation uses a form in which the logarithm of the average information amount is taken.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the document rank adjustment method applying the entropy and the user profile according to the present invention.

2 is a flow chart of a document rank adjustment method applying an entropy and a user profile according to the present invention, Figure 3 is a schematic diagram of a web information retrieval system to which the present invention is applied, Figure 4 is a graph of one embodiment of the present invention entropy function, Figure 5 is an embodiment of a mutual information amount module according to the present invention, Figure 6 is an embodiment of a mutual information amount calculation module according to the present invention, Figure 7 is an embodiment of a user profile reflecting the frequency of use of the user query of the present invention to be. 8 is an embodiment of entropy-based document ranking adjustment according to the present invention, FIG. 9 is an embodiment of a morpheme analyzer module for extracting an index word of the present invention, and FIG. 10 is an embodiment of identification power of an index word and a document of the present invention. 11 is an example of a word frequency and document frequency module according to the present invention, FIG. 12 is an embodiment of an inverted index file generation module according to the present invention, and FIG. 13 is an embodiment of the present invention reverse file data structure. It is an illustration. 14 is a diagram illustrating a comparison table of an index word data structure according to the present invention, and FIG. 15 is an embodiment of an extended inverse file data structure including semantic information according to the present invention. FIG. 16 is a user query word and a user profile according to the present invention. Is a conceptual diagram of entropy calculation with index words of each document, and FIG. 17 is an embodiment diagram of an algorithm for calculating an entropy value according to the present invention.

Referring to FIG. 2, the method of adjusting document rank by applying entropy and a user profile according to the present invention includes: recording a query word that a user wants to search (S10); Reflecting the semantic information of the user query word through a predetermined information theory and calculating a probability vector information amount of each word of the query word based on air information between the words of the query word (S20); Designing a user profile consisting of keywords used in the most recent search and their frequency of use to reflect the user's preferences in document ranking adjustment (S30); Performing entropy calculation with index terms of respective documents extracted from the user profile and query terms in a search result according to a user main term (S40); And resizing and outputting the documents in order of decreasing entropy value calculated in step S40 (S50). The user profile in step S30 is in the form of an annular linked list consisting of existing keywords according to each user's ID and password and the frequency of use thereof, and is sorted in the order of use frequency simultaneously with the user login. In order to increase the accuracy, the descriptive frequency and weight values of Bayesian learning of each subject word of the user are calculated and applied based on the calculation.

Referring to the operation and operation of the document ranking adjustment method applied to the present invention entropy and user profile configured as described above are as follows.

First, prior to describing the operation and operation of the present invention, the technical contents of the document ranking adjustment method to which the present invention entropy and the user profile are applied will be described.

Using document ranking to search for documents can lead to a more user-oriented search system. In such a system, the user can input a simple query such as a sentence or phrase rather than a Boolean linkage to retrieve a list of documents ranked in the order related to the user's intention. One of the representative models is the vector space model. There is this.

In the vector space model, each document and user query are represented by an N-dimensional vector space model, where N is the number of main words in each document. In this model, the matching function between the user query and each document is measured by the semantic distance according to the similarity between the user query and each document. In Salton's SMART system, the similarity measure between the user query and the document is calculated by the cosine angle between the vectors. In this case, the search results are returned to the user in decreasing order of similarity.

Calculating the similarity for all documents introduces a delay in search time according to the complexity. Therefore, in order to solve this problem, a method of searching only documents in which subject words satisfying a user query exist by referring to an inverted index file and all documents are grouped according to semantic similarity in advance and the similarity is calculated from the user query. There is a method of using only clusters that are most similarly semantically as search targets. This method is expected to save search time and improve search efficiency by searching only the document clusters related to information request topics, instead of searching the entire document to be searched.

The document clustering technique forms document clusters by using index terms or mechanically extracted keywords as the identification elements of document contents. The document cluster thus formed has a cluster profile representative of the cluster, and when searching for information, the clusters most similar to the information request are selected by comparing the information request with each cluster profile.

The application of document clustering techniques to web information retrieval is based on the hypothesis that closely related documents all fit the same information needs. This suggests that documents with similar content belonging to the same cluster are likely to be relevant documents for the same query. By using the document clustering technique, the entire document group is divided into several clusters by collecting similar documents in the same cluster. It means you can give.

The research on document clustering technique has been conducted very actively, and the representative research results are sequential cluster search and cluster document search. In general, cluster-based retrieval methods are superior in terms of physical and disk search efficiency. However, most clustering algorithms take a lot of time to form clusters, and have various problems such as poor performance in terms of search efficiency and search time, and undesirable attributes of formed clusters. Indeed, it is difficult to use such clustering algorithm effectively for a large amount of literature, and most systems experimentally apply only to hundreds of literatures. Therefore, rather than applying the document clustering algorithm to the entire documents to be searched, the problem of clustering time is solved by applying only the documents that satisfy the user query, and the documents to be searched are clustered according to the meaning of the user query. Research is being conducted in a satisfactory direction.

Looking at the existing researches on the Korean information retrieval system to improve the accuracy, most studies are focused on the processing of nouns and compound nouns for accurate index word extraction. One of these studies is to consider noun phrases and simple sentences in addition to each subject in consideration of the ambiguity of words caused by homonyms and derivatives, which are one of the characteristics of Korean characters, rather than constructing an information retrieval system using only the subjects representing documents. We introduced the concept of a keyfact, which means that the user wants to find something in the document. However, in order to extract a key fact, a large amount of verb / adjective dictionaries is required in addition to the noun dictionary, and thus a lot of time and effort are required.

Another study uses a thesaurus-based ranking algorithm to indicate the degree of satisfaction of a user query in a Boolean search system. Thesaurus is a kind of lexical classification dictionary that expresses words in conceptual relations according to their meanings and expresses the concepts and concepts in a hierarchical structure using specific relations such as hierarchical relations, whole-part relations, and relations. This thesaurus is used not only for the selection of appropriate index headings and the control of index words when indexing, but also for the selection of appropriate search words for information retrieval. Therefore, information retrieval using thesaurus contributes to the improvement of retrieval efficiency through expansion of search term in addition to the purpose of term control.

Since the thesaurus-based information retrieval system is selected from the thesaurus, documents having the same contents can be searched by the same index, regardless of the specific terms of the document, and the information retrieval system using the association information between the index terms. Can be used to increase the recall. However, since thesaurus types are built according to the meaning of words, they are somewhat different from the usage found in actual corpus. Therefore, using the similarity of the lexical hierarchy as it is during information retrieval may increase the recall and reduce the accuracy.

One embodiment of a thesaurus-based information retrieval system proposes a two-stage document ranking model using mutual information to improve accuracy in a natural language information retrieval system. In this study, the second document ranking adjustment calculates the mutual information value between the search terms in the query word and the key words of each document, and re-adjusts the document ranking based on this.

In the present invention, an entropy-based similarity measure using quantitative information representing a quantitative relationship between words and words is used to adjust the ranking of search results for a user query. Here, entropy is calculated based on the average amount of information, which is the degree of uncertainty in a message. The similarity between two documents is larger as the entropy value is smaller.

The entropy calculation in the present invention uses a form in which the logarithm of the average information amount is taken. As you can see from the nature of the logarithmic function, taking a log of some values is insensitive to changes in the values. For example, when there are four values of 10, 100, 1000, and 10000, the difference between the largest value and the smallest value is 9,990. However, if you log these values, you get 1 (= log10), 2 (= log100), 3 (= log1000), 4 (= log10000), and the difference between the largest and smallest values is only three. Therefore, the entropy that uses the mutual information amount and the mutual information amount as the information amount uses the similarity of the same concept, but acts sensitively or insensitively according to the change of the information amount.

The general search engines cannot understand the query form of natural language and cannot process the contents of the literature using semantic knowledge and subject area knowledge of language correctly. Most search engines also have many problems, such as lack of reasoning and inability to use prior information about users. In order to solve these problems, a research on an intelligent information retrieval system using a conformity feedback method using mutual information amount has been attempted.

To give intelligence to search engines, it is necessary to be able to utilize systematic knowledge in addition to simple data or information, and to have natural language comprehension and reasoning for problem solving. In other words, the intelligent search engine should be a knowledge-based system utilizing various knowledge bases, and a system capable of performing proper inferences using the established knowledge. The reasoning function can be explained in three steps as follows.

1) Inference of relationship between information request and literature using index knowledge

2) Appropriate Reasoning Using User's Knowledge

3) New Query Reasoning Using Subject Knowledge

An example of the overall configuration of the Korean web information retrieval system used in the present invention is shown in FIG.

In the present invention, unlike the existing Korean web information retrieval system, in order to give the system intelligence, the amount of mutual information, which is the degree of correlation between words, is calculated from the corpus.

In general, it is very important to grasp the propensity of information desired by a user, but there are many technical difficulties to actually model and implement the information. To this end, an interface that infers the user's interest indirectly by analyzing the user's behavior or input rather than the existing user query input method is required. In addition, in order to effectively implement information filtering by learning of the user's inclination, the information usage tendency of the user is expressed and updated according to the learning, the information on the web is effectively expressed, and the filtering by the learning is performed. Components such as methods are required.

In an information retrieval system, it is important to increase the user's actual satisfaction with the system by ranking the search documents that best fit the user's information needs at the top of the search results without retrieving and selecting the user's query word and reducing the recall rate. The purpose and scope of the present invention for increasing such actual satisfaction can be summarized as follows.

The existing reverse file-based Korean information retrieval system simply reflects whether or not the user query exists in the document. This negates the handling of contextual information, so there is a limit to the exact retrieval of documents that are semantically related to the user query. Most information retrieval system users require higher accuracy than reproducibility, and in particular, it is hoped that search result documents according to their information needs will be ranked at the top. In order to satisfy such a user requirement, the present invention calculates entropy between the user query word and the subject words of each document through probability information based on the mutual information of words. At this time, the Bayesian estimate and the weight of the inverse literature frequency were used for accurate entropy calculation, and a user profile composed of recent user queries was designed and used. Also, in order to obtain accurate entropy calculation value, a formula for applying mutual information amount to probability information calculation formula was derived. The calculated entropy value is used as a ranking method of documents for the search results, and the document ranking is readjusted in order of documents having the lowest entropy value for the final search result for the user query. This document ranking method may be used as a standalone model.

In order to improve the accuracy of information retrieval, the present invention proposes a document ranking algorithm using entropy and user profile. Entropy calculation for rank adjustment uses mutual information amount value based on air information between words, which is used to calculate probability information amount for finding similarity between documents for search results according to user query word. In addition, we design a user profile composed of the user's recent query words to reflect the semantic information about the user query in entropy calculation, and propose a weighting method by Sparck Jones's bibliographic frequency and Bayesian learning. We propose an algorithm to improve the accuracy of document rank adjustment by reflecting this in entropy calculation.

It looks at the operation and operation of the document rank adjustment method applying the present invention entropy and user profile in connection with the related technical content as described above.

The present invention is preferably used to design a search ranking adjustment system based on entropy and user profile to improve the accuracy of Korean web information retrieval system. To this end, the system to which the present invention is applied uses an information theory to reflect the semantic information included in the user query, unlike the existing search system that reflects only whether the user query exists in the inverse file (S10) ( S20).

In order to implement a document ranking adjustment system based on entropy and user profile, the present invention calculates the probability vector information amount of each word using information theory. Here, the calculation of the probability vector information amount is performed by the calculation of the mutual information amount based on the air information between words. For this purpose, a new calculation formula is derived and used (S20).

In addition, in order to reflect the user's preferences when adjusting document rankings, a user profile composed of the most recently used keywords and their usage frequency is designed (S30). The designed user profile is an annular linked list consisting of existing keywords according to each user's ID and password and the frequency of use. The user profile is arranged in order of frequency of use as the user logs in.

User profiles and query terms perform entropy calculations with index terms of the documents extracted from the search results according to the user's main subject.In order to improve the accuracy, the backlit frequency of each subject and the weight value by Bayesian learning Calculate and apply and calculate entropy based on this (S40). Finally, the document rank is readjusted in order of the documents having the lowest entropy value (S50).

In the present invention entropy and user profile, look at the information theory of step S20 as follows.

In information theory, the amount of information represents the degree of uncertainty about the message to be conveyed. Here, the greater the degree of uncertainty, the greater the amount of information, and the uncertainty is expressed as the average amount of information that takes the inverse of the probability of occurrence of the message and takes the log of this value as the weight of the probability of occurrence of each message. That is, the average amount of information that a message has is a quantification of the freedom of choice that is given when one message is selected from the various messages that an information source can produce. In this case, the degree of uncertainty as to which message is selected is expressed as entropy, which is the average amount of information of the message, and the uncertainty about the message can be obtained by information obtained by transmitting the message. Thus, the amount of information conveyed by a message is measured in entropy. That is, information is expressed in uncertainty or entropy.

Entropy H, which represents the amount of uncertainty, can be defined as in Equation 1 below.

(Formula 1)

In Equation 1, the amount of information logp _i of message _i is determined by the probability p _i that the message is selected. Therefore, H is the average amount of information possessed by n messages, and p _i represents the probability of occurrence of the i-th message.

In general, entropy H has the following characteristics.

4 is an entropy function Means. Here, H (p, 1-p) has a maximum value when p is a uniform distribution, that is, when p = 1/2.

In the entropy and user profile according to the present invention, the mutual information amount and entropy are as follows.

The amount of mutual information can be calculated based on the air word information of words, which means that two words appear together in a document. The appearance information of these words is used to construct the air information.

5 is an air information module used in the present invention, when executed using words, words, and window size input information, the number of documents containing a word, the number of times of air appearing in the document, and calculated using such information. The amount of mutual information can be obtained.

FIG. 5 is a module for calculating the position, frequency of occurrence of documents, and the number of times of airing when the window size is 1,000 for the keywords "engine" and "search". Indicates. It extracts air words within a range of windows (100 phrases) and calculates the frequency of occurrence of the index words extracted from the test target document. Calculate the amount of information.

Mutual information is a quantitative representation of the amount of random variable X provided by the random variable Y, that is, the joint relationship between the random variables X and Y. Here, by substituting words to be investigated as random variable values of random variables X and Y, the dependency relationship between words can be quantitatively represented.

Calculation of the mutual information amount can be calculated by Equation 2, which is a formula for quantitatively expressing the association between words. In Formula 2, the number of words x and y used in a corpus of size N is called f (x) and f (y), respectively. x, y), if N is large enough, the value of Equation 2, the exact amount of mutual information, can be approximated by Equation 3, the relative frequency of corpus of size N.

(Formula 2)

(Formula 3)

The meaning and characteristics of Equation 3 can be defined in six ways.

First, if the words x and y are related, then the probability p (x, y) will be greater than p (x) * p (y) and consequently the MI (x, y) value will be greater than zero.

Second, if the words x and y are not very relevant, then the probabilities p (x, y) and p (x) * p (y) will be about the same, resulting in MI (x, y) being close to zero. Will get the value.

Third, if the words x and y have no relation at all, the probability p (x, y) will be less than the value of p (x) * p (y), so that the value of MI (x, y) is 0 Will be.

Fourth, in general, the mutual information amount has a symmetric in which MI (x, y) = MI (y, x) is satisfied. However, the amount of mutual information applied in the present invention does not allow symmetry in extracting air words of index words. That is, a value for calculating the frequency of occurrence of the air word y in a range of windows after the index word x is selected by the weight calculation, and a value of the air word x in a range of windows after the index word y is selected by the weight calculation. This is because different distinct values are calculated when calculating the frequency of appearance.

Fifth, since the mutual information amount formula is a stochastic representation of the relationship between words and words, the information amount of the compound word can be quantitatively represented.

Sixth, the mutual information quantity formula should be applied to relatively large documents because words with low occurrence frequency are not sufficient grounds to confirm the objectivity of the dependencies between words.

FIG. 6 illustrates a module for calculating a list of user queries and deferred words and the amount of mutual information based on the case when the user query is "fax".

In calculating the amount of mutual information in FIG. 6, since the relationship between random variables in a document means a value for a relationship between words, it is not meaningful to obtain an average of occurrence frequencies of words. Therefore, Equation 4 is used to find the amount of mutual information.

(Formula 4)

Formula 4 is This can be redefined as Equation 5.

(Formula 5)

In Equation 5, p (y) means the set of indexes in the document containing the query y, so it can always be regarded as 1. That is, since log (py) = log1 = 0, it can be seen that the probability that x appears in the document y is proportional to the amount of mutual information between x and y.

(Formula 6)

In Equation 6, the probability vector information value for entropy calculation can be used as the amount of mutual information between words.

Since the probability information value p _i used for entropy calculation must satisfy the condition as shown in Equation 7, it is normalized.

(Formula 7)

If the information values of the probability vectors are not normalized, that is, when the individual values of the probability vectors do not have the same scale, the total entropy value is changed by the larger information value. In this case, information values having a relatively small scale cannot influence the change of the overall entropy value, and thus, a document in which the user's desired information is reflected cannot be retrieved. However, if the probability information values of all documents through the normalization process are the same scale, accurate search can be performed because the information of the user is equally received.

In the present invention, a profile and a user profile will be described.

A user profile describes a user's personal interests. In designing an information retrieval system, if the user's preferences or interests are known in advance, the user can be provided with convenience by first searching each user only the search results belonging to the user's interests.

In an IR system, user profiles are not a critical component of the system. This is because most systems can give users the search results they want without relying on user profiles. However, since most of the search results include results that users do not want, users have to check a large number of documents or modify the query again to get the results they want. These problems can be partially but partially solved by introducing user profiles.

The user profile systems studied so far include systems using word lists and associative feedback, systems using user models, systems using genetic algorithms or word-to-word air information, and systems using thesaurus. These systems are related to information filtration. However, user interest in information filtration can be seen as a rather wide range of interest, but user interest in information retrieval systems can be seen as a limited field of information needs for a specific field. Therefore, there is a problem in applying these systems to the information retrieval system as it is.

The researches on user profiles in IR systems include Bayesian Probability, Near Neighbor (NN), Decision Trees, Rocchio's Algorithm, and Neural Networks. Most of these studies have been conducted to learn the interest of users in the agent system.

7 is a user profile designed according to the present invention, and is composed of query words and frequency of use which are most recently used for searching. Here, the word with the highest frequency of use is considered to best reflect the user's interest, and the user profile is arranged in descending order according to the frequency of use.

In the user profile of FIG. 7, query information that has been used for a past search and frequency information of the corresponding query words is used for entropy calculation along with the user query word. It is expected that some accumulated frequency information is required until the user profile designed in the present invention reflects the search intention of the user. However, as time passed, the user's profile of interest acts as a weight for entropy calculation along with the user's query word.

In the present invention, entropy-based document ranking adjustment is as follows.

8 is an entropy based ranking system used in the present invention. An entropy-based ranking system can be largely classified into a morphological analysis module for extracting index words, an index word building module, an air information building module for constructing mutual information amount between each word, and finally a ranking adjustment module by entropy calculation. have.

In the entropy-based document ranking adjustment of the present invention, morphological analysis for index word extraction is as follows.

In order to improve the search efficiency of the web information retrieval system, an appropriate index word must be extracted from each web document, and unnecessary words must be removed for this purpose. In the English-speaking search system, the stemming technique is used as a method for removing unnecessary functional words, but in Korean, morphological analysis is required to extract unnecessary words. The first step in designing a morpheme analyzer is to identify the units that make up the word. Linguistics defined the "minimal unit of meaning" as a morpheme and proposed six principles for identifying morphemes. However, from the standpoint of mechanical processing, it is difficult to apply them as they are. Therefore, in the present invention, in terms of machine processing, "a component of a word understood by a morphological analyzer" is defined as a morpheme, and such a component becomes a headword of a dictionary as it is.

As in Korean, it is very difficult to accurately define the binding conditions between morphemes in a deadlock that expresses the grammatical functions of lexical morphemes as a combination of grammatical morphemes. Thus, combinations of morphemes of the same nature are treated as one morpheme. That is, the combination of the suffix and the suffix, the combination of the search and the investigation, and the combination of the ending endings are classified into one suffix, the investigation and the ending ending. In addition, the non-separated pre-fish mothers and the combined types of their mothers are treated as mothers, and those with irradiated horses are also treated as mothers. They are very difficult to express the coupling aspect as a rule, and even if the connection aspect is represented as a rule, it may allow a wrong coupling. Therefore, integrating them is more efficient because it avoids false analysis and reduces the burden of checking join order and join conditions. They are very difficult to express the coupling aspect as a rule, and even if the connection aspect is represented as a rule, it may allow a wrong coupling. Therefore, processing them integrally is more efficient because it prevents false analysis and reduces the burden of checking the join order and join conditions.

9 is a morpheme analyzer designed by the present invention for extracting an index word, and shows the result analyzed by the morpheme analysis algorithm at the bottom of the screen by receiving a text sentence at the top of the screen.

In general, the functionality of a morpheme analyzer may vary depending on the application, so the question of which functions the morpheme analyzer should have depends on what purpose the morpheme analyzer is used for. Since the morpheme analyzer used in the present invention is designed for the implementation of the web information retrieval system, unlike the morpheme analyzer used for the spell search, it tries to analyze the noun phrase more accurately than any other part.

The morphological analysis methods for Korean include the longest matching method, head-tail classification method, and tabular parsing method using left and right access information. The longest matching method and the head-tail classification method do not analyze all possible combinations when a word can be analyzed in various ways, causing backtracking at a higher level. The Tabular Parsing method, on the other hand, obtains all possible combinations, but in order to exclude erroneous associations, the dictionary's headings are given connectivity information that determines whether or not they can be combined with other morphemes. Difficult to maintain

The morpheme analyzer used in the present invention is a morpheme analyzer that obtains all possible combinations using only general part of speech information, and estimates the possible combinations if the original phrases can be composed of morphemes extracted from one word, Mismatching is a morphological analyzer using a multistage filtering technique that eliminates using the morphological analyzer's filtering capabilities.

In the present invention entropy-based document ranking adjustment, look at the index selection and classification as follows.

An index is an analysis of the text of an input document and extracts a word or word phrase that can represent the subject content of the document. Here, the text to be analyzed becomes the title of the full text or abstract of the document. In general, an index word can be defined as any word that is allowed to interpret a particular document and write a question to search a document file according to rules that define the usage of the term and the relationship between terms.

The most frequent words among the words that are the subjects of the index word are general words and are not worth using as the main words, while the low frequency words are also meaningless as the main words. Therefore, the remaining words except these meaningless words become the subject words of the index word.

In the present invention, as shown in FIG. 10, after determining the highest limit frequency and the lowest limit frequency in the frequency distribution of the word, the middle frequency words belonging to the two limit frequencies are considered to have a high discrimination ability in the document content, and the intermediate frequency is selected as the color language. .

What is important in the index determination technique is to assign a weight to the determined index, which is a function that supports efficient search. Since the determined index word is used when searching for the information desired by the user, it can be said that the index word determining technique determines the performance of the information retrieval system.

In general, index word selection techniques include statistical techniques, linguistic techniques, document structure techniques, and controlled vocabulary-based techniques. Linguistic techniques applied to automatic indexing can be divided into lexical, syntactic, and lexical stages. Can be.

With the recent development of natural language processing technology, indexing using natural language can be more effective than indexing using controlled vocabulary. Natural language indexing is easy to maintain the latest, low input cost, and database. It has the advantage of easy exchange.

In the present invention, in order to reflect the characteristics of the Korean language by post-processing by the morpheme structural rules based on the parts of speech extracted through morphological analysis, the structural method reflecting the characteristics of Korean as well as the statistical method is used in parallel. In addition, the statistical criteria for selecting index words in the present invention are mainly based on the frequency of occurrence of words.

The criteria that directly use the frequency of appearance are divided into simple frequency and relative frequency according to the frequency calculation method of words. Here, the simple frequency is divided into a word frequency and a document frequency according to where the word appears.

Word frequency (T _i) is the number of a particular word k appears in each document i is the index target document, And b _ik = 1 when f _ik ≥ 1 and b _ik = 0 when f _ik = 0.

The simple frequency does not consider the size of the document group, the length of the text to be analyzed, and the frequency of use of words. It is difficult to use the criteria for selecting index words based on this. Therefore, the relative frequency considering these factors is evaluated as a more suitable criterion.

11 is a module for calculating word frequency and document frequency based on statistical criteria for the keyword "engine". The upper part of the screen shows the document number, the document position, and the contents of the document. The lower part of the screen shows the stemming results of the document.

An index word file refers to a file that stores information on index words extracted by an index word determination technique. This file is shared during indexing and retrieval, and should use fast speed and efficient storage.

Commonly used techniques for organizing information files include using an inverted file that stores a reference to each index word, a bitmap index consisting of a bit stream of index word information corresponding to each document, and It can be classified by using a signature file. In this case, the dynamic configuration, the search speed, the size of the storage space, and the like are important factors in the reverse file method, and the retrieval technique, the size of the storage space, and the like are important factors in the bitmap and summary file method.

12 is an inverted index file generation module used in the present invention. Here, URLs of the homepage appear at the top of the screen, and contents of the homepage are shown at the bottom of the screen.

The indexing technique used in the present invention should consider not only the position and frequency of the index word but also the meaning of the index word, and since the configuration of the system is complicated by simply adding additional data structures, the index scheme is extended without using the bitmap index as the data structure. Use an inverted file structure.

In the entropy-based document ranking adjustment of the present invention, generation of an inverted index file is described as follows.

The inverted index file technique consists of a list of the documents it contains, or an inverted list, for each of the key words extracted from all stored documents, and using a file sorted according to the alphabetical order of the key words. Search.

13 is an example of an inverted file structure used in the present invention.

As shown in FIG. 13, the reverse file designed in the present invention stores information without any processing for index words, unlike a bitmap or a summary file method. First, the character reference file records the location information of the first syllable of the index term-by-syllable reference file sorted by the order of the first syllable of the index word. The reference file for each index term records the frequency of occurrence of index words, weights and document identifiers, and the relevant location information of the reference file. Finally, a list of documents having the corresponding index word can be obtained through the reference document arrangement based on the document information obtained from the document identifier reference file.

13 shows a process of finding a document having an index word of "information". First, the starting position of the index term reference file starting with 'jeong' through the character reference file to find "information" is obtained first. Then, the number of documents for the term "information" in the index term reference file, Obtain the index term frequency in the entire document, the starting position value of the document identifier reference file, etc., calculate the weight by calculating the term occurrence frequency in the reference document and the reference document in the document identifier reference file, and present the contents of the document to the user. do.

This method has the disadvantage of expensive storage update or reorganization rate and merging them when the list is large or long.However, in addition to the advantage of being able to search immediately without searching, it is possible to express the weight of key words or information about documents. It is easy to calculate similarity or search for homonyms for a query. Considering these points, the reverse file technique is generally suitable for static environments, that is, environments with frequent queries and few updates.

In the table of FIG. 14, the characteristics of the bitmap index, summary file index, and reverse file index are compared. As can be seen from the table of FIG. 14, except for the ease of system implementation, a bitmap index and an inverted file index are suitable for a data index for intelligent index extraction and additional information storage. The reverse file index is suitable for natural language processing using a dictionary and a dictionary.

In spite of the advantages shown in the table of FIG. 14, the inverse file has a disadvantage of degrading the performance of the system when the storage space update or reconfiguration rate and the list become long. Therefore, in order to additionally configure additional information on the index word, the expansion of the reverse file structure is essential, and the expansion should be carefully designed so as not to affect the performance of the system.

In general, a web information retrieval system provides a service through a common gateway interface (CGI). The problem that arises is how to increase the performance of the server. The server executes a CGI for each request to handle requests coming from the client's browser and creates a new process each time the CGI is executed. Then it passes it to CGI through environment variables and stdin. Information collected through CGI is eventually passed to the client via stdout. At this point, the problem is the processes that are created each time at the request of the client. Each time the server executes CGI, a new process is created each time, which increases the number of processes for a client's request and consequently decreases the server's performance. In particular, due to the nature of the reverse file, a large amount of documents must be indexed, and the size is proportional to the amount of documents to be indexed, and each CGI process loads the same reverse file data structure into a memory area for each request from a client. Will be done.

In order to solve this problem, an Internet server application (ISA) that interoperates with an HTTP server has been proposed. Since the Internet server application is implemented as an executable file in the form of a DLL (Dynamic Linking Library), a separate process is not created each time it is executed. Unlike traditional stand-alone executables, DLLs belong to the process that called the DLL, and once loaded into memory, they remain in memory until they are no longer used so they can continue to process requests from clients faster.

Even on systems using Internet server applications, the reverse file data structure must be loaded into memory once, causing unnecessary memory waste. In addition, since the system is opening the file while the HTTP server is running, the problem arises that adding and changing the reverse file data structure is impossible.

In the present invention, in order to solve this problem, the reverse file data structure is composed of a dynamic trie by the first syllable of the index word. In addition, index terms are designed to continue to contain additional information in addition to location and frequency information in the document.

15 is an extended inverted file structure containing semantic information designed by the present invention.

Since the inverted file of the dynamic tri-structure of FIG. 15 stores only information about the first syllable and child nodes of the inverted file, the size of the file to be initially loaded is 0.5% compared to the inverse file size which is not dynamically constructed. It is only In addition, by dividing about 2,000 index words extracted from 40,000 word dictionaries into 644 files, unnecessary nodes that are not used well are not loaded into memory, thus enabling efficient use of system resources. This dynamic tri inverse file loads only the nodes needed by the system, and once loaded into memory, all the Internet server applications running in the same process can be shared by the system. It is advantageous in terms of efficient use of resources and improvement of performance.

Due to these advantages, the expanded reverse file structure not only maximizes the performance of the web information retrieval system, but also improves the system performance compared to the existing static CGI method in terms of server resource sharing, and adds or maintains new information. The time and effort required for renovation are significantly reduced, which is also an excellent economic aspect.

In the entropy-based document ranking adjustment of the present invention, the document ranking adjustment using entropy and Bayesian estimation is as follows.

In order to reflect the contextual information on the words included in the user query, the present invention calculates entropy between the input vector consisting of the user query and the user profile vector and the index words of the dynamically searched documents.

16 shows an example for calculating entropy between an input vector and index words of dynamically retrieved documents.

In FIG. 16, if the user query word is "disease", and the main words of [Document # 1] are ["uterine cancer", "clinical", "research", and "patient", the entropy calculation at this time is performed by Equation 9. Can be calculated

(Formula 9)

In Equation 9, the value 4 of n means that there are four index words in [Document # 1], and each p _i is defined by Equation 6 for the disease and ["uterine cancer", "clinical", "study", "patient" Can be obtained from the mutual information amount This same working procedure is also carried out for H ₂ (virus), H ₃ (disease), H ₄ (patient). The final entropy calculation of the user query and user profile vectors and [Document # 1] is calculated by Equation 10.

(Formula 10)

In calculating the entropy between the user query-user profile vector and the index words of the dynamically retrieved documents, the weighting by the inverse literature frequency and the estimation by Bayesian learning are used. In this case, 1 * inverse literature frequency value was used as the user query word, and the word frequency was divided by the sum of the frequency of all the profile words in the three most frequent frequencies of the user profile.

Here, the weight of the user query and the user profile is determined using Bayesian posterior probability. The Bayesian posterior probability used here is defined as in Equation 11.

(Formula 11)

In Equation 11, t represents text and d ₁ , ..., d _n represents a document. In other words, Bayesian posterior probability can be expressed as the product of the likelihood function and prior probability of text in documents.

When there is no information about the dictionary probability p (t), a value of 1 is generally used. However, in the present invention, the information on the dictionary is increased by using the inverse text frequency of the text, and the likelihood function is used for each word in the corpus. Use the frequency of occurrence of

The post information obtained by using the bibliographic frequency and the frequency of each word in the corpus, which are specific information about the dictionary, is used as a weight for each word between the user query and the user profile.

For example, the Bayesian estimate in Equation 10 can be obtained from the user profile. For example, if the frequency information of 959 queries for "bottle", 9 queries for "patient" and 17 queries for "virus" were obtained from the user profile, the initial value of the Bayesian estimate is 959/1085, 9/1085, and 17/1085, respectively, and the final weight can be obtained by multiplying this value by the weight of the inverse literature frequency. Therefore, Equation 10 is recalculated to Equation 12.

(Formula 12)

The reason for using Bayesian weight in the present invention is to reflect the changing characteristics of the population in reflecting the frequency of the words in the user profile in the weight, and in practice it can be used to model the user's search behavior usefully. Of course you can check through.

17 shows an example of an algorithm for calculating an entropy value between a user query using a Bayesian estimate and an inverse bibliographic frequency and a user profile.

As described above, the document ranking adjustment method using the present invention entropy and the user profile may improve the accuracy of the search in the information retrieval system by using the entropy value and the user profile according to the probability information between the user query word and the subject words of each document. It provides the benefits of doing so.

In addition, the present invention uses a mutual information amount using the air information between words to calculate the entropy value, combined with the weighting method using Bayesian learning to the weight calculation method by the inverse literature frequency when calculating the entropy value search document It offers the advantage of achieving an improvement in accuracy for the system.

In addition, the present invention design a user profile consisting of the keywords used in the conventional search and the frequency of use thereof in order to reflect the recent interest and search behavior of users using the information retrieval system and the semantic information of the user query words to calculate the entropy In the actual entropy calculation, only the set number of queries having the highest frequency among the previous queries in the user profile participates in the calculation to suppress the delay of the search time due to the complexity of time. The ranking of the reassurance causes the total entropy to be readjusted in increasing order.

Although the preferred embodiments of the present invention have been described in detail above, those skilled in the art will appreciate that the present invention may be modified without departing from the spirit and scope of the invention as defined in the appended claims. It will be appreciated that modifications or variations may be made. Therefore, changes in the future embodiments of the present invention will not be able to escape the technology of the present invention.

Claims (5)

In the document ranking adjustment method using entropy and user profile, Recording a query that the user wants to search for; Reflecting the semantic information of the user query word through a predetermined information theory and calculating a probability vector information amount of each word of the query word based on air information between the words of the query word; Designing a user profile consisting of keywords used in the most recent search and their frequency of use to reflect the user's preferences in document ranking adjustment; Performing entropy calculation with index terms of respective documents extracted from the user profile and query terms in a search result according to a user main term; And re-ordering and outputting the documents in order of decreasing entropy value calculated in the step. The user profile is in the form of an annular linked list consisting of existing keywords according to each user's ID and password and the frequency of use thereof, and are arranged in order of use frequency at the same time as the user login. The entropy calculation calculates and applies the inverse literature frequency and Bayesian learning weight value of each subject of the user to increase accuracy, and performs the calculation based on the entropy and the user profile. How to adjust document ranking using. The method of claim 1, wherein the amount of information in the predetermined information theory indicates the degree of uncertainty of the message to be delivered, and the greater the degree of uncertainty, the greater the amount of information, and the uncertainty is inversely proportional to the probability of occurrence of the message. A method of adjusting document rank by applying entropy and user profile, characterized in that it is expressed as an average amount of information obtained by weighting a probability of occurrence of each message to a value obtained by logging a value. The method of claim 1, The entropy calculation uses a form in which the logarithm of the average information amount is taken, The amount of information represents the degree of uncertainty about the message to be conveyed, The entropy H, which represents the amount for the uncertainty, is a formula, Defined as, In the above formula, the amount of information logp _i of message _i is determined by the probability p _i of the message being selected, The entropy H is the average amount of information of n messages, p _i represents the probability of occurrence of the i th message, The entropy H is formulaally, Document ranking adjustment method using the entropy and user profile, characterized in that having the characteristics, such as. 2. The method of claim 1, wherein Bayesian probabilities are used for the design of the user profile. The method of claim 1, wherein the readjustment of the document rank, Applying entropy and user profile, which is achieved by a stemming module for extracting index words, an index word building module, an air information building module for building mutual information amount between each word, and a ranking adjustment module by entropy calculation. How to adjust document ranking.