KR20110023304A - Method and system of configuring user profile based on a concept network and personalized query expansion system using the same - Google Patents

Abstract

PURPOSE: A concept network based user profile configuration method and a system thereof, an individual query extension system using the same by using a session interest are provided to recommend an individualized query by comparing a user profile with a word which a user makes a question. CONSTITUTION: A keyword extractor(100) extracts a keyword from documents which a user refers to. A session interest making module(210) generates a session interest by using the keyword. The session interest making module makes a user profile by accumulating the session interest. A comparing module(220) compares the generated session interest with the concept of the user profile. A user profile updating module(230) adds the session interest and the concept of the user profile.

Description

METHOOD AND SYSTEM OF CONFIGURING USER PROFILE BASED ON A CONCEPT NETWORK AND PERSONALIZED QUERY EXPANSION SYSTEM USING THE SAME}

The present invention relates to a method and system for constructing a conceptual network-based user profile and a system for expanding a personalized query using the same, and more particularly, to a method and system for constructing a conceptual network-based user profile for a personalized search and a system for expanding a personalized query using the same. .

In general, an information retrieval system refers to a system that provides information requesters by quickly finding information suitable for a request from a database accumulated in an easy-to-find form by collecting, processing, and processing information or data that is expected to be needed by an information consumer.

In the past, a web portal site was constructed by using such an information retrieval system. The web portal site utilizes an information retrieval system to collect, process, and process information on a web site through the Internet, and provide the retrieved information to a user terminal, for example, a computer or a wireless terminal.

While currently available search engines perform well, there are problems with too many search results. This is because the search engine is basically a query based system that compares the similarity between the query and the document. A word can have many different meanings, and because each person's intention is different, it can't guarantee accurate results.

To solve this problem, many methods have been studied, such as query expansion and ranking of search results according to user preferences.

Query expansion is a method of improving the quality of search results by changing or extending the user's query to better words. Such query expansion requires a dictionary to find words related to the query, and the creation and maintenance of the dictionary is a very expensive task.

Therefore, the technical problem of the present invention has been made in view of the above, an object of the present invention is to provide a method for constructing a concept network-based user profile for personalized search by using the correlation between the user queryed words and keywords extracted from the visited web page will be.

Another object of the present invention is to provide a concept network-based user profile configuration system for performing the user profile configuration method described above.

Another object of the present invention is to provide a system for expanding a personalized query using the conceptual network-based user profile.

According to an embodiment of the present invention, there is provided a method of constructing a conceptual network-based user profile, including: extracting keywords from documents searched by a user using a search engine to create a user profile; Generating a session interest using the extracted keywords, and accumulating the generated session interests to create a user profile; and each time a new session interest is generated, concepts of the generated session interest and the user profile are generated. Comparing the session interest with the concept of the user profile if the session interest is the same as or similar to the concept of the user profile, and adding as a concept of the new user profile if different. Include.

In an embodiment of the present invention, the similarity comparison between the session interest and the concept of the user profile may be made using the web directory of the Open Directory Project (ODP).

The comparing of the concepts of the session interest and the user profile may include expressing each of the concepts of the session interest and the user profile in a term vector, and a term vector corresponding to the session interest and the user profile. Changing each term vector corresponding to the concept of to a first vector and a second vector having a category of the ODP as a dimension, and comparing the first vector and the second vector using cosine similarity. can do.

In an embodiment of the present invention, the extracting of the keyword may include extracting a word in each web document based on the TF-IDF weight, storing the words extracted from each document in one table, and counting the number of appearances. And calculating a session interest using only words having a number of appearances above a predetermined threshold based on the number of appearances.

In accordance with an embodiment of the present invention, a conceptual network-based user profile configuration system includes a keyword for extracting keywords from documents searched by a user using a search engine to create a user profile. A session interest generation module for generating a session interest by using the extractor, the extracted keywords, and accumulating the generated session interests to create a user profile, and a generated session access whenever a new session interest is generated. A comparison module for comparing the concept of the rest and the user profile, and when the session interest is the same as or similar to the concept of the user profile, when the concepts of the session interest and the user profile are summed and different, Users to add as a concept of new user profiles To include the file update module.

In an embodiment of the present invention, the comparison module may compare the similarity between the concept of the session interest and the user profile using the web directory of the Open Directory Project (ODP).

Here, the comparison module represents each of the concepts of the session interest and the user profile as a term vector, and changes the term vector corresponding to the session interest into a first vector having a category of the ODP as a dimension. After changing a term vector corresponding to a concept of a user profile into a second vector having a dimension as a category of an ODP, a cosine similarity may be used to compare the first vector and the second vector.

In an embodiment of the present invention, the keyword extractor extracts a word in each web document based on the TF-IDF weight, stores the words extracted from each document in a table, calculates the number of appearances, and then The session interest may be constructed using only words having the number of appearances above a certain threshold based on the number of appearances.

In an embodiment of the present invention, the keyword extracting unit is a web document collection module for querying a query engine and analyzing and storing a web page as a result, a single noun extracting module for extracting a single noun from stored web pages, and a TF And a TF-IDF weight calculation module for calculating IDF weights, and a most important word frequency calculation module for selecting keywords by calculating the most important word occurrence frequency with respect to the TF-IDF weight.

In one example, the most important word appearance frequency calculation module may extract the upper specific ratio of words based on the number of words in the document, calculate the most important word occurrence frequency and select keywords.

In another example, the most important word occurrence frequency calculating module may extract a word having a higher specific ratio based on the maximum TF-IDF weight in the document, calculate the most important word occurrence frequency, and select a keyword.

In order to realize the above object of the present invention, the personalized query expansion system according to an embodiment of the present invention provides a conceptual network and a query query extended by querying the conceptual network in response to the provided query query. And a query extension module for providing the obtained extended query terms to the client unit and, as a query term is selected by the client unit, requesting a search by providing documents corresponding to the selected query term to a search engine, and requesting a request. A search module for providing a document corresponding to a result corresponding to the client module, and comparing the session interest with a concept of a user profile stored in the conceptual network as a session interest is provided from the client device, wherein the session interest is obtained. Is the same as the concept of the user profile Includes a user concept network management module that adds the concepts of the session interest and the user profile to the concept network if similar, and adds them as concepts of a new user profile to the concept network if different. do.

In an embodiment of the present invention, the client unit receives a user input, extracts a keyword from the searched web page and provides the query input module to the query expansion module, and displays documents provided as documents are provided by the search module. And a keyword extraction module for extracting keywords from the document of interest selected by the user and providing the extracted keywords to the user concept network management module.

In an embodiment of the present invention, the keyword extraction module extracts words in each web document based on the TF-IDF weight, stores the words extracted from each document in one table, calculates the number of appearances, and The session interest may be configured using only words having the number of appearances above a predetermined threshold based on the number of appearances.

In an embodiment of the present invention, the keyword extraction module includes a web document collection module for querying a query engine and analyzing and storing a web page as a result, a single noun extraction module for extracting a single noun from the stored web pages; TF-IDF weight calculation module for calculating the TF-IDF weight, and the most important word occurrence frequency calculation module for selecting a keyword by calculating the most important word occurrence frequency with respect to the TF-IDF weight.

According to the conceptual network-based user profile composition method and system and the personalized query extension system using the same, the user's query can be compared with the user's profile and can be used for personalization of the search by recommending personalized query words.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in more detail with reference to the accompanying drawings. As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to the specific disclosed form, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Like reference numerals are used for like elements in describing each drawing. In the accompanying drawings, the dimensions of the structures are shown in an enlarged scale than actual for clarity of the invention.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. Singular expressions include plural expressions unless the context clearly indicates otherwise.

In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described on the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, parts, or combinations thereof.

Also, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

The present invention proposes a method for creating a user profile using a searched web document after using a search engine and using it as a dictionary for query expansion. Since the web document visited by the user is used, different profiles are configured for each user, and when used for query expansion, personal intention can be reflected in the search results.

In the present invention, to construct a user profile, a keyword is extracted using a table term frequency (TTF) technique from a web page visited by a user after using a search engine.

The most important word occurrence frequency technique is a method for extracting a keyword from a specific document set, using a term frequency (or word frequency) and an inverted document frequency (TF-IDF) weight.

TF-IDF weights are statistical measures that can be used to assess the importance of words appearing within a particular document in a document set.

TF (term frequency) is a value that indicates how often a particular word appears in a document. The higher the TF value, the more important the document can be considered. However, if the word itself is often used in a family of documents, this means that the word is common. This is called document frequency (DF), and the inverse of the DF value is called inverse document frequency (IDF). TF-IDF is the product of TF and IDF. The IDF value depends on the nature of the document family. For example, the word 'atoms' can be a key word in documents because of the high IDF value because they do not appear well among ordinary documents.However, in the case of a group of documents that contain atoms, the word becomes a common word. Other words that can be broken down into segments get high weights.

It is assumed that the above keywords and the words queried by the user point to a common concept, and the query expansion can be performed using the above keywords.

Next, a keyword extraction method using the most important word occurrence frequency technique will be described below, and a basic model constituting the personal profile will be described.

<Keyword extraction for search results>

In the present invention, in order to find a word having a high correlation with a user's query, a search engine searches for a specific word and extracts keywords from visited web pages. The most important word occurrence frequency is used to extract these keywords. The most important word occurrence frequency, given a document set, first extracts important words in a document using a TF-IDF weight. The important words extracted from each document can be duplicated and put into one table, and the number of occurrences is counted again. If the frequency of occurrence is the most important word occurrence frequency, the word with the high count value is regarded as a keyword in a given document set.

Hereinafter, a method for obtaining the TF-IDF weight used in the present invention and a method for obtaining the most important word occurrence frequency will be described.

<TF-IDF weights>

The TF-IDF weight is a weight used in information retrieval and text mining. It is a statistical value that indicates how important a word is in a particular document when there are a group of documents. It can be used to extract key words of a document, to rank search results in a search engine, or to obtain a degree of similarity between documents.

Words with relatively large TF-IDF weights are more important. In the present invention, the TF-IDF weight of the word i appearing in the document j is calculated by Equation 1 below.

는 아래의 수학식 2에 의해 정의된다. here,

Is defined by Equation 2 below.

는 단어 i가 문서 j에서 출현한 횟수고,

는 문서 dj에서 모든 단어가 출현한 횟수다.here,

Is the number of times word i appeared in document j,

Is the number of occurrences of every word in document dj.

Equation 1 is an expression for calculating the TF-IDF weight of the word i appeared in the document j, it is calculated as the product of the TF value and the IDF value.

First, the TF value means that a word with a high frequency is considered more important in a document, and a standardized value of dividing a frequency of a specific word by the total number of occurrences of all words is used as in Equation 2. In addition, the IDF value is used to mean that words appearing in fewer documents are more important, and are calculated using Equation 3 below.

는 문서 집합에 포함되어 있는 문서의 수이고,

는 단어 tj가 등장하는 문서의 수이다. here,

Is the number of documents in the document set,

Is the number of documents in which the word tj appears.

In the present invention, each web page visited by a user after querying a search engine is regarded as an individual document, and a single document set is constructed from the individual documents. After calculating the TF-IDF weight for this document set, the following Table Term Frequency (TTF) is applied.

<Table Term Frequency>

Using the TF-IDF weights described earlier, you can see which words are more important within the document. However, since TF-IDF weights do not represent a significant degree across the document set, the most important word occurrence frequency (TTF) is used to find the most important words in a given document set.

1A is a block diagram illustrating a conceptual network-based user profile configuration system according to an embodiment of the present invention. FIG. 1B is a block diagram for explaining an example of the keyword extracting unit shown in FIG. 1A.

1A and 1B, the conceptual network-based user profile composition system according to an embodiment of the present invention includes a keyword extractor 100 and a user profile generator 200.

The keyword extraction unit 100 includes a web document collection module 110, a single noun extraction module 120, a TF-IDF weight calculation module 130, and a most important word occurrence frequency calculation module 140. In this embodiment, the keyword extractor 100 is divided into a web document collection module 110, a single noun extraction module 120, a TF-IDF weight calculation module 130, and a most important word occurrence frequency calculation module 140. This is logically or functionally separated, not hardware.

The web document collection module 110 queries the query engine to the search engine, analyzes the result web page, and stores only the result web page selected by the user.

The single noun extraction module 120 extracts a single noun from the stored web pages.

The TF-IDF weight calculation module 130 calculates the TF-IDF weight. The calculation of the TF-IDF weight by the TF-IDF weight calculation module 130 will be described later.

The most important word occurrence frequency calculating module 140 calculates the most important word occurrence frequency based on the TF-IDF weight and selects a keyword.

For example, the most important word occurrence frequency may be calculated by extracting a higher specific ratio of words based on the number of words in the document. Then, the keyword is selected based on the calculated most important word occurrence frequency.

As another example, the most significant word occurrence frequency may be calculated by extracting a higher specific ratio of words based on the maximum TF-IDF weighting value in the document. Then, the keyword is selected based on the calculated most important word occurrence frequency. The calculation of the TF-IDF weight by the most important word occurrence frequency calculation module 140 will be described later.

The user profile generator 200 includes a session interest creation module 210, a comparison module 220, and a user profile update module 230. In this embodiment, the user profile creation unit 200 is divided into a session interest creation module 210, a comparison module 220, and a user profile update module 230, but this is logically or functionally divided into hardware. There is no distinction.

The session interest creation module 210 generates a session interest using the keywords extracted by the keyword extraction unit 100, and accumulates the generated session interest to create a user profile.

The comparison module 220 compares the generated session interest with the concept of the user profile whenever a new session interest is generated. The comparison module 220 compares the similarity between the concept of the session interest and the user profile using the web directory of the Open Directory Project (ODP). The comparison module 220 expresses each of the concepts of the session interest and the user profile as a term vector, and changes the term vector corresponding to the session interest into a first vector having a category of the ODP as a dimension. After changing the term vector corresponding to the concept of the user profile into a second vector having a dimension as a category of the ODP, the first vector and the second vector are compared using a cosine similarity.

The user profile update module 230 adds the concept of the session interest and the user profile to each other when the session interest is the same as or similar to the concept of the user profile according to the comparison result by the comparison module 220. Stored in the conceptual network 300, and if different, add as a concept of a new user profile and stored in the conceptual network 300.

FIG. 2A is a flowchart illustrating a method of calculating the most important word occurrence frequency in the entire document based on the TF-IDF weight calculated for each document.

Referring to FIG. 2A, a TF-IDF weight is calculated for each document (step S110). Here, each document is regarded as a separate document for each web page that the user visited after querying the search engine once.

2B is a table showing an example of a word and TF-IDF weight corresponding to each document.

Referring to FIG. 2B, the calculated TF-IDF weight of the word <movie> present in document number 1 is 0.5, the calculated TF-IDF weight of word <thriller> in document number 1 is 0.4, and the document number 1 The calculated TF-IDF weight of the word <open> is 0.1, the calculated TF-IDF weight of the word <movie> at document number 2 is 0.6, and the calculated TF-IDF weight of the word <thriller> at document number 2 is 0.4, and the calculated TF-IDF weight of the word <Korea> in Document No. 2 is 0.1.

The calculated TF-IDF weights show which words are more important in the document.

Returning to the description of FIG. 2A, only the most important words in each document, that is, words having a high TF-IDF weight are extracted (step S120). That is, in step S120, if only the top 60% of words are extracted based on the total number of words and the maximum TF-IDF weight, respectively, multiplying 0.4 by 0.4 instead of 0.6 extracts words larger than the product, and then shows the significant words among the words before appearing. Will take%. For example, as shown in Fig. 2B, since the total number of words in Document 1 is 4, 2.4 is calculated by the calculation of 4 * 0.6. Therefore, the word <movie> and the word <thriller> existing within the 2.4 ranking are extracted as shown in Fig. 2C as a word having a high TF-IDF weight.

On the other hand, as shown in Fig. 2B, since the total number of words in document 2 is 3, 1.8 is calculated by the calculation of 3 * 0.6. Therefore, the word <movie> existing within the 1.8 rank is extracted as shown in FIG. 2C as a word having a high TF-IDF weight.

In step S110, only a method of extracting a word at a specific ratio based on the number of words appearing in each document is used, but a method of extracting a word based on the maximum TF-IDF value in the document may also be used. For example, as shown in FIG. 2B, the largest TF-IDF value of document 1 is 0.5. Therefore, 0.2 is calculated by the calculation of 0.5 * 0.4. Therefore, <movie>, <thriller>, and <director>, which are words having a TF-IDF value greater than 0.2, are extracted as words having a maximum TF-IDF weight.

On the other hand, as shown in FIG. 2B, the largest TF-IDF value in document 2 is 0.6. Therefore, 0.24 is calculated by the operation of 0.6 * 0.4. Therefore, <movie> and <thriller>, which are words whose TF-IDF value is larger than 0.24, are extracted as words having the maximum TF-IDF weight.

Subsequently, the extracted words are put in one table, and the occurrence frequency is counted to calculate the most important word occurrence frequency (TTF) of the words (step S130).

Figure 2c is a table showing the frequency of the word most important words for each document.

Referring to FIG. 2C, the most important frequency of occurrence of the word <movie> is twice, and the most important frequency of occurrence of the word <thriller> is calculated once. In addition, corresponding to document 2, the most important frequency of occurrence of the word <movie> is twice, the most important frequency of occurrence of the word <thriller> is twice, and the most important frequency of occurrence of the word <director> is calculated once. .

In the above, we explained the process of extracting keywords using the most important word occurrence frequency and then extracting the words with the most important word occurrence frequency for the series of web documents that the user queries after searching the search engine.

Hereinafter, a personalized dictionary model, that is, a network-based personal profile model for query extension, will be described under the assumption that the user has a specific association among words having the most important frequency extracted in the above manner.

Network-based Personal Profile Model for Query Expansion

Query expansion is to get more accurate search result by adding another word that is closely related to the queried word as query word when querying with a word. To do this, you need a dictionary that defines the relationships between words. In addition, even if the same query is intended to be different for each individual, it is more reasonable to construct a dictionary for each individual.

Therefore, in the embodiment of the present invention, a network-based personal profile for query extension is created using a keyword extraction method for a query and used as a dictionary for query extension.

3 is a conceptual diagram illustrating a configuration model of a personal profile for query extension.

Referring to FIG. 3, among documents in which a user queries Query 1 for a search engine and returns a result, a document set is composed only for documents searched by a user by clicking a link, and the most important word occurrence frequency in the configured document set is shown. Use to extract keywords.

For example, the extracted keywords are keyword 1, keyword 2, and keyword 3, and keywords 1,2,3 and query 1 regard the concept 1 as the common points. In this relationship, concept 1 is defined as a concept that the words commonly point to.

4A is a conceptual diagram illustrating a conceptual network represented by a user profile. 4B is a conceptual diagram illustrating an example of query recommendation using a user profile. 4C is a conceptual diagram illustrating a conceptual network according to a query recommendation.

4A to 4C, the present invention creates a profile for each individual and utilizes it for personalization of a search. When expressing the interests of each individual in a user profile, the keywords shown in FIG. 4A are represented as a concept network. For example, the concept C1 is linked to the keywords Beer, Duff and alcohol and is defined by the three words mentioned above.

The user profile can be used directly for query recommendation. When the user inputs a specific word or alphabet, the personalized query expansion system according to the present invention finds a word starting with the specific word or alphabet in the user profile and uses the words connected to the concept connected to the word. Can extend the query.

For example, when the user enters 'b' as shown in FIG. 4B, the personalized query expansion system according to the present invention finds a word starting with the alphabet 'b' in the user profile. Here, the word starting with the letter 'b' is 'beer'. As shown in FIG. 4C, the word 'beer' is connected to Concept C1 and Concept C3. Accordingly, the personalized query expansion system according to the present invention combines words connected to concepts C1 and C3 as shown in FIGS. 4B and 4C to 'beer', 'beer Duff', 'beer alcohol', and 'beer drinking'. Extended words such as', 'beer Tavern', 'beer evening', 'beer Duff alcohol', 'beer Tavern drinking' and 'beer drinking evening' can be displayed.

5 is a conceptual diagram for explaining a session interest constituting a conceptual network.

Referring to FIG. 5, keywords are extracted and used for documents searched by a user using a search engine to create a user profile. For this purpose, session and session interest are defined and used. Here, the session is the time before the user queries the search engine and makes the next query. Session interest is also a major concern of a session.

One concept is defined by three words, for example Beer, Duff and alcohol. Beer is the word the user queries the search engine, and Duff and alcohol are the keywords extracted from the web documents that the user clicked on to view in the results page and then clicked to view it. .

Whenever a user inquires, these session interests are generated, and the generated session interests are accumulated to create a user profile.

6 is a conceptual diagram illustrating a series of procedures for configuring a user profile by accumulating session interests.

Referring to FIG. 6, when a new session interest is generated and compared with the concepts of pre-stored user profiles each time, a concept corresponding to the session interest and a concept of the user profile are combined and determined to be different. , As a concept. You can use a web directory called an open directory project to calculate the similarity between concepts. The Open Directory Project is a web directory where volunteers have categorized numerous web pages.

FIG. 7A is a conceptual diagram illustrating a hierarchy of categories in an open directory project, and FIG. 7B is a table illustrating an example of the number of categories and web pages according to the depth of the hierarchy.

Referring to FIG. 7A, in an open directory project, categories have a hierarchical structure. That is, under the Top category, the Arts category, the Business category, the Computers category, and the like are linked. The art category is linked to the animation category, the antiques category, the architecture category, and the like. The animation category is linked to the animated graphics category, the animation art gallery category, and the like. The animated graphics category is linked to the Adobe Flash category, the animated GIFs category, and the Artists category.

Referring to FIG. 7B, each web page is classified into a predetermined category. For example, at hierarchy depth 1 there are 89 web pages corresponding to 17 categories. At hierarchy depth 2, there are 6,427 web pages corresponding to 657 categories. In this manner, there are 661,485 web pages at tier depth 8 corresponding to 165,616 categories.

Each category and web page contains a title and description. Therefore, text data such as titles and descriptions recorded in categories and web pages of the Open Directory project can be utilized in the web page search.

8 is a conceptual diagram for explaining a dimension change for calculating the similarity between concepts in the ODP category dimension.

Referring to FIG. 8, each of a session interest and a concept of a user profile is first represented by a term vector (or a document vector).

The vector T _S is a term vector corresponding to the session interest, and the vector T ₁ is a term vectors corresponding to the first concept of the user profile. The vectors T _S and the vectors T ₁ are compared with each other being changed into vectors having ODP categories as dimensions. That is, the term vector T _S corresponding to the session interest is changed to the vector Ct having the category of the ODP as a dimension, and the term vector T ₁ corresponding to the first concept is the vector C having the category of the ODP as a dimension. Changed to ₁

FIG. 9 is a conceptual diagram for describing the dimension change of FIG. 8 in detail.

Referring to FIG. 9, in order to change the dimension of the term vector to the dimension of the ODP category, first, the ODP categories are also expressed as term vectors.

First, a document is generated for each ODP category by collecting strings of title and description items of each ODP category and a web page belonging to the corresponding ODP category.

The super documents of FIG. 9 are documents according to the corresponding ODP category, and the TF-IDF weights are again given to the corresponding documents and expressed as a term vector. The TF-IDF weight is a weight that shows which words are important in each document when there is a document set. The TF-IDF weight is calculated as the product of the term frequency and the inverse of the document frequency. Here, the term frequency is the number of occurrences of a word in a document, and the document frequency is the number of documents in which a word appears in a document set.

In FIG. 9, Tct represents a concept expressed as a term vector, and T ₁ , T ₂ , T ₃ , and T _N represent a category expressed as a term vector. Similarity between each ODP category and concept can be calculated through cosine similarity. Cosine similarity is a method of measuring similarity between documents using a vector model which is used most frequently in the field of information retrieval.

Referring to FIG. 9, the similarity between the term vector T ₁ corresponding to category 1 and the term vector Tct corresponding to the concept is cos (T ₁ , T _Ct ), and the term vector T ₂ corresponding to category 2 is _shown. ) And the similarity between the term vector Tct corresponding to the concept is cos (T ₂ , T _Ct ), and the similarity between the term vector T ₃ corresponding to category 3 and the term vector Tct corresponding to the concept is cos ( T ₃ , T _Ct ). In a similar manner, the similarity between the term vector T _N corresponding to category N and the term vector Tct corresponding to the concept is cos (T _N , T _Ct ).

Therefore, conceptual similarity in the category dimension is expressed as Equation 4 below.

As such, the concepts are calculated with similarity with each category represented by the term vector, and represented as a vector having the calculated similarity value as the dimension value of each category. The concepts can now be calculated at the category level.

10A and 10B are conceptual diagrams for describing a keyword extraction method.

As shown in Fig. 10A, keywords are extracted from web pages (documents) that the user has searched. The document set is composed of documents searched by the user in one session, and then the TF-IDF weight is calculated to selectively extract important words in each web document based on the weight. As shown in FIG. 10A, the document set, the TF-IDF weight calculation, and the selective extraction of the important words have been described in steps S110 and S120 described in FIG. 2A.

Subsequently, as described in step S130 of FIG. 2A, the words extracted from each document are put into one table and the number of appearances is calculated again. For example, as shown in FIG. 10B, the number of appearances is counted in a temporary table of the left table and described in a term frequency table of the right table.

The concept of session interest is constructed using only words having a number of appearances above a predetermined threshold based on the number of appearances. In FIG. 10B, the session interest Cs that is configured is defined by three words, such as beer, alcohol, and duff.

11 is a block diagram illustrating a personalization query extension system according to an embodiment of the present invention. In particular, a personalized query expansion system based on user profile construction techniques is shown.

Referring to FIG. 11, the personalized query expansion system according to an embodiment of the present invention includes a client unit 400 and a server unit 500.

The client unit 400 includes a query input module 410, a viewer module 420, and a keyword extraction module 430. The client unit 400 receives a user input and extracts a keyword from a searched web page to create a user profile. The server unit 500 transmits. In the present embodiment, the client unit 400 is divided into the query input module 410, the viewer module 420, and the keyword extraction module 430, but this is not logically or functionally divided.

The query input module 410 receives a query input by the user and provides the query to the server 500, and selects query terms selected by the user from among the extended query words provided from the server 500. To provide.

The viewer module 420 displays documents provided as documents are provided by the server unit 500.

The keyword extraction module 430 extracts a keyword (or short term concept network) from the document of interest selected by the user, and provides the extracted keyword to the user concept network management module 530. The keyword extraction module 430 may include components included in the keyword extraction unit described with reference to FIG. 1. Detailed description thereof will be omitted.

The server unit 500 includes a query expansion module 510, a search module 520, and a user concept network management module 530, and configures and manages a user profile. In the present embodiment, the server 500 is divided into a query expansion module 510, a search module 520, and a user concept network management module 530. no.

As the query is provided from the client unit, the query expansion module 510 queries the conceptual network in response to the provided query and obtains extended query terms and provides them to the client unit.

As the query is selected by the client unit, the search module 520 requests documents by providing documents corresponding to the selected query to the network search engine, and provides the client unit 400 with a document corresponding to the request. do.

The user concept network management module 530 provides the concept network 700 as a keyword (or a short term concept network) is provided from the client unit 400.

In FIG. 11, the query input module 410, the viewer module 420, and the keyword extraction module 430 are provided in the client unit 400, but the query input module 410 and the viewer module are described. At least one of the keyword extraction module 430 may be provided in the server unit 420.

12 is a flowchart illustrating a personalization query expansion method according to an embodiment of the present invention.

Referring to FIG. 12, when a user inputs a new query (step S210), a session interest is generated by query words and keywords extracted from web pages (step S220).

Next, the similarity between the term vector of the session interest and the term vectors of the ODP category is calculated (step S230). Here, the similarity is made by calculating the cosine similarity.

Subsequently, a session interest expressed by the category vector is generated (step S240).

Then, the similarity between the concept of the session interest and the concept of the pre-stored user profile is calculated and compared with each other (step S250). In this comparison, if the concept of session interest and the concept of user profile are the same or similar, the two concepts are put together and stored in the concept network, and if it is judged that the concept of session interest and the concept of user profile are different, the concept as a new concept. It is added to the network (step S260).

Next, an experimental example for verifying whether the most important word occurrence frequency is effective to use as a method for extracting a keyword and a correlation between a keyword queried by a user and a keyword extracted from a visited web page will be described.

Experimental Example

First, we searched the search site using specific words, and then we extracted the keywords using the most important word occurrence frequency and compared them with the query words. This experiment was conducted to verify the performance of the most important word occurrence frequency by constructing a document set with well-defined web documents and verifying that keywords extracted using the most important word occurrence frequency match the subject of the document set. Then, we added the extracted keywords as a query and checked whether the query showed better search results.

- Experiment environment

The search site used in the experiment is Google, which implements a series of experiments using the Java language to download web documents obtained from the search results and calculate the frequency of the most important word occurrences. In order to calculate the TF-IDF weights, the number of occurrences of words is counted a lot. The database management system (DBMS) is used for efficient processing. Here, the database management system (DBMS) used MySQL. We use the Korean Language Technology (KLT) library to extract a single noun from the collected web documents. The KLT library provides index word extraction and stemming functions. On the other hand, English is processed using a natural language processing tool such as Standard POS Tagger.

-Keyword extraction section for query

In this experiment, as shown in FIG. 1, a prototype keyword extracting unit is configured to collect web documents searched after a user queries and extract keywords. In order to store the web documents collected by the keyword extraction unit described in FIG. 1 and extract the keywords, a table for document sets, documents, dictionaries, word appearance facts, IDF, TF-IDF, and most important word occurrence frequency is created. . Next, the keyword extractor extracts keywords for the user query in the following order.

Web document collection for queries and word facts

The web document collection module 110 of FIG. 1 first receives a query word and queries the search engine 50 for the word to bring a search result. After the experimenter selects only the web document that meets his intention from the search results shown by the web document collection module 110, the web document collection module 110 collects the web documents selected by the experimenter and stores them in the document repository. At this time, the web documents are all assigned the same document set number, and each is stored as one document.

The stored documents are processed through preprocessing such as removing non-Hangul characters and removing HTML tags, and then extracting a single noun using the KTL library. The extracted nouns are stored in the word appearance fact information repository.

-TF-IDF weight calculation and most important word occurrence frequency calculation

IDF is calculated first using word occurrence facts, and then TF-IDF is calculated. The IDF and the TF-IDF values are calculated and stored in a database management system using simple SQL statements using 'INSERT' and 'SELECT' statements. The process of calculating the TF-IDF is a simple operation such as counting and adding the number of occurrences of words, and thus can be easily calculated using a database management system.

After calculating the TF value and TF-IDF value by using the SQL statement, and extracting a word that is considered to be important in the document at a certain rate based on the calculated TF-IDF weight, the number of times each word is extracted is counted. Calculate the frequency of significant word occurrences.

-Comparison experiment between query word and keyword

In order to verify the performance of the keyword extraction technique used for personalized query expansion according to the present invention, an experiment was conducted to measure and compare keywords in a web document obtained by querying a specific query.

The query used 'movie', 'thriller', 'action', etc. When querying, 'thriller' or 'action' must include both 'movie' and 'thriller' or 'action'. 'Movie' can be searched, and the keyword is extracted using the most important word occurrence frequency for the entire search result obtained by this query and shows the result. In order to confirm the number of documents needed to extract the appropriate keywords, the number of documents included in the document set for extracting keywords is changed and the result is checked.

-Query Expansion Experiment

In the present invention, a personalized profile model for query expansion is proposed. In this model, it is assumed that query terms are related to one or more concepts. In the query expansion experiment, we use the keywords extracted by the keyword extraction unit to check whether the meaning of words can be properly limited to a single concept. The experimental method is shown in Figs. 13A and 13B.

13A is a conceptual diagram illustrating a searched result screen according to a conventional query method, and FIG. 13B is a conceptual diagram illustrating a searched result screen by a query expansion experiment according to the present invention.

Referring to FIG. 13A, when a query engine is generally used using the word A, the search engine includes a non-expanded query and performs a search and provides the search result to the user. In this case, since the query is not expanded, the search result screen provided to the user is mixed with unwanted results. Therefore, since the user is provided with a search result that he / she does not want, the user has to find a search result that he / she wants while performing a KEPCO search or perseverance for unnecessary search results.

However, referring to FIG. 13B, only a desired search result is displayed on the search result screen provided to the user.

In other words, for query expansion experiments, we first select the query subject. In this experiment, words such as 'stock,' 'music,' and 'North Korea' were selected as query words. The concept of each word is defined and the documents are collected, 'value investment' for 'stock', 'indie music' for 'music', and 'human rights issue' for 'North Korea'. Decided.

Next, extract the keywords by collecting the web documents for each abstract concept. Query these keywords and each query together to see if the subject of the web documents being retrieved matches the concepts you have specified.

In these experiments, the concepts of 'value investment', 'indie music', and 'human rights problem' assigned to each query can be seen as the correct subjects to be searched by the documents searched after the query expansion. We can see that the query expansion using keywords is appropriate.

In this experiment, the document set for each concept consisted of four web documents, and the words with the most important word frequency values were selected as keywords in the document set.

<Experiment Result>

-Keyword Extraction Result by Query

14A and 14B show the results of extracting a keyword from a web document obtained as a result of a search using words such as 'movie', 'thriller', and 'action' as a query word.

14A and 14B are tables for explaining keyword extraction results for movie, action, and thriller queries.

Referring to FIGS. 14A and 14B, keywords extracted while changing the number of web documents constituting a document set from 2 to 100 are presented. 14A and 14B are ranked according to the most important word occurrence frequency values.

As a result, when we composed a document set with 8 or more collected web documents and calculated the most important word occurrence frequency, the words 'movie', 'action', 'thriller' and etc. used as query words got the top rank.

The reason that 'action' is missing from keywords extracted using 2, 4, or 6 web documents is to include 'action' or 'thriller' when querying the search site. This is because only the thriller is included and the action is not included. These results show that keywords can be effectively extracted even for a document set composed of a small number of web documents.

-Query Expansion Experiment

First, keywords extracted from web documents corresponding to each concept are shown in FIG. 15.

15 is a table for explaining keywords extracted from a web document set for each concept.

 Referring to FIG. 15, in the case of <value investment>, <investment>, <market>, <long-term>, <stock>, <value investment>, and <investor> have the largest value of the word occurrence frequency. And extracted with keywords. From the above keywords, it can be seen that <value investment> is related to <long term> investment.

16 shows the ratio of the search results when the concepts are queried with the designated query.

16 is a table for explaining a ratio of results belonging to a designated concept after query expansion.

Referring to FIG. 16, as a result of querying the word <stock> with a keyword related to <value investment>, the results of all the search results matched the concept of <value investment>.

Similarly, the same result was obtained when the word <North Korea> was expanded using a keyword extracted from a web document related to <Human Rights>.

In addition, in the case of <Music>, a small number of results did not belong to the specified concept, and the number of search results was 31, which is because too many extracted keywords resulted in too many queries. The reason why there are many extracted keywords is that many words have the same highest word occurrence frequency.

In the experiment according to the present invention, the keyword was extracted from a document set consisting of a small number of web documents and the results were verified to verify the performance of the keyword extraction method using the most important word occurrence frequency.

Although described above with reference to the embodiments, those skilled in the art can be variously modified and changed within the scope of the invention without departing from the spirit and scope of the invention described in the claims below. I can understand.

As described above, according to the present invention, as a method of finding a conceptual network which is a basic model of a personal profile for use as a dictionary used for query expansion, and a basic correlation for constructing the model, in a web document search, A keyword extraction method using an important word occurrence frequency is provided.

Using the most important word occurrence frequency calculated based on TF-IDF weights, we can find important keywords in a given document set. Also, the keywords found by constructing a document set from the web documents visited after the user queries the search engine can find words related to the user's query.

In this way, the association between the words found and the query can be used to construct a user profile based on the conceptual network. When a user actually uses a search engine, the user visits only a small number of search results, so that a meaningful keyword can be extracted from a document set consisting of a small number of documents.

1A is a block diagram illustrating a conceptual network-based user profile configuration system according to an embodiment of the present invention.

FIG. 1B is a block diagram for explaining an example of the keyword extracting unit shown in FIG. 1A.

FIG. 2A is a flowchart illustrating a method of calculating the most important word occurrence frequency in the entire document based on the TF-IDF weight calculated for each document.

2B is a table showing an example of a word and TF-IDF weight corresponding to each document.

Figure 2c is a table showing the frequency of the word most important words for each document.

3 is a conceptual diagram illustrating a configuration model of a personal profile for query extension.

4A is a conceptual diagram illustrating a conceptual network represented by a user profile. 3B is a conceptual diagram illustrating an example of query recommendation using a user profile. 3C is a conceptual diagram illustrating a conceptual network according to a query recommendation.

5 is a conceptual diagram for explaining a session interest constituting a conceptual network.

6 is a conceptual diagram illustrating a series of procedures for configuring a user profile by accumulating session interests.

FIG. 7A is a conceptual diagram illustrating a hierarchy of categories in an open directory project, and FIG. 7B is a table illustrating an example of the number of categories and web pages according to the depth of the hierarchy.

8 is a conceptual diagram for explaining a dimension change for calculating the similarity between concepts in the ODP category dimension.

9 is a conceptual diagram for explaining the dimension change in detail.

10A and 10B are conceptual diagrams for describing a keyword extraction method.

11 is a block diagram illustrating a personalization query extension system according to an embodiment of the present invention.

12 is a flowchart illustrating a personalization query expansion method according to an embodiment of the present invention.

13A is a conceptual diagram illustrating a searched result screen according to a conventional query method, and FIG. 13B is a conceptual diagram illustrating a searched result screen by a query expansion experiment according to the present invention.

14A and 14B are tables for explaining keyword extraction results for movie, action, and thriller queries.

15 is a table for explaining keywords extracted from a web document set for each concept.

16 is a table for explaining a ratio of results belonging to a designated concept after query expansion.

<Description of the symbols for the main parts of the drawings>

100: keyword extraction unit 110: Web document collection module

120: single noun extraction module 130: TF-IDF weight calculation module

140: most important word occurrence frequency calculation module 200: user profile creation unit

210: session interest creation module 220: comparison module

230: user profile update module 400: client unit

410: query input module 420: viewer module

430: keyword extraction module 500: server unit

510: query extension module 520: search module

530: User Concept Network Management Module

Claims (18)

Extracting a keyword from documents searched by a user using a search engine to create a user profile; Generating a session interest using the extracted keywords, and accumulating the generated session interest to create a user profile; Comparing the concept of the created session interest with the user profile each time a new session interest is created; And If the session interest is the same as or similar to the concept of the user profile, adding the concepts of the session interest and the user profile to each other, and if different, adding as a concept of a new user profile; How to Configure User Profiles. The method of claim 1, wherein the similarity comparison between the session interest and the concept of the user profile is made using a web directory of an Open Directory Project (ODP). The method of claim 2, wherein the comparing of the concept of the session interest and the user profile comprises: Expressing each of the concepts of the session interest and the user profile in a term vector; Changing each of the term vectors corresponding to the session interest and the term vectors corresponding to the concept of the user profile into first and second vectors having a category of the ODP as a dimension; And And comparing the first vector with the second vector using a cosine similarity. The method of claim 3, wherein the changing to the first vector and the second vector comprises: Generating a super document for each category by collecting strings of each category corresponding to the session interest and web pages belonging to the category; Expressing the super documents as term vectors weighting term frequency-inverse document frequency (TF-IDF); And Calculating a similarity with each category expressed as a term vector and expressing the similarity value as a vector having the dimension value of each category, Comparing the first vector and the second vector And comparing the term vector corresponding to the concept and the term vector corresponding to the category in a category dimension by using cosine similarity. The method of claim 1, wherein the extracting of the keyword comprises: Extracting words in each web document based on the TF-IDF weights; Storing the words extracted from each document in one table and calculating the number of appearances; And And constructing a session interest using only words having a number of appearances above a predetermined threshold based on the number of appearances. A keyword extraction unit for extracting keywords from documents searched by a user using a search engine to create a user profile; A session interest creation module which generates a session interest using the extracted keywords and creates a user profile by accumulating the generated session interest; A comparison module for comparing the concept of the generated session interest with the user profile each time a new session interest is generated; And And a user profile update module for adding the concept of the session interest and the user profile to each other when the session interest is the same as or similar to the concept of the user profile, and adding the concept of the new user profile as a difference. Conceptual network-based user profile configuration system. The system of claim 6, wherein the comparison module compares the similarity between the session interest and the concept of the user profile using a web directory of an open directory project (ODP). The method of claim 7, wherein the comparison module, Represent each of the concepts of the session interest and the user profile as a term vector, Changing a term vector corresponding to the session interest into a first vector having a category of the ODP as a dimension; After changing the term vector corresponding to the concept of the user profile into a second vector having dimensions as a category of an ODP, And a conceptual network based user profile composition system using cosine similarity to compare the first vector and the second vector. The method of claim 6, wherein the keyword extraction unit, Extract words from each web document based on the TF-IDF weight, After storing the words extracted from each document in one table, calculating the number of appearances, The concept network-based user profile composition system of claim 1, wherein the session interest is configured using only words having the number of appearances above a predetermined threshold based on the number of appearances. The method of claim 6, wherein the keyword extraction unit, A web document collection module for querying a query engine on a search engine and analyzing and storing a web page as a result; Single noun extraction module for extracting a single noun from the stored web pages; A TF-IDF weight calculation module for calculating a TF-IDF weight; And Conceptual network-based user profile configuration system comprising a most important word occurrence frequency calculation module for selecting a keyword by calculating the most important word occurrence frequency for the TF-IDF weight. The method of claim 10, wherein the most important word occurrence frequency calculation module A concept network-based user profile composition system characterized by extracting words of a specific ratio based on the number of words in a document, calculating the most important word occurrence frequency and selecting keywords. The method of claim 10, wherein the most important word occurrence frequency calculation module A conceptual network-based user profile composition system characterized by extracting words of a specific upper ratio based on the maximum TF-IDF weight in a document, calculating the most important word occurrence frequency and selecting keywords. Concept network; A query extension module for querying the concept network corresponding to the provided query word to obtain extended query terms and providing the obtained extended query term to the client portion as a query is provided from a client portion; A search module for requesting a search by providing documents corresponding to the selected query to the search engine as the query is selected by the client unit, and providing a document that is a result corresponding to the request to the client unit; And As the session interest is provided from the client unit, the session interest is compared with the concept of the user profile stored in the conceptual network, and if the session interest is the same as or similar to the concept of the user profile, the session interest And a user concept network management module for adding the concepts of the user profile to the concept network by adding them together and, if different, adding them as concepts of a new user profile to the concept network. The method of claim 13, wherein the client unit, A query input module for receiving a user input and extracting a keyword from the searched web page and providing the keyword to the query expansion module; A viewer module for displaying documents provided as documents are provided by the search module; And And a keyword extraction module for extracting keywords from the document of interest selected by the user and providing the extracted keywords to the user concept network management module. The method of claim 13, wherein the keyword extraction module Extract words from each web document based on the TF-IDF weight, After storing the words extracted from each document in one table, calculating the number of appearances, And a session interest using only words having the number of appearances above a predetermined threshold based on the number of appearances. The method of claim 13, wherein the keyword extraction module, A web document collection module for querying a query engine on a search engine and analyzing and storing a web page as a result; Single noun extraction module for extracting a single noun from the stored web pages; A TF-IDF weight calculation module for calculating a TF-IDF weight; And And a key word frequency calculation module for calculating a key word frequency with respect to the TF-IDF weight to select a keyword. The method of claim 16, wherein the most important word occurrence frequency calculation module A personalized query expansion system, comprising: extracting a word having a high specific ratio based on the number of words in a document, calculating a frequency of the most important word and selecting a keyword. The method of claim 16, wherein the most important word occurrence frequency calculation module A personalized query expansion system, comprising: extracting a high specific ratio of words based on the maximum TF-IDF weight in a document, calculating the most important word occurrence frequency, and selecting a keyword.

Images