KR20030082109A - Method and System for Providing Information and Retrieving Index Word using AND Operator - Google Patents

Abstract

PURPOSE: An index word search and information providing service system and method is provided to extract index words, to calculate a frequency of each index word included in specific documents, to calculate scores for the documents based on the calculated frequency, and to output the documents from the highest to the lowest score. CONSTITUTION: An index agent receives document data such as contents and URLs from a robot agent via an FA(Foreign Agent)(S510). The index agent extracts index words from the received document data, and calculates a frequency of each index word in a document(S520). The index agent requests the robot agent to transmit a next document(S530), and the robot agent checks whether the requested document is the last in a URL database(S540). If the requested document is not the last, the step S520 is repeated. Otherwise, the frequency of each index word is calculated for all the document, all the documents including the index words are extracted and simultaneously, scores for the extracted documents are calculated based on a score calculation equation(S550). The data on the documents including the index words is stored at an index database(S560).

Description

Method and System for Providing Information and Retrieving Index Word using AND Operator}

The present invention relates to an index word search and information providing system and method using an AND operator, and more particularly, extracts an index word to be searched, calculates the frequency of occurrence of each index word in a specific document, and calculates the frequency of occurrence of each index word. The present invention relates to an index word search and information providing system and method using an AND operator that calculates a value multiplied with each other as a score of a corresponding document and provides search results in the order of documents having a high score.

In recent years, as most documents are written on a computer and documents are distributed and acquired through a communication network, the importance of finding a document effectively becomes very important. Moreover, with the spread of the Internet, it is common for not only experts but also ordinary people to access and provide information through communication networks, and accordingly, the amount of information accessible through the Internet is increasing exponentially. As a result, search engines (eg, altavista, yahoo, infoseek ultra, dejanews, lycos, empas, etc.) are becoming the most successful applications on the Internet, a huge information warehouse and information acquisition infrastructure that is unprecedented in history.

Since the early Internet search engines were not large in size, there was no need to build a few databases into a database. Early web search engines such as Yahoo used a topic search method that was convenient for development and search when the database was small. It was. For example, the menu of each stage, including the initial menu will have about 10 sub-menu and the full menu is a total of 1000 (10 ³⁾ nd appear in a tree structure format it, assuming supported up to step 4 It can hold as much data. If you add one more step, you can have up to 10000 (10 ⁴ ) pieces of data. However, since the current number of records of Internet search engines ranges from as little as one million to as many as 50 million, the final data can only be accessed through several stages when searching through the subject search method. If you make a mistake in any of the steps, it is impossible to retrieve data from subtopics unless you go back to the parent topic. As the size of the Internet continues to grow, it is no longer possible to search smoothly by just searching the topic, and the number of records held by the search engine must increase according to the size of the rapidly expanding web. Checking and adding it as a record cannot keep up with the rapid growth of the web, and even if you build a database by indexing hundreds of thousands of homepages by this manual work, it takes a lot of time and effort for users to search it menu-wise. You should pay attention.

Since then, the concept of robot (eg robots, wanderers, spiders, worms, etc.) agents has been introduced to the Internet. Robots are a type of automatic traversal program that automatically searches and indexes existing homepages. It is a program that makes a database. Most of the databases created by these robots are designed to be index-searchable, and from then on, Internet search engines began to switch from topic search to index search. That is, a user inputs a corresponding search expression as an index word to search for information desired by the user, and accesses related information by a Boolean query method or a vector query method using the relationship between the input index words.

In the related art, a search is performed on an index of index words constructed in a system in consideration of a relationship between input index words (ie, weights between index words, etc.) and provides corresponding information to a user. The method of accessing the desired information in consideration of the relation between the index words is to pre-analyze and store the morphemes of the unit index words, extract the index word related to the index word, and the bigram or trigram of the search index word. Finding index words using information Using this, we have developed a search engine to obtain the information that Internet users want quickly. Search engines navigate the Internet at high speed on behalf of Internet users, looking for information that meets their needs. In other words, the Internet user tells what information he / she wants in accordance with the index word format provided by the search engine, and the search engine finds the information and provides it to the Internet user.

However, such a conventional information retrieval method using a search engine has been developed with a focus on processing speed and stability. Firstly, in order to calculate the order of the search results using the AND operator, a number of methods are used. The accuracy is inferior because it calculates the frequency of occurrence in a specific document among the index words and scores the document based on the index word with the smallest frequency of occurrence of each index word. Difficult to search For example, when the frequency of occurrences of two specific index words a and b in any document are tf _a , tf _b and tf _a <tf _b , then the score of that document for a particular index word is a word with a low frequency of appearance, tf _a is the score of the document. Therefore, when searching for information by two or more index words using the AND operator, since the document score is calculated based on a word having a low occurrence frequency, it is difficult to calculate an accurate search result. Second, when many Internet users request the search at the same time, the search efficiency decreases as the search time and response time become longer.

Accordingly, the present invention has been made to solve the above-mentioned general problems, and an object of the present invention is to calculate the frequency of appearance in a particular document among a plurality of index words in calculating the order of search results using the AND operator. The present invention provides an index word search and information providing system and method using an AND operator that calculates and multiplies the frequency of occurrence of each index word and calculates a value as a score of a corresponding document.

Another object of the present invention is to provide a system and method for index word search and information using an AND operator that reduces the searching time and provides a highly accurate search result by precomputing and databaseting the frequency of occurrence of all index words in a specific document. There is.

1 is a block diagram showing the overall configuration of the index word search and information providing system according to the present invention,

2 is an exemplary diagram of a table structure showing an example of a score for a corresponding document for each index word stored in an index DB according to the present invention;

3 is an exemplary diagram of a table structure showing document information for each index word according to the present invention;

4 is a flowchart illustrating a document collection process according to the present invention,

5 is a flowchart illustrating a document score calculation process for each index word according to the present invention;

6 is a flowchart illustrating an index word processing process input by a client according to the present invention;

7 is an exemplary view illustrating an example of calculating a document score including all index words according to the present invention.

♣ Explanation of symbols for the main parts of the drawing ♣

10: Internet 20: Robot Agent

30: index agent 40: search agent

50: client 100: information providing system

In order to achieve the above object, the present invention analyzes an index word requested by a client and lists the desired information to provide the client with an index word search and information providing method. The method includes accessing a URL on the Internet based on registered URL address information. Collecting the document and storing URL information including the URL address where the document is located and the contents of the document; Receiving the URL information to calculate the index word and the appearance frequency of the corresponding document, calculating a score of the corresponding document for the index word using a scoring formula, and storing document information for each index word; And receiving the document information for each index based on the index word input by the client using the AND operator, multiplying the scores of the corresponding documents for each index word, and providing the client with a search list in high order. It is characterized by.

In addition, the present invention is an index word search and information providing system that analyzes the index words requested by the client to list the desired information to provide to the client, the system automatically collects the various information posted on each home page while traveling the web server on the Internet And a robot agent which indexes and makes a database; Interconnected with the robot agent, index the documents collected by the robot agent to extract the index word, calculate the frequency of occurrence of each index word in the document, and calculate the score of the corresponding document for the adaptation word using a scoring formula An index agent for storing the calculated document information in an index DB; And the index agent, and analyze the index word input by the client, extract the document containing each index word by referring to the index DB, and multiply each document's score for each index word stored in the index DB It characterized in that it comprises a search agent for providing the client by creating a search list in the order of high score.

Hereinafter, exemplary embodiments of an index word search and information providing system and method using an AND operator according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing the overall configuration of the index search and information providing system 100 (hereinafter, abbreviated as "information providing system") according to the present invention, the information providing system 100 is the Internet (10), It consists of a robot agent 20, an index agent 30, a search agent 40, and a client 50. In addition, the robot agent 20 may refer to the collection robot 22, the tracking robot 24, the management robot 26, the facilitator (hereinafter abbreviated as 'FA') 28 and the URL database 29. Index agent 30 includes FA 32, index module 34, and index database 36, and search agent 40 includes FA 42 and search server 44.

The robot agent 20 is connected to the Internet 10, and the robot agent 20 automatically searches and indexes the homepage instead of manually searching the homepage by the automatic traversal program and performs indexing. Perform the function. The robot agent 20 collects new information and collects a robot 22 and a tracking robot 24 which collects information by tracking the change of contents of existing information, and prevents the collection of duplicate information and stores optimal information. And a management robot 26 for managing the URL database 29, and an FA 28 for managing the communication between the agents and managing the related agents. The URL DB 29 stores the contents of a specific document and URL information where the document is located. In addition, the tracking robot 24 may include a robot that searches for a new Internet homepage (unregistered homepage), a robot that tracks a changed homepage (including a deleted homepage), and a robot that tracks a homepage that fails to access later. Can be. The robot agent 20 eventually appears to function like a web browser in terms of the functional aspects of accessing a web server and importing data, but the web browser shows the imported data on the screen and performs functions such as hyperlinks. The agent 20 has a difference in that it analyzes data, extracts a URL therein, and connects to another URL. Therefore, when the robot agent 20 is executed, the robot agent 20 automatically finds a homepage on the Internet and collects information. Representative robot agents 20 include spiders and crawlers. The robot agent 20 reads all the contents of the web page visited by itself, visits all the linked sites in order, visits the sites visited in the past periodically for a period of time, and checks whether the corresponding page is updated or not. If yes, collect that information. If there is a web document linked to where the robot agent 20 has visited, the robot automatically enters there to collect information, and if there is no linked web document, the robot agent 20 designates as a starting point to search from there Index collection takes place.

The index agent 30 is interconnected with the robot agent 20, extracts index words by indexing documents collected by the robot agent 20, calculates the frequency of occurrence of each index word in the corresponding documents, and The score of the document is calculated and stored in the index database 36. This function is performed by the index module 34. The indexing operation performed by the index module 34 extracts an index word from the collected information, a scoring operation that calculates a score of a corresponding document for the extracted index word, and indicates the location of the information about the index word. It consists of indexing operations that provide a foundation for Here, each of the FAs 28 and 32 is in charge of the communication between the robot agent 20 and the index agent 30. The index module 34 removes the tag (e.g., HTML tag) of the information (e.g., HTML document) transmitted from the robot agent 20 through the FA 32, and selects an index word with the contents of the remaining information. It extracts and stores the score of the document for each index word in the index database 36. Here, it is preferable that the index word does not include adverbs and verbs, and the index word is mainly composed of nouns, adjectives, or noun forms of verbs. The method of calculating the score of the corresponding document for each index word will be described later in detail.

The search agent 40 is interconnected with the index agent 30, analyzes index words input by the client 50, extracts documents containing each index word by referring to the index DB 36, and index DB ( 36) It multiplies the scores of the corresponding documents for each index word stored in each other and provides the search results in the order of the high scores of the documents. This function is performed by the search server 44. The search server 44 includes a query analyzer 44a and a query analyzer 44a that perform tasks such as inputting a query, validating a query, and analyzing a query. Thread manager 44b and thread manager that perform tasks such as creating a thread, transforming a query, analyzing and generating results, and integrating and reranking results. And a result generator 44c for determining the information processed by 44b in ascending order in the form of a list according to the URL, title, accuracy, and the like to generate a final information search list. In particular, the thread manager is programmed to multiply the scores of the corresponding documents for each index by each other to produce the final search results for each document with the highest score.

The client 50 is a computer used by ordinary Internet users to access the search agent 40 of the present invention, and means a web browser or other client software such as Netscape, Internet Explorer, etc. installed in the computer. . An internet user may access the search agent 40 using a web browser and input an index word to receive desired information. The client 50 also includes a user interface, which allows the user interface to store a query from an Internet user in a buffer so that the search server 44 can process it, and to store the result buffer for the query with the Internet user. To browse. As mentioned above, communication between each agent is handled by each FA 28, 32, 42.

2 is an exemplary diagram of a table structure showing an example of a score for a corresponding document for each index word stored in an index DB according to the present invention. Before describing the index table for each index word, the number of documents stored in the URL DB of the robot agent is limited to five, and the index word extracted from each document is limited to that shown in the table. This is limited to facilitate the description of the present invention, and of course, it is actually made up of a plurality of documents and a plurality of index words.

First, the indexing module of the indexing agent receives a document to be indexed through the FA from the robot agent and extracts an index word of the corresponding document. As mentioned above, index terms should not include adverbs and verbs, but should be constructed around nouns, adjectives, or verb nouns. As shown in FIG. 2, the index word may be extracted in the order of appearance, but may be extracted in the order of the highest frequency of appearance in a specific document. The index module extracts an index word of a specific document, grasps the number of occurrences of the index word in a specific document, calculates a score of the document for each index word using a normalized scoring formula, and stores the index word in the index DB. In addition, the index DB stores the document information including the index word. The scoring formula preferably normalizes the frequency of index words to values between 0 and 1. Here, one example of the scoring formula is as shown below.

[Equation 1]

Here, tw is an abbreviation of 'term weight', the relative importance of a specific index word for any document, and TF is an abbreviation of 'Term Frequency', which means any document at the maximum occurrence frequency of a specific index word for the whole document. IDF is an abbreviation of 'Inverse Document Frequency', and is the importance of a specific index in the entire document. The less frequent the occurrence of a particular index in the entire document, the higher the IDF value. Also, tf is the frequency of a specific index word in any document, max tf is the maximum frequency of a specific index word in the entire document, N is the total number of documents to be indexed, and n is the number of documents in which the specific index word appears in the entire document. It is a number. Here, max tf may be set to arbitrary reference values (for example, 10, 50, 100, etc.). The scoring formula mentioned above is just one example, and of course, you can score a particular document for each index word.

For example, the scoring formula described above is used to score Document 1 for the index term 'car'.

The value calculated above (i.e., '0.041') is Document 1's score for the index word 'Car', and after the Index module calculates Document 1's score for the index word 'Car', the Document 1 score for the remaining index terms. Are calculated respectively. Next, the index module calculates the score of the corresponding document for each index word like document 2 to document 5 and then stores the score for the corresponding document for each index word calculated in the index DB. The score of each document for each index word is as shown in FIG. 2, and the higher the number of documents including the index word in the entire document is assigned. The score for the document by index word stored in the index DB is used when listing the search results for the search index word entered by the client.

3 is an exemplary diagram showing a table structure showing document information for each index word according to the present invention. When the index module calculates the score of the corresponding document for each index word in the process of FIG. 2, the index module stores the document information for each index word as a database. Here, the document information for each index word includes a document including the index word, a score of the document including the index word, and a URL address where the document is located. For example, the index module extracts all documents (eg, document 1, document 2, document 4, document 5) including a specific index word (eg, a car) from all documents, and the corresponding index word document calculated in FIG. 2. The document information for each index word is stored in the index DB based on the score (eg, 0.041, 0.020, 0.081, 0.012). As shown in FIG. 3, it can be seen that the fewer documents containing a specific index word, the higher the document score.

Hereinafter, an operation relationship of an index word search and information providing system and method using an AND operator according to the present invention will be described in more detail with reference to the accompanying drawings.

4 is a flowchart illustrating a document collection process according to the present invention, and assumes that the robot agent tracks a URL to be searched using a tracking robot and stores it in a URL DB.

First, the information providing system extracts a URL stored in a URL DB, selects a URL to be searched for the first time, and executes a robot agent (S410). The collecting robot of the robot agent accesses the selected URL (S420) and determines whether the corresponding URL allows the access of the collecting robot (S430). That is, the collecting robot accesses http: //hostname/robots.txt according to the URL host name, analyzes the contents of robots.txt, and determines whether the URL can be accessed. As a result of the determination, if access is allowed, documents of the site are collected based on the analyzed robots.txt and URL information (eg, URL address, etc.) is stored in the URL DB (S440). At this time, all URLs are converted to absolute URLs and not stored relative URLs. That is, the relative URL (eg, /dir/index.html) is converted into an absolute URL (http: //hostname/dir/index.html) and stored. The extracted URL is saved in the URL DB for later use and the related URL is also saved. For example, if http: //host/dir/subdir/file.html is the URL to save, the URL other than this URL (for example, http: // host / dir / subdir /, http: // host / dir /, http Also save: // host /). In addition, when storing the extracted URL in the database, it is checked whether it is already registered so that duplicate URLs are not registered.

Next, the information providing system moves to the next URL based on the URL information stored in the URL DB (S450), and determines whether the moved URL is the last URL stored in the URL DB (S460). And repeat the subsequent steps, and if the moved URL is the last URL, all processing ends.

5 is a flowchart illustrating a document score calculation process for each index word according to the present invention.

First, the index agent requests the robot agent through the FA to receive document information (eg, document content, URL address where the corresponding document is located) to be indexed (S510). That is, the robot agent stores the document information collected using the tracking robot in the URL DB, and when requested by the index agent, extracts the document information from the URL DB and delivers the document information to the index agent through the FA. The index agent extracts the index word of the corresponding document based on the document information received from the robot agent and calculates the frequency of appearance in the document of each index word (S520). Index terms do not include adverbs and verbs, and it is preferable to extract noun forms of adjectives, adjectives, or verbs.

After calculating the index word and the appearance frequency of one document, the index agent requests the next document from the robot agent (S530), and determines whether the requested document is the last document stored in the URL DB of the robot agent ( If it is not the last document, the process proceeds to the step S520, and the subsequent steps are repeated. If the requested document is the last document, the frequency of occurrence of a specific index word appears in the document, and the score is calculated using a formula. Simultaneously extracting all the documents including the index word and calculating the score of the corresponding document for a specific index word (S550). An example of the frequency of occurrence of a specific index word and the scoring of the document is as shown in FIG. 2. Next, the index module is based on the document score for each index word calculated in the step S450 (for example, the document containing the index word, the document containing the index word, the score of the document containing the index word and the document is located) URL address, etc.) are stored in the index DB (S560). An example of document information for each index word is shown in FIG. 3.

6 is a flowchart illustrating an index word processing process input by a client according to the present invention.

First, the client accesses the search server of the information providing system of the present invention (S610), and inputs an index word to be searched using the AND operator (S620). It is preferable to input nouns and adjectives or nouns of verbs without index terms including adverbs and verbs. The search agent (in particular, the search server) delivers the index word input by the client to the index agent, and receives the index information document information from the index agent (S630). That is, the index agent extracts all documents including the received index words and the scores of the corresponding documents for each index word, and delivers the corresponding information to the search agent (S630).

The search server of the search agent (particularly, the thread manager) calculates a document score including all index words by multiplying the scores of the corresponding documents for each index word based on the document information for each index word received from the index agent (S640). The search list is created in order of high score, and the search list is provided to the client (S650). An exemplary diagram illustrating an example of calculating a document score including all index words according to the present invention is shown in FIG. 7. For example, suppose a client enters 'car AND avante' as an index, the search agent sends 'car' and 'Avante' to the index agent, and the index agent searches the index database to find documents containing each index. Extracts and selects documents containing both 'car' and 'Avante'. As shown in FIG. 3, the document including 'car' includes Document 1, Document 2, Document 4, and Document 5, and the document including 'Avante' includes Document 1, Document 2, Document 3, Document 4, and Document 5. There is. However, since the client inputs the index word using the AND operator, the client should select a document including both the car and the avante. In other words, Document 3 has 'Avante' but not 'Car', so it is excluded from the output list. The index agent transmits the extracted document information for each index word to the search agent, and the search agent multiplies the scores of the documents including the respective index terms by creating a search list in high score order and provides the search list to the client. As shown in FIG. 7, since the score of the document including each index word increases in the order of document 5, document 2, document 1, and document 4, the search list is provided in order of document 4, document 1, document 2, and document 5. .

Experimental Example

Retrieval using a conventional AND operator and an AND operator according to the present invention using a document set (TREC: English) and a document set commonly used in Korea (HANTEC) The following is an example of measuring and evaluating the accuracy by comparing the search results according to the method with the standard results.

Old way Manner of the invention all top 20 all top 20 TREC 0.1215 0.2451 0.1429 (17.61%) 0.3085 (25.87%) HANTEC 0.1012 0.1315 0.1183 (16.90%) 0.1565 (19.01%)

TREC is an arbitrary document set consisting of 1 million English documents and HANTEC is 300,000 Korean documents, and there are 50 to 100 standard queries for each set. The list of standard documents associated with each standard query statement is predefined. After indexing this document set, each standard query statement compares the list of results with the standard document list to determine how accurately the search engine finds relevant documents. The table above is part of the experimental results for Trek and Hantec, the 'all' item is for the entire result list, and the 'top 20' item compares the accuracy with the standard document list only for the top 20 documents in the result list. One result. In addition, the number in parentheses in the 'method of the present invention' indicates the degree of improvement of accuracy over the existing method. As shown in the results, it can be seen that the method of the present invention has an improved accuracy compared to the existing method, and in particular, it is possible to provide a more accurate search result to the client by bringing the improved accuracy in the top 20 result list.

The above description is only for explaining one embodiment, and the present invention is not limited to the above-described embodiment and can be variously changed within the scope of the appended claims. For example, the shape and structure of each component specifically shown in the embodiments of the present invention may be modified.

As described above, according to the index word search and information providing system and method using the AND operator according to the present invention, in calculating the order of the search results using the AND operator, the frequency of occurrence in a specific document among a plurality of index words is calculated. In addition, the search result is calculated by multiplying the occurrence frequency of each index word by the score of the corresponding document, thereby reducing the search time and providing a high-precision search result.

Claims (7)

In the index word search and information providing method for analyzing the index words requested by the client to list the desired information to provide to the client, (a) collecting a document by accessing a URL on the Internet based on the registered URL address information and storing URL information including a URL address where the document is located and contents of the document; (b) receiving the URL information to calculate an index word and a frequency of appearance of the document, calculating a score of the document with respect to the index word using a scoring formula, and storing document information for each index word; And (c) receiving the document information for each index based on the index word inputted by the client using the AND operator, multiplying the scores of the corresponding document for each index word, and generating a search list in the order of high scores to the client. Index search and information providing method using the AND operator characterized in that it comprises a step of. The method of claim 1, wherein step (b) (b1) requesting document information to be indexed; (b2) receiving the document information, extracting an index word of a document to be indexed, and calculating a frequency of appearance of each index word in the document; (b3) calculating a score of a corresponding document for a specific index word using the score calculation formula; And and (b4) storing document information for each index word based on the calculated document score for each index word. The method of claim 1 or 2, wherein the score calculation formula is ego, tw is the relative importance of a particular index for any document, TF is the frequency of occurrence of a specific index, IDF is the importance of a particular index in the entire document, tf is the frequency of a particular index in any document, max tf is the maximum frequency of a specific index word in the entire document, N is the total number of documents to be indexed, n is the number of documents in which a specific index word appears in the entire document. The index word according to claim 1 or 2, wherein the document information for each index word includes a document including the index word, a score of the document including the index word, and a URL address where the document is located. How to Search and Provide Information. The method of claim 1, wherein step (c) (c1) accessing a search server and inputting an index word using the AND operator; (c2) receiving document information for each index word including all documents including each index word and scores of corresponding documents for each index word, and multiplying the scores of the corresponding documents for each index word by each other; And and (c3) listing the documents including the index terms in high score order and providing the documents to the client. In the index search and information providing system for analyzing the index words requested by the client to list the desired information to provide to the client, A robot agent that traverses a web server on the Internet and automatically collects, indexes and databases various kinds of information posted on each homepage; It is interconnected with the robot agent, extracts the index word by indexing the documents collected by the robot agent, calculates the frequency of occurrence of each index word in the document, and scores of the corresponding document for each index word using a score calculation formula. An indexing agent for estimating a value and storing the calculated document information in an index DB; And It is interconnected with the indexing agent, analyzes the index word input by the client, extracts a document including each index word by referring to the index DB, and scores corresponding documents for each index word stored in the index DB. And a search agent for generating a search list in order of multiplying documents and providing them to the client, the index word searching and information providing system using an AND operator. The method of claim 6, wherein the score calculation formula is ego, tw is the relative importance of a particular index for any document, TF is the frequency of occurrence of a specific index, IDF is the importance of a particular index in the entire document, tf is the frequency of a particular index in any document, max tf is the maximum frequency of a specific index word in the entire document, N is the total number of documents to be indexed, n is the number of documents in which a particular index word appears in the entire document, index index search and information providing system using the AND operator.