KR100426341B1 - System for searching an appointed web site - Google Patents

Abstract

The present invention relates to a designated web site search reporting system and method for searching a document posted on a designated web site and classifying and searching the searched results by category. The specified web site search reporting system includes a search agency server. And a text mining server;

The search agency server connects to the designated service provider server by using the URL information registered in advance in each search cycle, receives the documents up to the designated step from the connected server, and transmits the documents to the text mining server.

The text mining server classifies each category of documents transmitted from the search agency server according to a document classification function generated based on the learning document assigned to each category, and searches the classified client documents through the search agency server. It is characterized by the reporting.

Description

Web site document search reporting system {SYSTEM FOR SEARCHING AN APPOINTED WEB SITE}

The present invention relates to an automatic document retrieval system, and more particularly, to a designated web site document retrieval reporting system for retrieving a document posted on a designated web site and classifying and searching the retrieved results by category.

With the spread of personal computers (PCs) and the rapid increase of Internet users, the Internet has emerged as an information providing medium that is growing faster than any conventional media. For this reason, companies are paying close attention to advertising their products and managing their images through the Internet. Especially, monitoring personnel actively collect opinions of netizens posted on various sites and reflect them in their management.

However, on the Internet, called the "sea of information," friendly documents about the company (e.g. maximizing the operating profits of the company's export improvement posted on the economy of the newspaper site) and unfriendly documents (e.g. It is almost impossible to monitor all of our product boycotts and rumor distribution articles or content for stock manipulation. Because the current document automatic retrieval system generally has to list all the data related to the search word and find the desired data in the listed documents, there is a problem of lengthening the search time. In addition, since the current document automatic retrieval system is not equipped with a report function for the results of the monitored document, as a result, the monitoring agent is inconvenient to newly edit the content of the retrieved document for reporting. In addition, even when searching documents using a monitoring agent, each site has to visit each site and search the posted contents, which causes inefficiency in terms of time and personnel management.

Accordingly, an object of the present invention is to multi-thread a system for retrieving documents published on designated web sites, as well as automatically importing documents of designated web sites and automatically analyzing the analyzed results to a document search client. It is designed to provide a designated web site document search reporting system that can reduce the system load by designing.

It is still another object of the present invention to provide a designated web site document search reporting system that can automatically and consistently recommend a learning document suitable for each category to increase the accuracy of document classification.

1 is a block diagram of a designated web site document search reporting system according to a preferred embodiment of the present invention.

2 is a program structure diagram of a designated web site document search reporting system and an administrator terminal according to a preferred embodiment of the present invention.

3 is a flowchart of a designated web site document retrieval reporting process according to a preferred embodiment of the present invention.

4 is an exemplary diagram of a URL registration screen according to a preferred embodiment of the present invention.

5 is an exemplary view illustrating a search period setting screen according to an exemplary embodiment of the present invention.

6 is an exemplary view of a report management screen according to a preferred embodiment of the present invention.

7 is a view showing a document view screen of report management according to a preferred embodiment of the present invention.

8 is a diagram illustrating an automatic assignment document learning screen according to a preferred embodiment of the present invention.

9 is an exemplary diagram of an administrator registration screen according to a preferred embodiment of the present invention.

A designated web site document search reporting system according to an aspect of the present invention for achieving the above object includes at least a search agency server and a text mining server;

The search agency server connects to the designated service provider server by using the URL information registered in advance in each search cycle, receives the documents up to the designated step from the connected server, and transmits the documents to the text mining server.

The text mining server classifies each category of documents transmitted from the search agency server according to a document classification function generated based on the learning document assigned to each category, and searches the classified client documents through the search agency server. It is characterized by the reporting.

In addition, the designated web site document retrieval reporting method according to an aspect of the present invention is executed by at least one server connectable to a plurality of service provider servers;

Connecting to a designated service provider server at each search period;

Reading web page documents up to a designated step from a connected server;

Analyzing the read documents and classifying them into one of categories previously classified by learning;

And reporting the categorized documents to the service provider server which requested the search.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted.

First, Figure 1 shows a block diagram of a designated Web site document search reporting system according to a preferred embodiment of the present invention. Designated web site document search reporting system 200 according to an embodiment of the present invention is largely composed of a search agent server (Censor Agent Server) 210 and a text mining server (TMS) 220, the Internet network ( 130 is connected to a plurality of Internet service provider servers (100, 110, 120).

The search agency server 210 is connected to the designated Internet service provider server by using the URL information registered in advance in the TMS 220 at each search cycle, and is connected to the documents (HTML documents) from the connected server up to the designated step. Read the URL of the server and delivers it to the TMS (220). This search agency server 210 is connected to the Internet service provider server as shown in Figure 1, the web robot 212 to retrieve the document up to the specified step and the URL of the server, and parses the syntax of the imported document And a parser 214 for removing and outputting a tag. Since the web robot 212 can be designed and operated in a multi-threaded form, it can reduce the system load and shorten the document retrieval time.

Meanwhile, the text mining server TMS 220 generates a category for document censoring based on the learning document, analyzes the documents transmitted from the search agency server 210, classifies the document into one of the classified categories, and classifies the category. It performs a role of reporting the respective documents to the service provider server requested to search through the search agency server (210). The TMS 220 is connected to databases 222 and 224 which store the reporting result and the URL of the designated web site.

For reference, the text mining process performed in the TMS 220 in order to help the understanding of the present invention in more detail. First, the text mining server 220 according to an embodiment of the present invention performs the following process. Automatically classify documents.

1. Define categories that meet the characteristics of the document set you want to classify.

2. Allocate learning documents to the characteristics of the defined categories.

3. Create document classification function for automatic classification based on assigned learning document.

4. Classify documents automatically according to the created document classification function.

5. If necessary, define additional new categories.

6. Update or add the learning document assigned to each category to increase the accuracy of document classification.

That is, the TMS 220 according to an embodiment of the present invention first analyzes the learning document assigned to a specific category. In this case, the appearance of a specific feature word for the learning document for each category is examined or the frequency of occurrence is counted. The feature word is composed of appropriate nouns or verbs that can be used for document classification, can be determined by an administrator or by statistical analysis, and can be limited to an appropriate number.

Accordingly, a coordinate value is assigned to each learning document according to the frequency of appearance or presence of feature words. Such a coordinate system may be viewed as a coordinate value in a vector space consisting of feature words. For example, a feature consists only of three things: "computer", "keyboard", and "program." In some learning documents, the "computer" feature is three times, the "keyboard" feature is one time, and the "program" feature is If it appears twice, the coordinate value in the vector space of the feature words of this learning document is (3.1.2). This coordinate value is an n-dimensional (assuming n feature words) sphere of space that is normalized to length 1 so that each document can be displayed on a sphere surface of n-dimensional space normalized to radius 1. It is normalized to a vector of length 1 so that it can be displayed on the (sphere) surface.

Next, the probability of occurrence of the feature word extracted from the input document for all the learning documents in a specific category is calculated by Equation 1 below.

In Equation 1, the left side is a probability that the feature t _i appears in all documents in a specific category. On the right side, tf (t, c) is the frequency of occurrence of the feature t in category c. V is 1 when normalized to the size of the feature space. That is, the right side represents the ratio of the frequency of occurrence of the feature word t _i to the frequency of occurrence of all the feature words in category C _i , and the right side is an approximation formula for calculating the left side. The addition of 1 to a numerator is a technique to avoid zeroing the probability that the whole is zeroed and the resultant multiplication is zero.

Once this value is obtained, it means that we are ready to determine which category the input document belongs to. For all feature words in the input document d, a probability is calculated as in Equation 2 below.

The probability expressed by the formula on the left side refers to the probability of occurrence of document d, that is, the probability of appearance of a document having a coordinate value in the feature vector space, such as document d, among all documents belonging to a specific category. Each multiplication term on the right side is obtained by Equation 1 above. The above calculation is based on the assumption that the probabilities for each multiplication term on the right side are independent of each other.

Next, the probability that a specific document belongs to category c _i is calculated by Equation 3 below.

The above values correspond to the aforementioned document classification function. Where P (c _i ) is the category among the total number of documents The probability of category appearance, calculated as the number of documents belonging to. In addition, P (d) is the probability of appearance of a document having the same coordinate value as the document d in the feature vector space. However, the above document classification function does not need to calculate the denominator P (d) which has the same value in all cases because it is sufficient to make relative comparisons for each category. The resulting document classification function is then used to calculate the relative probability of document d for each category. In other words, If all the values are obtained and the document d is assigned to the category which is the maximum value, the documents are classified by categories.

On the other hand, the accuracy or reliability of automatic document classification is greatly influenced by which document the learning document is selected in the text mining server 220 described above. In the text mining server 220 according to an embodiment of the present invention, the administrator analyzes the current situation and appropriately updates the learning document assigned to each category. As a result, not only the labor burden collected by one person but also several people working at the same time or at the same time may result in a loss of consistency in the selection of learning documents, resulting in instable document classification.

To this end, another embodiment of the present invention is to automatically recommend the appropriate learning document for each category for the classified document set to be able to achieve a more accurate document classification. In more detail,

First of all, in order to improve the performance of document classification, the learning document should be selected so that the vector space composed of feature words forms a good coordinate system. In other words, the learning document should be selected so that the appearance frequency characteristics of the feature words are clearly different for each category.

In more detail,

The frequency distribution curve of feature w in each category ci,

Calculate all And Select the learning document so that the relative distance between the category distribution curves for each category is as far as possible.

In the present invention, the distance between these temperature distribution curves is defined as in Equation 4.

And in Equation 4 Is defined by Equation 5 below.

Equation 4 is a dot distribution curve Wow The distance between the two is defined, and Equation 5 has already been described in Equation 1. In Equation 5, the lambda value is 1.

However, this distance function is nonsymmetric so that the distance from a to b is different from the distance from b to a. In order to eliminate the asymmetric property, the weighted average value is calculated and is defined as Equation 6 as the actual distance function.

This KLdist (Kullback-Leibler Distance) function means entropy when assigned to a specific category as compared to when assigned to another category. In order to improve the performance of the automatic classification function, the learning document should be selected in the direction of increasing the disorder value. That is, in order to make this KLdist as large as possible, the TMS 220 according to another embodiment of the present invention selects a document d that can be an unclassified learning document candidate in a specific category. Each other category, assuming Distance from KLdist ( , ) If the distance value is larger than a predetermined threshold value, the document may be assigned as a learning document of category c _i , so that automatic classification of the document may be performed more accurately than a general text mining system. For reference, the threshold value may be determined by, for example, obtaining an average value of KLdist values between existing learning documents and subtracting the value or a small value from the value.

2 is a diagram illustrating a program structure of a designated web site document search reporting system 200 and an administrator terminal 270 according to an exemplary embodiment of the present invention. The censorship server 210 and the TMS 220 according to the embodiment of the present invention are implemented as one physical server, but may be operated as separate servers to execute the present invention.

Referring to FIG. 2, the server 200 includes a plurality of layered programs, which are various operating systems 260 such as Windows NT / 2000, Linux, or UNIX. The web server 250 that depends on the operating system is located on the base, and the Java virtual machine 240 that is provided depending on the operating system is mounted on the web server 250. In addition, an application program (Java Servlet / JSP / Java Application) 230 depending on the JVM 240 is located thereon. On the other hand, the manager terminal 270 also has an operating system such as Windows 98/2000 / Linux, and a web browser and a robot control management page are sequentially positioned on the operating system to perform data communication according to the communication protocol defined with the server 200. do.

Hereinafter, the operation of the designated web site document search reporting system 200 having the above-described configuration will be described in detail with reference to FIGS. 3 to 9.

3 is a flowchart illustrating a designated web site search reporting process according to a preferred embodiment of the present invention, and FIGS. 4 to 9 are displayed while the designated web site search reporting system 200 is operated according to a preferred embodiment of the present invention. The screens are examples.

Referring to FIG. 3, first, in order to periodically search for a desired web site to a desired depth, a URL, a search period, and a depth of a web site to be searched should be set (step 300). That is, the administrator selects the environment setting on the initial screen as shown in FIG. 4, selects "URL management" in the setting menu, and inputs and selects the URL of the website to be searched. The URL of the input web site is registered in URL DB 222 of 200. In addition, the administrator selects "search management" in the setting menu as shown in FIG. 5, and sets the search period and depth in units of months / weeks / days so that the designated web site search reporting system 200 is configured for each search period. You will be able to search the specified Web site up to the specified level. The " depth " is a command for indicating how many steps to follow a linked site by visiting designated web sites.

As described above, when the URL, the search period, and the depth are set by the administrator, the web robot 212 of the search agency server 210 determines whether the search period is set (step 310), and accesses to the designated web site at each set search period. Attempt (step 320). If there is a user authentication request while attempting to access the designated web site (step 330), an ID and password for authentication processing are transmitted (step 340). The ID and password for authentication processing are also registered together at the time of URL registration as shown in FIG. On the other hand, if a user authentication request is not required, the web robot 212 reads the URL and HTML document from the connected web site (step 350) and transmits the parser 214, which is a parser. The parser 214 then interprets the transmitted HTML document and transmits the untagged document to the TMS 220 (steps 360 and 370).

Meanwhile, the TMS 220 compares the depth of the untagged document, that is, the depth of the current document with the designated depth registered in step 300 (step 380), and then removes the tagged document until the depth of the current document is greater than the specified depth value. Save (step 390). If the depth of the current document is greater than the designated depth value, the stems of the stored documents are analyzed (400), the feature is extracted (step 410), and the stored documents are automatically classified according to the category according to the document classification function generated based on the assigned learning document. (Step 420). As such, an example of a screen of a document automatically classified by category is illustrated in FIG. 6. The screen shown in Fig. 6 is one screen of the "Report Management" item, and the category name is "Samsung Friendship". If you look at the view screen of the documents automatically categorized using "Samsung Friendship" as the category name, it consists of the title, the collected document, the edited document, and the search time. In such a screen, the edited document is a document that has been modified by an administrator and reported to the search client. Unmodified collections may be used for reporting purposes. For the requesting client, it would be advisable to report the collection itself to gather accurate information or opinions.

On the other hand, if the read documents cannot be classified into one of the pre-categorized categories, they are separately stored as unclassified documents for setting a new category, and the document classification is performed as a new category through the category setting and learning step.

FIG. 7 illustrates a document screen when one of the "Samsung Friendship" category names shown in FIG. 6 is selected. Documents collected or edited on the basis of the designated web site are stored in the reporting DB 224 and are read out at intervals set by the request of the document search client, and are then searched for by the client terminal or the search client through the search agent server 210. Reporting to the service provider server (step 430).

Thus, a client who is a document retrieval client can receive reports of friendly or unfriendly documents about the company without directly monitoring a large number of designated websites. In addition, the document retrieval client may use the reported documents as a database to improve future customer service.

8 is a diagram illustrating an example of a learning document setting screen according to a preferred embodiment of the present invention. As shown in Fig. 8, a correlation diagram is described on the right side of a document automatically classified under a specific category name. Correlation is a numerical representation of the correlation between the classified document and the learning document, the higher the value is closer to "1" can be said.

Therefore, the administrator may improve the document classification performance of the designated web site document search reporting system 200 by selecting a document having a high correlation and instructing to update the learning documents assigned to each category.

9 is a diagram illustrating an example of an administrator registration screen according to a preferred embodiment of the present invention, and is a screen when the setting menu is selected as an administrator registration in the "environmental setting" item. This screen allows you to register administrators for editing classified documents or set up administrators to monitor unclassified documents.

As described above, the present invention can take the documents of designated web sites, analyze them automatically, and automatically report the analyzed results to the document search requester, thereby reducing the financial and time efforts to be administered for document retrieval. There is this.

In addition, the present invention can reduce the load on the system and minimize the document retrieval time because the system for retrieving the documents posted on the designated Web sites in a multi-threaded form, and the monitoring results are periodically Since it can be provided as a database, it can also be used as a material that suggests the direction of future customer service.

In addition, the present invention has the advantage of providing accurate information contents to the requestor by categorizing and reporting the contents posted on the designated web site by the subdivided category by learning, and consistently automatically recommends the learning documents suitable for each category. There is an advantage that more accurate document classification can be achieved than document classification by a general text mining system.

Although the present invention has been described with reference to the embodiments shown in the drawings, this is merely exemplary, and it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible. Therefore, the true technical protection scope of the present invention should be defined only by the appended claims.

Claims (9)

delete delete delete delete delete delete delete In a designated web site document retrieval reporting system accessible via a network with multiple service provider servers, The designated web site search reporting system includes a search agency server and a text mining server; The search agency server connects to the designated service provider server by using the URL information registered in advance in each search cycle, receives the documents up to the designated step from the connected server, and transmits the documents to the text mining server. The text mining server classifies each category of documents transmitted from the search agency server according to a document classification function generated based on the learning document assigned to each category, and searches the classified client documents through the search agency server. Report to each of the categories Calculates the distribution of the number distribution of feature words (w), and selects document d, which can be an unclassified learning document candidate, in a specific category. Assume that you have been assigned to another category If the distance value is larger than the preset threshold after calculating the distance from when it is assigned to Designated Web site document search reporting system, characterized in that for updating the learning document by assigning the learning document. The apparatus of claim 8, wherein the threshold is; Designated web site document search reporting system, characterized in that the average value of the distance value according to the equation (7) between the pre-assigned learning documents.