KR20150121268A - Crawling System And Method For Web Documents Based On Auto-Generated Rules Using Patterns Of User Activities - Google Patents

Abstract

Disclosed are a system and a method for collecting a web document based on automatic rule generation according to a user behavior pattern. The system for collecting a web document according to an embodiment of the present invention includes: a scenario processing unit generating a scenario including rules for both heterogeneous document consisting of text and object, and user behavior pattern related to items of interest on web document and web document; and a web document collecting unit for collecting heterogeneous web documents based on the scenario generated by the scenario processing unit.

Description

Technical Field [0001] The present invention relates to a Web document collection system and method based on automatic rule generation based on a user behavior pattern,

The present invention relates to a system and method for collecting web documents, and more particularly, to a system and method for collecting web documents in order to automatically collect web documents in order to analyze information existing in the web, And a web document collection system and method based on automatic rule generation according to a behavior pattern.

The present invention is derived from research conducted and conducted by Saltlux Co., Ltd. as part of the SW Computing Industry Source Technology Development Industry (SW) of the future Creation Science Department. [Research period: 2013.05.01 ~ 2014.04.30, Research institute: Korea Industrial Technology Evaluation & Management Service, Research title: WiseKB: Big data understanding based self-learning knowledge base and inference technology development, task number: 10044494]

In order to analyze various documents existing on the web, documents on the web can be collected and stored in advance. One of the collection methods of these web documents is that a person can visit each web document and copy and paste the contents of the document. However, due to time and cost problems, systems for automatically collecting Web documents through programs and the like have been developed. Furthermore, as the web environment changes, there is a growing demand for the collection of interactive web documents that can interfere with user behavior in addition to web documents based on Hyper Text Markup Language (HTML).

The present invention provides a system and method for collecting web documents, which can efficiently and automatically collect interactive web documents, in which user's actions can be intervened, along with web documents HTML based web documents.

Each of the web document collection systems according to an embodiment of the present invention is common to heterogeneous web documents implemented with text and objects, A scenario processing unit for generating scenarios including rules, and a web document collecting unit for collecting the heterogeneous web documents based on the scenarios generated by the scenario processing unit.

According to the web document collection system and method according to the embodiment of the present invention, a behavior pattern of a user's web document is reflected as rules, and a web document is collected based on a generated scenario, It is possible to collect a web document without being influenced by a layout change of the web document or the like, as compared with the case of collecting the web document by reflecting the method of repetition as it is.

According to the web document collection system and method according to the embodiment of the present invention, by collecting web documents according to scenarios in which a behavior pattern of a user is reflected, not only information-oriented web documents implemented in text but also JavaScript It is possible to efficiently collect complex interactive web documents such as a community site and a social network service that can be implemented as a web page.

Furthermore, according to the web document collection system and method according to the embodiment of the present invention, it is possible to collect more focused web documents at the request of the user.

1 is a diagram illustrating a web document collection system according to an embodiment of the present invention.
FIGS. 2 and 3 are views for explaining an example of the operation of the user interface unit of FIG. 1, respectively.
4 is a diagram showing an example of the scenario verifying unit of Fig.
5 is a diagram illustrating a web document collection system according to another embodiment of the present invention.
6 is a diagram illustrating a knowledge base building system according to an embodiment of the present invention.
7 is a diagram illustrating an artificial intelligence system according to an embodiment of the present invention.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The embodiments of the disclosure are provided to more fully describe the present disclosure to those skilled in the art. The present disclosure is capable of various modifications and may take various forms, and specific embodiments are illustrated and described in the drawings. It is to be understood, however, that the intention is not to limit the present disclosure to the particular forms disclosed, but to include all modifications, equivalents, and alternatives falling within the spirit and scope of the appended claims. Like reference numerals are used for similar elements in describing each drawing. In the accompanying drawings, the dimensions of the structures are shown enlarged or reduced from the actual size for the sake of clarity of the present disclosure.

1 is a diagram illustrating a web document collection system according to an embodiment of the present invention. Referring to FIG. 1, a web document collection system 100 according to an embodiment of the present invention includes a scenario processing unit 120 and a web document collection unit 140.

The scenario processing unit 120 provides an interface to a user to generate a scenario for collecting a web document that the user wants to collect. It is noted that in the present disclosure the web document represents all or part of the web document. The scenario processing unit 120 may include a user interface unit 121, a user pattern analysis unit 122, a scenario generation unit 123, a scenario verification unit 124, and a scenario storage unit 125.

The user interface unit 121 inputs information on the items of interest on the web document and the behavior patterns on the web document from the user. The user interface unit 121 provides an interface to a user so that an area (information or data) to be collected on a web document can be designated by a user in order to create a scenario based on rules for collecting the web document. For example, as shown in FIG. 2 for illustrating an example of the operation of the user interface unit of FIG. 1, an interface may be provided to a user through a browser or the like. When a URL (Uniform Resource Locator) for a web document is inputted through the user interface unit 121, the web document of the URL can be downloaded from the web server and displayed on the user interface unit 121. In the example of Fig. 1, the URL is inputted through the URL input field (1), and the web document can be displayed in the web document display area (2).

In addition, the user interface unit 121 may receive at least one item selected on the web document. For example, a commodity may be selected as an item in a shopping mall page (web document) displayed in a browser. Also, at least one field may be set for each item. For example, a field of a product name, a price, and a user review may be set in a product item. The user interface unit 121 can provide the user with an interface for selection (creation) of items and fields. In the example where the product is an item, the user can click on the item creation button (3) of FIG. 2 provided by the user interface unit 121 to create a new item through the pop-up window.

The user interface unit 121 provides an interface so that a user can define fields to be included in items and items. For example, as shown in FIG. 3 for explaining an example of the operation of the user interface unit of FIG. 1, a field of an item can be set by capturing a product name, a price, and a user review on a Web document. By the capture, the source code corresponding to the area can be inputted. For example, source code such as HTML (Hyper Text Markup Language), XTHML (Extensible HTML), JavaScript (JavaScript), or Ajax (Asynchronous JavaScript and XML)

The user interface unit 121 may also present a preset item to the user by default. For example, after a product is set as an item, the user can be provided with an interface for adding a new product.

Each field may include at least one or more sub-fields. For example, a field for a user review may include a creator name, a created date, and a subfield of the created content. The setting for the subfield may also be the same as above.

The user interface unit 121 can receive information on the behavior pattern of the user. For example, when the user review is composed of a plurality of pages, an interface is provided to receive information on whether to enlarge and apply the items set for the first page (and the fields and subfields included in the items) to other pages can do.

Although the interface in the shopping mall page has been described above, the present invention is not limited thereto. For example, the user interface unit 121 sets the knowledge of the wikipedia as an item, and when the web document for the knowledge 1 is collected, the related information included in the "together view" By providing an interface that can be set to link with knowledge, it is possible to collect a web document to be collected by the user and a user behavior pattern thereon. Alternatively, the user interface unit 121 may collect the web document on the bulletin board of the category 1 of the community site, for example, and then collect the collection rule of the web document in another category (for example, a category 2) can collect the web documents on the web page and the user behavior patterns of the web documents to be collected by the user.

In addition, the user interface unit 121 may provide an interface for setting a cycle for web document collection. For example, an interface can be provided to periodically update the collected web documents. The above-described interface can be provided through a button on a browser or a pop-up window or the like.

Information or data about the user's web document collection rules received through the user interface unit 121 may be processed by each component of the web document collection system 100 described later.

1, the user pattern analyzer 122 analyzes the information or data received by the user interface unit 121 and extracts characteristics (rules) of the user's web document collection behavior. For example, the user pattern analyzer 122 may extract the type and order of the user's web document collection behavior. For the example of the shopping mall page described above, the user pattern analyzing unit 122 extracts from the tag or object of the source code of the item (s) captured by the user and the field (or subfield) You can extract rules such as types. In addition, for the example of the above-described community site, the user pattern analyzing unit 122 may extract the visiting order of the bulletin board set by the user.

The scenario generating unit 123 generates a scenario in which a time variable is considered, that is, a time-dependent change can be reflected, based on characteristics of a user's web document collection behavior from the user pattern analyzing unit 122. [ For the example of the shopping mall page described above, the scenario generating unit 123 generates a scenario so that items of a field or a sub-field can be collected for a new product according to the source code analyzed by the user pattern analyzing unit 122 can do. For example, a scenario may be created so that the rules set for product 1, product name, price, and user review can be collected for different products. In this case, the scenario may be generated to include a tag or object for the brand name, price, and user review on the web document.

The scenario generating unit 123 may apply the rule set (extracted) to the first web document to at least one second web document. With respect to the above-described example of the community site, the scenario generating unit 123 generates a scenario in which the user applies the matters set for the web document on the bulletin board of the category 1 to the bulletin board 2 of the category 1 set to be visited by the user can do. For example, the scenario generation unit 123 may generate a scenario in which the web document on the bulletin board of category 2 is collected with a keyword set for the web document on the bulletin board of category 1.

The scenarios generated by the scenario generating unit 123 are thus common to heterogeneous web documents implemented with text and objects. In the present invention, an object is not limited to an object such as JavaScript, but includes a concept of all languages other than text such as a function. For example, the scenario may be based on a web document implemented as an object, executed by user action (e.g., click or play) Scenarios that reflect actions (e.g., from page 1 to page 2) can be created.

As described above, the scenario generating unit 123 can generate scenarios in which time variables are considered. For example, the scenario generating unit 123 may generate a scenario by setting a cycle for web document collection. In this case, for example, in the first cycle, a web document related to "A and B establishing a company C" is collected, and in the second cycle, a web document related to "C company exporting 1 billion dollars" is collected .

The scenario generating unit 123 may also generate scenarios with different weights for the rules. For example, if there are a plurality of prices for merchandise 1, a scenario may be created such that a web document for the lowest price is collected.

The scenario verification unit 124 can perform verification of the generated scenario. With respect to the above-described example of a shopping mall page, the scenario verifying unit 124 can collect information on other goods in the generated scenario and test whether accurate collection is performed according to the scenario. The user interface unit 121 may provide an interface for verifying the scenario, for example, a test button on the browser.

4 is a diagram showing an example of the scenario verifying unit of Fig. Referring to FIG. 1, the scenario verification unit 124 may include an automatic verification unit 124_2 and a manual verification unit 124_4. The automatic verification unit 124_2 can automatically search for and correct an error for a temporary scenario generated by the scenario generation unit 123 by a predetermined algorithm or the like. Alternatively, the automatic verification unit 124_2 can automatically search for and correct errors in the temporary scenario based on the Internet, cloud sourcing, and external knowledge data (e.g., Wikipedia) by a predetermined algorithm or the like. The automatic verification unit 124_2 can search for and correct an error in a temporary scenario through, for example, a feedback on the scenario using a social network or the like.

The manual verification unit 124_4 can instruct the temporary scenario to manually search for or correct the error. For example, the manual verification unit 124_4 may be implemented with an interface (for example, a lighting or a pop-up window, etc.) for notifying a user (or a knowledge curator) when a manual search or correction of an error is required. The manual verification unit 124_4 can search for or correct an error for a temporary scenario based on data or information input from the user (or knowledge curator) through the interface. The manual verifying unit 124_4 can perform the above-described operation when there is a request from the automatic verifying unit 124_2, for example. When the automatic verification unit 124_2 receives an error for the temporary scenario from the social network as in the above example, it can send a request for error correction to the manual verification unit 124_4. Alternatively, the manual verification unit 124_4 may periodically perform the same operation even if there is no request from the automatic verification unit 124_2.

By the combination of the automatic verification operation and the manual verification operation as described above of the scenario verification unit 124, the web document collection system 100 according to the embodiment of the present invention can generate a more efficient, accurate and useful scenario have. The scenario verified by the scenario verification unit 124 is stored in the scenario storage unit 125. [

1, the web document collecting unit 140 collects web documents (or data on a web document) on the web (Internet) based on the verified scenarios, A web document analysis unit 142, and a web document storage unit 143.

The web document download unit 141 can collect and download a web document based on a scenario received from the scenario storage unit 125. [ For example, for the example described above, web documents for "A "," B ", or "Company C" can be collected and downloaded according to the cycle defined by the scenario.

The web document analysis unit 142 analyzes the downloaded web document according to a scenario received from the scenario processing unit 120, and extracts a web document corresponding to the scenario. The metadata and content of the downloaded web document can be extracted according to the scenario received from the scenario storage unit 125. [ The web document analysis unit 142 may include a first type web document analysis unit 142_2 and a second type web document analysis unit 142_4. The first type web document analyzing unit 142_2 may analyze a web document in a text form implemented by, for example, HTML or XHTML to extract content (content) between the tags. The second type web document analyzing unit 142_4 may extract a content of a user action defined by the object by analyzing a web document in the form of an object implemented by, for example, JavaScript or Ajax.

5 is a diagram illustrating a web document collection system according to another embodiment of the present invention. Referring to FIG. 5, the web document collection system 100 according to another embodiment of the present invention includes a scenario processing unit 120 and a web document collection unit 140, as in FIG. However, the scenario processing unit 120 of the web document collection system 100 of FIG. 5 may further include a rule conversion unit 126. FIG. The user interface unit 121 may provide the user with an interface for editing web document collection rules, such as items and fields. The rule conversion unit 126 may receive information on the conversion of the web document collection rule of the user received through the user interface unit 121 and may control the scenario generation unit 123 to generate a scenario reflecting the same. For example, the rule conversion unit 126 may add rules for product weight, etc. in addition to the product name, price, and user review for the product.

The web document collection unit 140 of the web document collection system 100 of FIG. 5 may further include a scenario control unit 144 and a web document collection control unit 145. The scenario control unit 144 may control the processing sequence and the process timing of the scenario received from the scenario processing unit 120. [ For example, the scenario control unit 144 can control that the web document collection according to the scenario 1 of the product and the scenario 2 of the scenario 2 of the community site is performed earlier than that of the scenario 1. In addition, the scenario control unit 144 can control the web document collection for scenario 1 to be performed once, for example, once a week.

The scenario control unit 144 can further assign a weight to the web document collection or the web document analysis according to the scenario. For example, in collecting a web document according to a scenario for a product, a weight can be given to a web document collected at a major Internet shopping mall or an Internet shopping mall specialized in the product.

The web document download unit 141 and the web document analysis unit 142 may operate according to the control of the scenario control unit 144 and the web document collection control unit 145 as described above.

As described above, according to the web document collection system and method according to the embodiment of the present invention, by collecting web documents based on the generated scenarios, the behavior patterns of the user's web documents are reflected as rules, It is possible to collect a web document without being influenced by a layout change of the web document or the like, as compared with the case of collecting the web document reflecting the method of recording and repeating the same.

According to the web document collection system and method according to the embodiment of the present invention, by collecting web documents according to scenarios in which a behavior pattern of a user is reflected, not only information-oriented web documents implemented in text but also JavaScript It is possible to efficiently collect complex interactive web documents such as a community site and a social network service that can be implemented as a web page.

Furthermore, according to the web document collection system and method according to the embodiment of the present invention, it is possible to collect more focused web documents at the request of the user.

The web document collection system and method according to an embodiment of the present invention may be controlled by a processor or the like of a computing system that includes or includes. Also, for example, the web document collection system and method according to the embodiment of the present invention can be implemented as a workbench, and can be conveniently used by the user.

6 is a diagram illustrating a knowledge base building system according to an embodiment of the present invention. 6, a knowledge base construction system 600 according to an embodiment of the present invention includes a web document collection system 100, a knowledge acquisition unit 610, a self knowledge acquisition unit 620, a reasoning unit 630, And a knowledge data storage unit 640. The web document collection system 100 may be the web document collection system of Fig. 1 or Fig. 5 described above.

The knowledge acquisition unit 610 may receive a web document from the web document collection system 100 or may crick external resources by receiving big data from the Internet or the like. The knowledge acquiring unit 610 can acquire knowledge by classifying and refining data through natural language processing and meaning giving from the external resources as described above, sorting and semantic integration of knowledge resources, and the like.

The self-knowledge learning unit 620 can self-learn the knowledge acquired by the knowledge acquisition unit 610 and process it into more refined knowledge. For example, the self-knowledge learning unit 620 can analyze and integrate fragmented knowledge through learning. For example, the self-knowledge learning unit 620 can connect the fragmented knowledge or correct the contradictory knowledge through external knowledge data learned through a wiki or the like. The knowledge learned by the self-knowledge learning unit 620 may be stored in the knowledge data storage unit 640 in the form of knowledge data. The knowledge data may be implemented in a triple or schema form. In addition, the self-knowledge learning unit 620 may utilize the knowledge data stored in the knowledge data storage unit 640 as a seed knowledge.

The reasoning unit 630 can deduce the query received from the outside based on the knowledge data generated from the self-knowledge learning unit 620 or the knowledge data stored in the knowledge data storage unit 640. The results deduced by the reasoning unit 630 may be stored in the knowledge data storage unit 640 as knowledge data or may be used to update or correct the knowledge data stored in the knowledge data storage unit 640. [

The knowledge base construction system 600 according to the embodiment of the present invention includes the web document collection system 100 according to the embodiment of the present invention and uses a variety of accurately focused web documents, Base can be constructed.

7 is a diagram illustrating an artificial intelligence system according to an embodiment of the present invention. Referring to FIG. 7, an artificial intelligence system 700 according to an embodiment of the present invention includes a web document collection system 100, a knowledge base construction system 600, and an intelligent question answering system 710. The web document collection system 100 may be the web document collection system of Fig. 1 or Fig. 5 described above. The knowledge base construction system 600 constructs a knowledge base by receiving a web document from the web document collection system 100 or receiving big data from the Internet or the like. The intelligent question answering system 710 generates a response using a knowledge base building system 600 for a query from a user.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

100: Web document collection system
120: scenario generating unit
140: Web document collection unit

Claims (9)

A scenario processor for generating a scenario that is common to different types of web documents implemented as text and objects, and includes a rule for a topic of interest on a user's web document and a behavior pattern for a web document; And
And a web document collection unit for collecting the heterogeneous web documents based on scenarios generated by the scenario processing unit. The scenario management system according to claim 1,
A user interface unit for inputting information on an item of interest on the web document and a behavior pattern on the web document from a user;
A user pattern analyzer for extracting rules of a user's web document collection operation based on information input to the user interface unit;
A scenario generating unit for generating the scenario based on the rule extracted by the user pattern analyzing unit;
A scenario verifying unit for verifying a scenario generated by the scenario generating unit; And
And a scenario storage unit for storing the verified scenarios. 3. The method of claim 2,
Wherein the rule is generated so as to apply a rule set for the first web document to at least one second web document by reflecting a rule of the web document collection behavior of the user extracted from the web document implemented by the object. system. 3. The method of claim 2,
Wherein the web document collection system is generated such that a web document is collected by another rule according to a time, reflecting a rule of a web document collection operation of a user extracted from the web document implemented by the object. The scenario management system according to claim 2,
Further comprising a rule conversion unit that receives information on a conversion of a user's web document collection rule received through the user interface unit and controls the scenario generation unit to generate a scenario reflecting a changed rule, . The web document collecting system according to claim 1,
A web document downloading unit for collecting and downloading a web document based on a scenario received from the scenario processing unit;
A web document analysis unit for analyzing the downloaded web document according to a scenario received from the scenario processing unit and extracting a web document corresponding to the scenario; And
And a web document storage unit for storing the web document extracted by the web document analysis unit. [7] The method of claim 6,
A first type web document analyzing unit for analyzing a web document corresponding to the scenario based on a tag of the downloaded web document and a content between the tags; And
And a second type web document analyzing unit for analyzing whether the web document corresponding to the scenario is based on the object of the downloaded web document. [7] The method of claim 6,
Further comprising a scenario control unit for setting a processing sequence and a processing time of a scenario received from the scenario processing unit and controlling an operation of the web document download unit to collect and download the web document. The method according to claim 6,
Further comprising a scenario control unit for assigning weights to web document collection or web document analysis based on scenarios received from the scenario processing unit.

Images