KR20020040145A - Selective Information Gathering System and Methods - Google Patents

Abstract

PURPOSE: A system and method for collecting information selectively is provided to extract only wanted information out of information scattered on a network. CONSTITUTION: A manager user interface(100) designates a site to be collected and designates a place for selecting and storing collection-wanted information in the site. That is, the manager user interface(100) creates a rule with respect to site information, elements to be collected, and database information to be stored and stores the created rule in a database(300). A robot agent(200) connects to a site for collecting information based on rules with respect to site information, elements to be collected, and database information to be stored designated by the manager user interface(100), collects information, and stores the collected information in the database(300). The database(300) stores user interface information created by the manager user interface(100) and information collected by the robot agent(200).

Description

Selective Information Gathering System and Methods

The present invention relates to a system and method for collecting information, and more particularly to an information collection system and method for extracting only the desired information from the information scattered on the Internet.

Recently, due to the development of the Internet, information providers providing information on the Internet are increasing, and thus information provided by the information provider is scattered on the Internet. Recently, as the amount of information supplied from the Internet has rapidly increased, many people search for necessary information using a search robot.

The conventional search robot collects HTML (HyperText Markup Language) files in bulk form on the Internet and uses a method of rearranging them. However, according to this conventional method, since the search robot first collects a large number of related information (in which the desired information and the undesired information are mixed) and re-arranges the information desired by the user, the accuracy of the search is improved. There was a problem that it takes a lot of time to search and fall.

The technical problem to be achieved by the present invention is to solve the problems of the prior art, to provide an information collection system and method for extracting only the desired information from the information scattered on the network.

1 is a diagram showing an information system according to an embodiment of the present invention.

2 is a diagram schematically showing a method for collecting selective information according to an embodiment of the present invention.

3 is a view showing in detail the administrator UI unit according to an embodiment of the present invention.

4 is a diagram illustrating a category registration screen according to an embodiment of the present invention.

5 is a diagram illustrating a category registration process according to an embodiment of the present invention.

6 is a diagram illustrating a category collection screen according to an embodiment of the present invention.

7 is a diagram illustrating a category collection process according to an embodiment of the present invention.

8 illustrates a parsing method according to an embodiment of the present invention.

9a shows an example of general HTML source code,

9B shows an example of a tree node according to an embodiment of the invention.

10 is a diagram illustrating a rule generation screen according to an embodiment of the present invention.

11 is a diagram illustrating a rule generation method according to an embodiment of the present invention.

12 is a view showing in detail the robot agent according to an embodiment of the present invention.

13 is a diagram illustrating an information retrieval method using a robot agent.

Selective information collection method according to one feature of the present invention for achieving the above object

Registering a page including a list of pages for which information is desired to be collected as a category page; A second step of registering an information page including contents to be collected most comprehensively among the category pages as a lower node of the category page; A third step of registering, as a rule, information about contents to be collected by a user among the information pages and location information for moving to the information page; And a fourth step of collecting category pages to be collected at the corresponding site based on the rules registered in the third step.

In addition, the selective information collection method according to an aspect of the present invention

A fifth step of storing the rule registered in the third step in a database; And a sixth step of accessing the corresponding site based on the rules stored in the database, collecting a list of information pages, accessing the information page to extract desired information, and storing the extracted information in the database. have.

The first step is

Displaying contents of a site accessed by a user through a category URL view; When the user selects a page displayed through the category URL view, the method may include registering content in the category URL view as a subcategory of the currently selected category.

The fourth step is

Determining whether there is a selected category; Determining whether there is a registered subcategory of the selected category; Collecting pages of the selected category and creating registered subcategories as sample categories; Extracting a URL from the collected pages of the selected category; When the URL extracted in the step is a URL similar to the sample category, the method may include generating a category inherited from the sample category, setting a category attribute, and registering the category.

The third step is

Creating a tree node by parsing source code of information that a user wants to collect among information pages; And registering location information in the tree node as a rule.

The sixth step

Reading URL information from a URL information database and collecting a designated URL; Reading rule information from a rule information DB to create a parsing tree; finding an element in the parsing tree that matches the rule; And storing the found element in a specific data structure.

Optional information collection system according to an aspect of the present invention

An administrator user interface unit for designating a site to be collected, selecting a collection element to be collected in the site, and generating a rule for database information to store the selected collection element; A robot agent accessing a site to collect information based on rules for site information, collection elements, and database information designated by the administrator user interface unit to collect information; And a database storing rule information generated by the manager user interface unit and information collected by the robot agent.

Here, the administrator user interface unit

A browsing unit for bringing resources on the Internet; A category manager to register and manage categories to be collected; A category collecting unit which automatically collects other categories in the same site based on the category registered by the category managing unit; A parser for analyzing various resources collected by the browsing unit to object various tags; And a rule generator for generating location information of the objects analyzed by the parser.

The database is

A site URL information database that stores information about the site URL; And a rule information database storing location information on elements to be collected in the site.

The robot agent

A document collector configured to read site URL information from the URL information database and read URL resources of each site through a network; A parser for parsing resources read from the document collection unit; And a rule filter that reads rule information from the rule information database, creates a parsing tree, and finds elements matching the rules read from the rule information database in the parsing tree and stores them in a specific data structure.

Hereinafter, with reference to the drawings will be described an embodiment of the present invention;

1 is a diagram showing an information system according to an embodiment of the present invention.

As shown in FIG. 1, an information system according to an exemplary embodiment of the present invention includes an administrator user interface (UI) unit 100, a robot agent 200, and a database 300.

The administrator UI unit 100 designates a site to be collected and selects and stores a desired information to be collected in the site. That is, the administrator UI unit 100 generates rules for site information, elements to collect, and database information to store, and stores the generated rules in the DB 300.

The robot agent 200 collects and stores information in a database 300 by accessing a site to which information is to be collected, based on the site information specified by the administrator UI unit 100, the elements to be collected, and the DB information to be stored. do.

The database 300 stores UI information generated by the manager UI unit 100 and information collected by the robot agent 200.

The following describes the selective information collection method according to an embodiment of the present invention.

2 is a view schematically showing an information collection method according to an embodiment of the present invention.

First, according to an embodiment of the present invention, a user first decides which information is collected from which site, and then registers a site by selecting a site name, a uniform resource locator (URL), a group, etc. to collect the information. Here, the group refers to grouping sites of the same personality, and the DB table stored according to the group may be designated as the same or different tables.

When a site is registered, a page containing a list of pages that you want to collect information is registered as a category while navigating the registered site. (S105) Here, the category page refers to a page having lists of pages (information pages) for collecting information. For example, a page on which product lists of a shopping mall site appear, that is, a page including a refrigerator list, is called a category page.

Then, one information page is selected from the category page and registered as a lower node of the category (S110). Here, the information page is a web page including contents to be finally collected, for example, product information in a shopping mall. If you want to collect, a page with a description of a particular product becomes a page to collect information. At this time, the category page and the information page are registered as one vertical category, that is, in the form of one-level classification category-> two-level classification category-> three-level classification category-> ...-> information page.

When the category page and the information page are registered (S105 and S110), the rules for the information to be collected are registered in the information page. That is, the information desired to be collected on the information page, for example, a product name, a product price, a manufacturer, a product description, and the like are registered as a rule.

Then, it moves to the category page just before the information page, and registers the link information of the information page as a rule. In the category page, lists of information pages exist, and link information (for example, URL) linked to the information page exists in this list. The rules created in steps S115 and S120 are used by the robot agent to go to the page to collect information, as described below.

When rules for information to be collected in steps S115 and S120 and rules for link information (URL) for moving to an information page are registered, all categories to be collected in the corresponding site are collected. At this time, as the categories to be collected are generated by inheriting all properties from the category of the same depth as the category already registered as a sample category, as described below, all newly collected categories maintain the same rules. It becomes possible.

After collecting the category, the rule about the information to be collected and the rule about the link information (URL) for moving to the information page are stored in the database 300. At this time, the rule storage may be stored before the category collection step (S125).

After storing the rules for collecting the desired information in the database, the robot agent is started. (S135) When the robot agent is driven, the robot agent prepares tasks for collecting information by loading rules created to collect desired information from the database 300 into memory. (S140)

The robot agent reads the rules loaded in the memory and accesses a site to collect information, (S145) collects a list of pages to collect information, and when the collection of the list is completed, the robot agent displays information on the information page to collect information. Access and extract the desired information with the registered rules and save it in the specified database. (S150)

Once the information has been collected, the list is collected again and the information page is collected again.

3 is a view showing in detail the administrator UI 100 according to an embodiment of the present invention.

As shown in FIG. 3, the administrator UI unit according to an exemplary embodiment of the present invention may include a browsing unit 260, a category managing unit 210, a category collecting unit 220, an HTML parser 230, a rule generator 240, and a DB. Interface 250.

The browsing unit 260 serves to pull resources (such as HTML files) on the Internet, and the category manager 210 serves to register and manage categories to be collected. The category collector 220 automatically collects other categories in the same site based on the category (classification criteria) registered by the category manager 210. The HTML parser 230 analyzes the resources dragged by the browsing unit 260 to object various tags, and the rule generator 240 serves to create location information of the analyzed objects. The DB interface 250 serves to interface with the database 300.

Next, the function of the category manager 210 according to an exemplary embodiment of the present invention will be described in detail with reference to FIGS. 4 and 5.

4 is a category registration screen managed by a category manager according to an exemplary embodiment of the present invention.

As shown in Fig. 4, the category registration screen according to the embodiment of the present invention includes a category tree list view 211 on the left side and a category URL view 212 on the right side. According to an embodiment of the present invention, a user navigates to a page that he / she wants to register by navigating in the category URL view 212, as in navigating the Internet in a general browser. When the page to be registered appears in the category URL view 212, the name and URL fields at the top are automatically filled in as shown in FIG. At this time, according to an embodiment of the present invention, the text information indicated by the mouse pointer is written as it is, and the URL field is recorded in the URL field.

After confirming that the page to be registered is displayed in the category URL view 212 and clicking the register button on the category registration screen to register a category, the name of the category (page) to be registered in the category tree list view 211. In addition, it is displayed on the desired position, that is, the subnode ('cleaner' in FIG. 4) of the currently selected node ('home appliances' in FIG. 4), and the category attributes can be edited through the category tree list view 211. Provide an interface.

As described above, a process of registering a category through the category URL view and the category tree list view will be described in more detail with reference to FIG. 5.

If the user clicks on the site registration menu in the registration screen shown in Fig. 4 (S200), the category manager 210 outputs a dialog box for site registration on the screen. (S205) The user attempts to access the site by inputting the site name, URL, group name on the dialog box. (S210, S215) Here, a group refers to a user's arbitrarily designating to bind sites of the same personality. (For example, if a user wants to obtain product information about a computer from multiple shopping sites, the user can enter 'computer' as a group and manage it as a separate DB.)

If the desired site is accessed in step S215 (S220), the category manager 210 displays the content of the site in the category URL view 212 and fills the URL field of the category URL view with the current URL field. (S225, S230) On the other hand, when the user navigates the site continuously in the category URL view 212 and accesses another site, the URL field of the category URL view is similarly filled with the current URL field. (S235, S240, S245) )

On the other hand, if the category registration button is clicked in step S230, the category manager 210 registers the contents of the category URL view in the category tree list view 211 as a subcategory of the currently selected category. In other words, the category manager 210 may include a subnode of the desired location, that is, the currently selected node ('home appliances' in FIG. 4) together with the name of the category (page) to be registered in the category tree list view 211 (S4). In Fig. 4, a 'cleaner' is input.

Then, the category manager 210 outputs a category input dialog box and a sub page setting dialog box. (S260, S265)

Next, the function of the category collector 220 according to an embodiment of the present invention will be described in detail with reference to FIGS. 6 and 7.

6 shows a category collection screen managed by a category collecting unit according to an embodiment of the present invention.

As shown in FIG. 6, when the user presses the category collecting button 221 on the category collecting screen, the category collecting unit 220 is based on a category (hereinafter, referred to as a “sample category”) previously registered under the selected category. Search for URLs on the category page selected by using and automatically extract and collect categories with similar rules, and then automatically register the collected categories as subcategories.

Therefore, according to an embodiment of the present invention, a user does not need to register all categories directly, but registers one or more categories hierarchically such as major classification, subclassification, subclassification, and subclassification, and selects parent categories of categories that the user wants to collect. When the category collecting button is clicked, the category collecting unit 220 automatically collects and registers sub categories of the selected category.

A method of collecting categories through a category collecting unit according to an exemplary embodiment of the present invention will be described in more detail with reference to FIG. 7.

When the user clicks the category collecting button, the category collecting unit 220 determines whether there is a selected category (S300). (S305) If there is no category selected in step S305, the category collector 220 outputs a message box with no category selected (S310). If there is a category selected in step S305, (for example, in FIG. Home appliance ') category collector 220 determines whether there is a registered subcategory.

If there is a subcategory registered in step S315 (for example, 'cleaner' in FIG. 6), the category collector 220 collects a web page of the selected category and creates a registered subcategory as a sample category. (S320, S325)

Then, the category collecting unit 220 determines whether there is a URL to be extracted from the collected category page (S335). If there is an extracted URL, the category collecting unit 220 determines whether the extracted URL is a URL similar to the sample category created in step S325. (S340) If the URL extracted in step S340 is a URL similar to the sample category, the category collector 220 generates a category inherited from the sample category, sets the category attribute, and then registers the category in the category tree list view. (S345, S350, S355) In FIG. 6, 'Gas Range', 'Refrigerator', and 'Microwave Oven' having a similar URL structure are collected as categories as 'Cleaners'.

Meanwhile, if there is no URL extracted in step S335, it is determined whether there is a category to move to (S360). If there is no category to move to, it is terminated, and if there is a category to move (for example, furniture (not shown in FIG. 6)). ), The next category is selected, and the above step S315 is repeated.

Next, referring to Figures 8, 9a and 9b will be described in detail the function of the HTML parser 230 according to an embodiment of the present invention.

The HTML parser 230 according to the embodiment of the present invention parses the general HTML source code as shown in Fig. 9A to create a tree node as shown in Fig. 9B. According to an embodiment of the present invention, desired information of a desired location is extracted using node location information in the tree. The parsing method according to an embodiment of the present invention is applicable to general markup languages such as XML and SGML as well as HTML. In the embodiment of the present invention, HTML is described as an example of a markup language.

First, the HTML parser 230 reads a document to be parsed and loads it into the memory, and then reads one character from the memory one by one and determines whether the character read from the memory is a '<' character (that is, a start mark of a markup language tag). (S400, S405, S410, S415) If the character read from the memory in the step is a '<' character, it is determined whether the extracted tag name is the correct tag after extracting the tag name. (S425, S430) If the character read from the memory in step S415 is not '<' or if the tag name is not a correct tag in step S430, the read character is processed as content. (S420)

If it is determined in step S435 that the tag name is a correct tag, after generating a new element, the tag string and the attribute of the tag are found to form a tag node and naming to form a tree. (S435, S440)

On the other hand, if the extracted tag is a new tag, it is determined whether the tag can come as a child node of the current working element (S445), and if it is a child, it is registered as a child of the current element (S450). If it cannot come as a child, it sets the parent element as the current element. (S455)

Next, a process of making a tree according to an embodiment of the present invention will be described briefly with reference to FIGS. 9A and 9B.

First, the HTML parser checks the HTML document shown in Fig. 9A by one character and extracts the tag name. Since the tag name extracted first in FIG. 9A is html, as shown in FIG. 9B, the location information is set to '1', and the node currently in operation is set to the html tag. Since the tag following in FIG. 9A is head and html may have child tags, the HTML parser registers "head" as a child node of the html node. At this time, since the head position information is registered as a child of the node (node information 1) of html, as shown in FIG. 9B, 1 is added to 1.1 and the currently working node is set to head.

Since the next tag in FIG. 9A is title, the currently working tag is head, and head is also a tag that can have child tags, the HTML parser registers the title as a child node of head, as shown in FIG. 9B. Set the location information to 1.1.1. The next tag is </ title> which is the end tag of title, so the end tag attribute is set to true and the current working node is set to head, the parent node of title. The following tag is also </ head>, which is the end tag of the head tag, so the end tag attribute of the head node is set to true, and the currently working node is set to html, which is the parent node of head. Since the next tag in FIG. 9A is a body, it is registered as a child node of html, and the position information of the body is set to 1.2 since it is a second child node of html. Certain tags such as table can have child tags, and certain tags such as img cannot have child tags, so if you repeatedly execute the above operations using this information, you will see a tree with location information as shown in Figure 9b. Can be reconfigured to In the embodiment of the present invention, as will be described later, only the desired information can be extracted using this location information.

Next, the functions of the rule generator 240 according to the embodiment of the present invention will be described in detail with reference to FIGS. 10 and 11.

10 illustrates a rule generation screen managed by the rule generator 240 according to an embodiment of the present invention.

According to an embodiment of the present invention, in order to collect specific information in HTML, the user must work in the attribute editing window 242 as shown in FIG. When the user selects a list for collecting information from the category tree list view shown in FIG. 10 and clicks the attribute edit menu button, a page to edit the attribute is opened in the right HTML view 241 and the rule is edited in the lower right. The property editing window 242 is opened. In this case, when the user moves and clicks an element to collect in the HTML view 241, the rule generator 240 displays the rule name, tag name, content, and rule of the selected area in the property edit window 242 at the bottom. Information such as information and subrule names are shown. When the user clicks the rule registration button because the information to be collected matches the information displayed in the attribute editing window 242, the rule generator 240 registers the information displayed in the attribute editing window 242 as rule information. By repeating this task, the user can continue to create rules for the information he wants to collect.

A rule generation method of the rule generator 240 according to an embodiment of the present invention will be described in detail with reference to FIG. 11.

After the user selects a node to create a rule in the category tree view shown in FIG. 10, (S500) and clicks the property editing menu. (S505) After the rule generator 240 accesses the URL of the selected node and retrieves HTML, In addition, a parse tree for the retrieved HTML is created and output to the HTML view 241. (S510, S515, S520, S525)

When the user clicks a specific element output in the HTML view (S530), the rule generator 240 finds a node in the tree and then outputs the tag name, tag content, and rule information of the found node in the property edit window 242. (S535, S540, S545) When the user clicks the rule registration button for the information output in the property edit window, the rule generator 240 registers as a rule.

Next, the functions of the robot agent 200 according to the embodiment of the present invention will be described in detail with reference to FIGS. 12 and 13.

The robot agent 200 reads rule information stored for selective collection by the manager UI 100 from the database 300. In more detail, the robot agent reads information about the site URL from the site URL information DB 310 and reads information about which part of the site is read from the site-specific rule information DB 320.

As shown in FIG. 12, the robot agent 200 includes a document collector 210, a parser 220, and a rule filter 230.

The document collection unit 210 reads information about a site to be visited from the site URL information DB 310 in advance, and then reads the URL of each site through the Internet or an intranet. The parser 220 parses the html document read by the document collector 210. The rule filter 230 reads the rule information from the site-specific rule information 320 and then creates an HTML parsing tree. At this time, the rule filter finds an element that matches the rule read from the site-specific rule information DB 320 in the parsing tree, and stores it as a specific data structure.

13 is a view showing a document collection method according to the robot agent according to an embodiment of the present invention.

First, the robot agent 200 reads the URL information from the URL information DB 310 and collects a specified URL. (S600, S605, S610) Then, the robot agent 200 reads the rule information from the rule information DB 320 to create an HTML parse tree, and then determines whether the read rule information is the end of the parse tree. (S620, S625)

If the rule information is not at the end of the parsing tree in step S630, the robot agent finds an element that matches the rule in the parsing tree and stores the matching element as a specific data structure. (S635, S640, S645) If it is determined that the end of the tree in step S630, the selected information is stored in the designated location of the DB (S650) and the process is repeated.

While the embodiments of the present invention have been described above, the present invention is not limited only to the above embodiments, and many modifications and variations are possible.

For example, the embodiment of the present invention mainly described HTML as a markup language, but can be applied to XMl and SGML.

As described above, according to the present invention, since only desired information can be extracted from information scattered on the network, accurate search is possible and quick search can be performed.

Claims (12)

In the method for selectively collecting information supplied through the network, Registering a page including a list of pages for which information is desired to be collected as a category page; A second step of registering an information page including contents to be collected most comprehensively among the category pages as a lower node of the category page; A third step of registering, as a rule, information about contents to be collected by a user among the information pages and location information for moving to the information page; And And collecting a category page to be collected at a corresponding site based on the rules registered in the third step. The method of claim 1, A fifth step of storing the rule registered in the third step in a database; And Optional information further includes a sixth step of accessing the site based on the rules stored in the database, collecting a list of information pages, accessing the information page to extract desired information, and storing the extracted information in the database. Collection method. The method of claim 1, The first step is Displaying contents of a site accessed by a user through a category URL view; And when the user selects a page displayed through the category URL view, registering content in the category URL view as a subcategory of the currently selected category. The method of claim 1, The fourth step is Determining whether there is a selected category; Determining whether there is a registered subcategory of the selected category; Collecting pages of the selected category and creating registered subcategories as sample categories; Extracting a URL from the collected pages of the selected category; If the URL extracted in the step is a URL similar to the sample category, generating a category inherited from the sample category, setting a category attribute, and then registering the category. The method of claim 4, wherein The fourth step is If there is no URL to be extracted from the collected pages of the selected category, determining whether there is a category to move; An optional method of collecting information, further comprising selecting the next category if there are categories to move. The method of claim 1, The third step is Creating a tree node by parsing source code of information that a user wants to collect among information pages; And registering location information in the tree node as a rule. The method of claim 6, How to make the tree node Extracting a tag name from the source code; Creating a tag node by finding a tag string and an attribute of the tag after generating a new element when the tag extracted in the above step is correct; Determining whether the tag extracted in the above step is a tag that can come to a child node of an element currently being worked on; Registering as a child of the current element if it can come as a child and setting the parent element as the current element if it cannot come as a child. The method according to any one of claims 2 to 7, The sixth step Reading URL information from a URL information database and collecting a designated URL; Reading rule information from a rule information DB to create a parsing tree; Finding an element in the parse tree that matches the rule; And storing the found elements in a specific data structure. In the system for selectively collecting information supplied through the network, An administrator user interface unit for designating a site to be collected, selecting a collection element to be collected in the site, and generating a rule for database information to store the selected collection element; A robot agent accessing a site to collect information based on rules for site information, collection elements, and database information designated by the administrator user interface unit to collect information; And And a database storing rule information generated by the manager user interface unit and information collected by the robot agent. The method of claim 9, The manager user interface unit A browsing unit for bringing resources on the Internet; A category manager to register and manage categories to be collected; A category collecting unit which automatically collects other categories in the same site based on the category registered by the category managing unit; A parser for analyzing various resources collected by the browsing unit to object various tags; And And a rule generator for generating location information of the objects analyzed by the parser. The method of claim 9 or 10, The database is A site URL information database that stores information about the site URL; And An optional information collection system that includes a rule information database that stores location information about elements of the site that you want to collect. The method of claim 11, The robot agent A document collector configured to read site URL information from the URL information database and read URL resources of each site through a network; A parser for parsing a resource read from a document collection unit; And And a rule filter that reads rule information from the rule information database, creates a parsing tree, and finds elements matching the rules read from the rule information database in the parsing tree and stores them in a specific data structure.