JP5225331B2 - Data extraction apparatus and method - Google Patents

Description

  The present invention relates to a data extraction apparatus and method.

  In recent years, with the spread of the Internet and the like, a user can access the Internet or the like from a terminal and easily browse various web pages. A web page is composed of a document described in HTML (HyperText Markup Language) or the like and image data, and is viewed by a web browser. By browsing and searching such a web page, the user can easily obtain the information he / she wants.

  A technique for retrieving information from a web page includes a technique for extracting specific information using a description format such as HTML. These techniques use a tag structure in HTML or the like, and extract information using the commonality of the tag structure. For example, Patent Documents 1 to 3 are known.

  Patent Document 1 discloses a system that creates an extraction rule that can specify only a body part from information provided on the WWW. The system disclosed in Patent Document 1 includes a text extracting means for extracting a text part from web page data collected in advance, and analyzing data on the web page data to indicate a location where the text part extracted by the text extracting means appears. An extraction rule creation means for creating an extraction rule expressed by a structure, and an application for grouping a plurality of URLs to which the same extraction rule created by the extraction rule creation means is applied, and associating the grouped URLs with the extraction rules Extraction rule creating means.

  Patent Document 2 discloses a system that allows a general user to easily extract and use text contents having useful information without analyzing tags or creating extraction rules. The system disclosed in Patent Literature 2 includes a storage unit that stores a pattern format having a regular expression, an extraction rule generation unit that generates an extraction rule that extracts text content that matches the pattern format from an HTML page, and a predetermined number of extraction rules. A format conversion unit for converting to the above format.

  Patent Document 3 discloses an information extraction device that extracts specific information related to a certain search term from a page of an unspecified website in units of blocks. The information extraction device disclosed in Patent Document 1 is based on a pattern set including patterns representing a structural positional relationship between a search word in semi-structured information described in HTML or the like and specific information related to the search word. Specific information candidates related to a search term are extracted in block units from a set of web pages, and specific information is selected from the extracted specific information candidates using an information classification technique. For example, the information extraction device uses a learning process to determine a pattern (pattern tree, distance between nodes) from a structural positional relationship (a tree structure based on a tag structure) between a search word and specific information related to the search word in the acquired web page Etc.). Then, the information extraction device applies the pattern to the web page, extracts information contained in the subtree rooted at the node matching the pattern tree and the target node whose distance between the nodes is within the limit, and specifies the specific information. Specific information is selected using an information classification technique.

JP 2004-220251 A JP 2006-236262 A JP 2007-47974 A

  However, since the invention disclosed in Patent Document 1 specifies and extracts only the body part from the web page data, it cannot extract the same data as the data specified and desired by the user. The invention disclosed in Patent Document 2 takes out text content that matches the pattern format from the HTML page, so the user needs to understand and specify the pattern format in order to obtain the desired data. Since the invention disclosed in Patent Document 3 extracts a node whose distance between nodes is within the limit from a web page as a matching node, it also extracts data having a meaning different from that specified by the user. Resulting in. Furthermore, since the invention disclosed in Patent Document 3 selects specific information using information classification technology with the extracted information as a candidate, it is expected that it takes time until extraction.

  Therefore, there is a need for a data extraction apparatus and method that can easily and quickly extract data desired by a user from an HTML document.

  It is an object of the present invention to provide a data extraction apparatus and method that can easily and quickly extract data desired by a user from an HTML document.

  The present invention provides the following solutions.

  (1) an HTML document storage unit that stores an HTML document, a target node reception unit that receives designation of two or more target nodes among the nodes constituting the HTML document stored in the HTML document storage unit, An ancestor node specifying unit that specifies a common ancestor node that is an upper node common to all the target nodes received by the target node receiving unit, and the specified from the common ancestor node specified by the ancestor node specifying unit Based on the path extraction means for extracting all paths to the target node, and the repetition structure of the nodes constituting the HTML document and the path extracted by the path extraction means, the extraction target node extraction rules to be extracted are A search path from the common ancestor node to the extraction target node is generated. A data extraction apparatus comprising: a search path generation means; and a data extraction means for extracting the extraction target node extracted according to the search path generated by the search path generation means from the HTML document including the common ancestor node .

  According to the configuration of (1), the data extraction apparatus according to the present invention has the HTML document storage means for storing the HTML document. The data extraction apparatus accepts designation of two or more target nodes among the nodes constituting the HTML document stored in the HTML document storage means, and is a common ancestor that is an upper node common to all the accepted target nodes. Identify the node. Next, the data extraction device extracts all paths from the specified common ancestor node to the designated target node, and based on the extracted path and the repetition structure of the nodes constituting the HTML document, the extraction purpose to be extracted A search path from the common ancestor node to the extraction target node indicating the node extraction rule is generated. Then, the data extraction device extracts the extraction target node extracted according to the generated search path from the HTML document including the common ancestor node.

  That is, the data extraction apparatus according to the present invention specifies a common ancestor node of all target nodes that have received a specification, generates a search path from the specified common ancestor node to the extraction target node, and extracts according to the generated search path Extract the target node to be extracted. Therefore, the data extraction apparatus according to the present invention can accept designation of a target node as a hint from the user, and can easily and quickly extract data desired by the user from the HTML document based on the hint. .

  (2) The search path generation unit generates the search path having an expression including a part of a wild card that matches a plurality of patterns based on all the paths extracted by the path extraction unit (1). The data extraction device described in 1.

  Therefore, the data extraction apparatus according to (2) generates a search path having an expression that partially includes a wild card that matches a plurality of patterns, based on all the extracted paths. Thereby, even when fluctuations are included in the repetitive structure of the HTML document, the data desired by the user can be easily extracted from the HTML document.

  (3) The search path generation unit performs DP (Dynamic Programming) matching on all the paths extracted by the path extraction unit, and generates a search path having an expression partially including the wildcard (2 ) Data extraction device.

  Therefore, the data extraction apparatus according to (3) has an expression including a wild card as a part of the DP matching of these paths even when the plurality of extracted paths do not completely match. Estimate and generate a search path. Thereby, even when fluctuations are included in the repetitive structure of the HTML document, the data desired by the user can be extracted from the HTML document more easily.

  (4) The ancestor node specifying means is a common upper node of the two target nodes by combining in order from the lower target node of the HTML document among the two or more target nodes that have received the designation. This is repeated until the number of the common upper node becomes one, the means for obtaining the first common node, the means for obtaining the second common node that is the common upper node of the two first common nodes, and the one common upper node. Means for identifying two common upper nodes as the common ancestor node, wherein the path extraction means extracts a path from the common ancestor node to a common upper node that is one level lower than the common ancestor node; If the upper node is not the extraction target node, a path from the common upper node one level lower to the common upper node one level lower is extracted. And a repeating unit that repeats this to extract a path to the extraction target node, and the search path generation unit includes a search path from the common ancestor node to a common upper node that is one level lower, Means for performing DP matching on the extracted path, and if the one-level lower common upper node is not the extraction target node, the one-level lower common upper node to the one-level lower common upper node (1) including means for generating a search path by performing DP matching of the extracted path, and means for generating a search path to the extraction target node by repeating this search path. Data extraction device.

  According to the configuration of (4), the data extraction apparatus according to (4) is combined in order from the target node in the lower order of the HTML document among the two or more target nodes that have received the designation in (1). Find the first common node that is the common upper node of the two target nodes, find the second common node that is the common upper node of the two first common nodes, and repeat this until there is one common upper node The one common upper node is specified as a common ancestor node. Next, the data extraction apparatus according to (4) extracts a path from the common ancestor node to the common upper node that is one step lower, and when the common upper node that is one step lower is not the extraction target node, the common upper node that is one step lower A path from the node to the common upper node one level lower is extracted, and this is repeated to extract a path to the extraction target node. Then, the data extraction apparatus according to (4) generates a search path from the common ancestor node to the common upper node that is one step lower by performing DP matching of the extracted path, and the common upper node that is one step lower is used for the extraction purpose. If it is not a node, a search path from the common upper node one level lower to the common upper node one level lower is generated by DP matching of the extracted path, and this is repeated until the extraction target node Generate search path for.

  In other words, the data extraction device according to (4) specifies the common ancestor node of all target nodes that have received the designation, generates a search path from the specified common ancestor node to the extraction target node for each stage, and generates The extraction target node extracted according to the generated search path is extracted by the search path for each stage. Therefore, the data extraction apparatus according to (4) receives designation of a target node serving as a hint from the user, and extracts data desired by the user from the HTML document more easily and at high speed based on the hint. be able to.

  (5) A method executed by a data extraction apparatus having an HTML document storage means for storing an HTML document, wherein two or more target nodes among the nodes constituting the HTML document stored in the HTML document storage means A target node receiving step for receiving a designation of an ancestor, an ancestor node specifying step for specifying a common ancestor node that is an upper node common to all the target nodes received in the target node receiving step, and specifying in the ancestor node specifying step A path extraction step for extracting all paths from the common ancestor node to the designated target node, and a repetition structure of the nodes constituting the HTML document and the path extracted in the path extraction step, Indicates the extraction rule for the extraction target node to be extracted A search path generating step for generating a search path from the common ancestor node to the extraction target node; and the HTML document including the common ancestor node is extracted according to the search path generated in the search path generating step A data extraction step of extracting an extraction target node.

  Therefore, as in (1), the method according to the present invention can accept data from a user and extract data desired by the user from an HTML document easily and at high speed.

  According to the present invention, data desired by a user can be easily and quickly extracted from an HTML document.

It is a figure which shows the example of the HTML document data for demonstrating the characteristic of this invention. It is a functional block diagram which shows the function structure of the data extraction apparatus which concerns on one Embodiment of this invention. It is a figure which shows another example of the HTML document data of the data extracting device which concerns on one Embodiment of this invention. It is a figure which shows the example which performs DP matching in the production | generation of the search path | pass of the data extraction device which concerns on one Embodiment of this invention. It is a flowchart which shows the processing content of the data extraction device which concerns on one Embodiment. It is a flowchart which shows the common ancestor node specification process of the data extraction device which concerns on one Embodiment. It is a flowchart which shows the search path | pass production | generation process of the data extracting device which concerns on one Embodiment. It is a figure showing an example of data extraction of a data extraction device concerning one embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The data extraction device 10 of this embodiment is applied to a computer and its peripheral devices. Each unit in the present embodiment is configured by hardware included in a computer and its peripheral devices, and software that controls the hardware.

  The hardware includes a storage unit, a communication device, a display device, and an input device in addition to a CPU (Central Processing Unit) as a control unit. Examples of the storage unit include a memory (RAM: Random Access Memory, ROM: Read Only Memory, etc.), a hard disk drive (HDD: Hard Disk Drive), and an optical disk (CD: Compact Disc, DVD: Digital Versatile Drive, etc.). It is done. Examples of the communication device include various wired and wireless interface devices. Examples of the display device include various displays such as a liquid crystal display and a plasma display. Examples of the input device include a keyboard and a pointing device (mouse, tracking ball, etc.).

  The software includes a computer program and data for controlling the hardware. The computer program and data are stored in the storage unit, and are appropriately executed and referenced by the control unit. The computer program and data can be distributed via a communication line, or can be recorded on a computer-readable medium such as a CD-ROM and distributed.

  FIG. 1 is a diagram showing an example of HTML document data for explaining the features of the present invention. FIG. 1A is a diagram illustrating an example of HTML document data using a material or the like in “How to Make Curry and Rice” as a specific example. FIG. 1B is a diagram showing an example of the HTML document data of FIG. In this example, the data extraction apparatus 10 accepts designation of a target node that is a hint of an extraction target node that is a purpose of data extraction. Specifically, in the example of FIG. 1A, as will be described in detail later with reference to FIG. 8, the data extraction device 10 receives a click operation on the display screen with a mouse pointer or the like, thereby “how to make curry rice”, “Beef”, “200 g”, “Carrero”, and “1/2 pack” are accepted as target nodes that are hints of the extraction target node. Then, the data extraction apparatus 10 is based on the repeated structure of the nodes constituting the HTML document, in addition to the target node that has received direct designation as a hint such as “how to make curry rice”, “beef”, “200 g”, For example, “carrot”, “one”, etc. are automatically extracted as extraction target nodes.

  In other words, the data extraction apparatus 10 includes, for example, “how to make curry and rice”, “beef”, and “as a target node that is a hint for extracting data (extraction target node) desired by the user in an HTML document. "200g", "Kararu" and "1/2 pack" are accepted. Next, the data extraction apparatus 10 combines the target node corresponding to “beef” and the target node corresponding to “200g” in combination with the first group from the lower target nodes of the HTML document as the first group. To do. Furthermore, the data extraction apparatus 10 sets the target node corresponding to “Carrero”, which is the target node in parallel with these, and the target node corresponding to “1/2 pack” as the second group. Furthermore, the data extraction apparatus 10 sets a node corresponding to “how to make curry and rice”, which is a higher-order target node, as a third group. Here, the data extraction apparatus 10 extracts <dl> which is a common upper node of the first group and the second group, and the first group and the second group are combined into a fourth group. Next, the data extraction apparatus 10 extracts <body> which is a common upper node of the combined fourth group and the third group, and the fourth group and the third group are combined into a fifth group. Then, the data extraction apparatus 10 specifies <body> as a common ancestor node because there is no other group for extracting an upper node.

  Next, the data extraction apparatus 10 estimates a search path from the identified common ancestor node <body> to the extraction target node indicating the extraction rule of the extraction target node to be extracted sequentially in the reverse order of the above processing. I will do it. That is, the data extraction apparatus 10 passes the path <body>-<h1> from the common ancestor node <body> specified as the fifth group to the target node corresponding to the “how to make curry rice” in the third group one level lower. Then, the search path <body>-<h1>-<dl> is estimated and generated from the path <body>-<dl> to the common upper node corresponding to the fourth group. Next, the data extraction apparatus 10 passes the path <dl>-<dt> from <dl>, which is the common upper node corresponding to the fourth group, to the destination node corresponding to “beef” of the first group, which is one level lower. And a path <dl>-<dd> to the target node corresponding to “200g” of the first group, and a path <dl>-<dt> to the target node corresponding to “carreaux” of the second group, and Similarly, the search path <body>-<h1>-<dl>-<dt> from the path <dl>-<dd> to the target node corresponding to “1/2 pack” of the second group to the extraction target node. -Estimate and generate <dd>. Here, since “beef”, “200 g”, “carrero”, and “1/2 pack” are hints (target nodes) for which designation is accepted, the data extraction apparatus 10 estimates and generates a search path here. Complete the process.

  Then, the data extraction apparatus 10 selects “carrot”, “1”, “onion”, “2” corresponding to the extraction target node specified according to the estimated / generated search path from the HTML document including the common ancestor node. , “Potato” and “two” are extracted. Therefore, the data extraction apparatus 10 receives, for example, “how to make curry rice”, “beef”, “200 g”, “curry roux”, and “1/2 pack” from the user, and makes the curry rice desired by the user. Therefore, the material and the amount of the material can be easily and quickly extracted from the HTML document.

  FIG. 2 is a functional block diagram showing a functional configuration of the data extraction device 10 according to an embodiment of the present invention. The data extraction apparatus 10 includes a web data DB 31 as a web data storage unit that stores web data acquired from a web server 50 connected via the Internet 70, a target node reception unit 11 as a target node reception unit, An ancestor node specifying unit 12 as an ancestor node specifying unit, a path extracting unit 13 as a path extracting unit, a search path generating unit 14 as a search path generating unit, and a data extracting unit 15 as a data extracting unit. The function of the data extracting apparatus 10 will be described in detail for each part.

  The web data DB 31 stores web data including HTML documents. For example, the description format of an HTML document has a tag structure composed of tags (see FIG. 1A). In the present invention, a tag and a text included in the tag are associated with a node, and a tree structure is applied to the HTML document.

  The target node receiving unit 11 receives the designation of two or more target nodes among the nodes constituting the HTML document included in the web data stored in the web data DB 31. For example, the target node accepting unit 11 may, for example, “how to make curry and rice”, “beef”, “200 g”, “carrero” and “ “1/2 pack” designation is accepted.

  The ancestor node specifying unit 12 specifies a common ancestor node that is an upper node common to all the target nodes received by the target node receiving unit 11. That is, the ancestor node specifying unit 12 is a common upper node of the two target nodes by combining the two or more target nodes received by the target node receiving unit 11 in order from the lower target node of the HTML document. Find one common node. Next, the ancestor node specifying unit 12 obtains a second common node that is a common upper node of the two first common nodes. Then, the ancestor node specifying unit 12 repeats the extraction of the upper node until there is one common upper node, and specifies the one common upper node as the common ancestor node.

  For example, in an HTML document that forms a tree structure according to tag types as shown in FIG. 1, the ancestor node specifying unit 12 has the target nodes “how to make curry and rice”, “beef”, “200 g”, “carrero” and Among the “1/2 pack”, the beef and “200 g” are grouped in the first group by combining the target nodes in the lower order of the HTML document, and “Carreru” and “1/2 pack” are the first group. There are two groups, and the higher-level target node “how to make curry and rice” is the third group. Next, the ancestor node specifying unit 12 extracts <dl>, which is a common upper node of the first group and the second group, and combines the first group and the second group into a fourth group. Next, the data extraction apparatus 10 extracts <body> which is a common upper node of the combined fourth group and the third group, and the fourth group and the third group are combined into a fifth group. Then, the ancestor node specifying unit 12 specifies the common upper node <body> that has become one as a common ancestor node.

  The path extraction unit 13 extracts all paths from the common ancestor node specified by the ancestor node specification unit 12 to the designated target node. That is, the path extraction unit 13 extracts a path from the common ancestor node to the common upper node one level lower. Next, the path extraction unit 13 extracts a path from the common upper node one level lower to the common upper node one level lower when the common upper node one level lower is not the extraction target node to be extracted. Then, the path extraction unit 13 repeats the extraction of the path up to the common upper node that is one step lower until the common upper node that is one step lower becomes the extraction target node, and estimates and extracts the path to the extraction target node.

  For example, the path extraction unit 13 extracts a path from the common ancestor node <body> to the extraction target node in the reverse order of the process of the ancestor node specification unit 12. That is, the path extraction unit 13 performs the path <body>-<h1> from the common ancestor node <body> to the third group “how to make curry rice” one level lower, and the common upper node corresponding to the fourth group. The path <body>-<dl> up to <dl> is extracted. Next, the path extracting unit 13 further passes the path <dl>-<dt> and the path <dl>-<dd> up to “200 g” from the fourth group to the first group “beef” that is one step lower, and Then, the path <dl>-<dt> to “Carreru” in the second group and the path <dl>-<dd> to “1/2 pack” are extracted.

  The search path generation unit 14 indicates the extraction target node from the common ancestor node that indicates the extraction rule of the extraction target node to be extracted based on the path extracted by the path extraction unit 13 and the repeated structure of the nodes constituting the HTML document. Generate a search path up to. For example, the search path generation unit 14 performs DP (Dynamic Programming) matching (see FIG. 4 described later) on all the paths extracted by the path extraction unit 13 and includes a part of wild cards that match a plurality of patterns. Generate a search path with a representation. In other words, the search path generation unit 14 generates a search path from the common ancestor node to the common upper node one level lower by performing DP matching on the extracted path. Next, when the common upper node that is one step lower is not the extraction target node, the search path generation unit 14 searches the search path from the common upper node that is one step lower to the common upper node that is one step lower to the extracted path. Generate by performing DP matching. In this way, the search path generation unit 14 repeatedly generates a search path for each stage, and estimates and generates a search path to the extraction target node.

  For example, the search path generation unit 14 extracts <body>-<h1>, which are paths extracted as search paths from the common ancestor node <body> to the common upper nodes <h1> and <dl> that are one step lower, DP matching with <body>-<dl> is performed to generate a search path <body>-<h1>-<dl>. Next, since the one-stage lower common upper node <dl> is not the extraction target node, the search path generation unit 14 further lowers the one-step lower common upper node <dt> and <dd> from the one-step lower common upper node <dl>. DP matching between <dl>-<dt>, which is the extracted path, and <dl>-<dd> is performed as a search path up to and including <body>-<h1>-<dl>-<dt >-<Dd> is generated. The search path generation unit 14 performs DP matching of paths, so that even when a plurality of extracted paths do not completely match (for example, in the case of FIG. 3 described later), a plurality of patterns are obtained. Expressed as a wild card that matches and estimates and generates an optimal search path. The path DP matching will be described in detail with reference to FIGS.

  The data extraction unit 15 extracts an extraction target node extracted according to the search path generated by the search path generation unit 14 from the HTML document including the common ancestor node. For example, the data extraction unit 15 performs “carrot” corresponding to the node searched according to the generated search path <body>-<dl>-<dt>, <body>-<dl>-<dt>-<dd. > “1” (see FIG. 1) corresponding to the searched node is extracted.

  FIG. 3 is a diagram showing another example of HTML document data of the data extraction apparatus 10 according to an embodiment of the present invention. FIG. 3A is a diagram illustrating an example of data having <img> in HTML document data.

  In the example of FIG. 3, the data extraction apparatus 10 starts from the specified common ancestor node <body>, which are extraction method nodes “how to make curry and rice”, “beef”, “200 g”, “carrero”, and “1/2”. <Body>-<dl>-<dt>-<img>? -Generate <dd>. <Img>? Is an expression as a wild card indicating that <img> immediately before “?” May or may not appear.

  That is, the data extraction device 10 searches the search path <body>-<dl>-<dt>-<dd> up to “200 g” and the search path <body>-<dl> − up to “1/2 cup”. The search paths are merged from <dt>-<img>-<dd>, and <body>-<dl>-<dt>-<img>? -Generate <dd>. The data extraction device 10 merges this search path using, for example, DP matching.

  FIG. 4 is a diagram illustrating an example in which DP matching is performed in search path generation of the data extraction apparatus 10 according to an embodiment of the present invention. In the example of FIG. 4, input 1 <body>-<dl>-<dt>-<img>-<dd> in which <img> exists until the target node is reached as in “1/2 cup”. And an input 2 <body>-<dl>-<dt>-<dd> in which <img> does not exist, such as “200 g”, shows an example of a path to reach the target node.

  In the example of FIG. 4, the data extraction apparatus 10 sets a path where input 1 and input 2 match (match) as score +1, a path omitting input 1 as score −1, and a path omitting input 2 as score−. Set to 1. Here, the data extraction apparatus 10 cannot use a path 533 in which the input 1 and the input 2 do not match (match) as a route. The data extraction apparatus 10 calculates the score of the path that reaches the target node, and sets the path having the highest calculated score as the optimal path.

  For example, in the search path from the start node 901 to the target node 902, scores are calculated for the search path A and the search path B. The search path A includes a path 511 (score + 1), a path 522 (score + 1), a path 5033 (input 1 omitted: score-1), a path 5304 (input 2 omitted: score-1), and a path 5404 (input 2). Is omitted: score -1), pass 554 (score +1), and pass 565 (score +1) reach the target node 902, so the calculated score is +1. The search path B includes a path 511 (score + 1), a path 522 (score + 1), a path 5303 (input 2 omitted: score-1), a path 543 (score + 1), a path 554 (score + 1), and a path 565 (score). Since the destination node 902 is reached by +1), the calculated score is +3. Therefore, the data extraction apparatus 10 determines that the search path B is superior to the search path A. Similarly, the data extraction apparatus 10 calculates a score of another path, determines that the search path B is the optimum path by comparison with the search path B, and expresses it as a wild card that matches a plurality of patterns. >-<Dt>-<img>? -Generate <dd>.

  FIG. 5 is a flowchart showing the processing contents of the data extraction apparatus 10 according to an embodiment.

  In step S101, the CPU (target node receiving unit 11) receives a target node for extracting data desired by the user. Thereafter, the CPU moves the process to step S102.

  In step S102, the CPU (ancestor node specifying unit 12) performs a common ancestor node specifying process, and specifies the common ancestor node of the target node received in step S101. Thereafter, the CPU moves the process to step S103.

  In step S103, the CPU (path extraction unit 13, search path generation unit 14) performs a search path generation process, and indicates a search path from the common ancestor node to the extraction target node, which indicates an extraction rule for the extraction target node to be extracted. Is generated. Thereafter, the CPU moves the process to step S104.

  In step S104, the CPU (data extraction unit 15) extracts data using the search path generated in step S103. Thereafter, the CPU ends the process.

  FIG. 6 is a flowchart showing the common ancestor node identification process of the data extraction device 10 according to an embodiment.

  In step S201, the CPU (ancestor node specifying unit 12) groups the received target nodes. More specifically, the CPU groups the target nodes by combining in order from the lower nodes based on the tree structure of the HTML document acquired from the web data DB 31. Thereafter, the CPU moves the process to step S202.

  In step S202, the CPU (ancestor node specifying unit 12) extracts a common upper node between groups. More specifically, the CPU positions the grouped group according to the type of tag, the group positioned at the lowest position among the positioned groups, and the group positioned at the same or the next higher level as the group. The common upper node of is extracted. Thereafter, the CPU moves the process to step S203.

  In step S203, the CPU (ancestor node specifying unit 12) stores a group one level lower in association with the extracted common upper node. More specifically, the CPU stores a group having the common upper node as an upper node in association with the extracted common upper node. Thereafter, the CPU moves the process to step S204.

  In step S <b> 204, the CPU (ancestor node specifying unit 12) takes one group from which common upper nodes are extracted. Thereafter, the CPU moves the process to step S205.

  In step S205, the CPU (ancestor node specifying unit 12) determines whether there are two or more groups. That is, the CPU determines whether there are two or more groups as a result of combining the groups into one. If this determination is YES, the CPU moves the process to step S202, and if NO, the CPU moves the process to step S206.

  In step S206, the CPU (ancestor node specifying unit 12) specifies the common upper node extracted last as the common ancestor node. Thereafter, the CPU ends the process, and shifts the process to the next step after the step shifts to the present process.

  FIG. 7 is a flowchart showing search path generation processing of the data extraction apparatus 10 according to an embodiment.

  In step S301, the CPU (path extraction unit 13) sets the identified common ancestor node as a start node. More specifically, the CPU sets the common ancestor node identified in step S206 as a start node. Thereafter, the CPU moves the process to step S302.

  In step S <b> 302, the CPU (path extraction unit 13) acquires a one-step lower group having the start node as a common upper node in the path to the extraction target node. More specifically, in step S203, the CPU acquires a group having the start node as the common upper node among the groups stored in association with the common upper node. Thereafter, the CPU moves the process to step S303.

  In step S303, the CPU (search path generation unit 14) obtains an optimum route to the group by DP matching based on the acquired groups (see FIG. 4). Thereafter, the CPU moves the process to step S304.

  In step S <b> 304, the CPU (search path generation unit 14) generates an optimum route including a wild card expression that matches a plurality of patterns from the common ancestor node to the group based on the obtained optimum route. Thereafter, the CPU moves the process to step S305.

  In step S305, the CPU (search path generation unit 14) determines whether or not the nodes constituting the group are extraction target nodes. If this determination is YES, the CPU moves the process to step S307, and if NO, the CPU moves the process to step S306.

  In step S306, the CPU (search path generation unit 14) sets a node constituting the group as a start node. Thereafter, the CPU moves the process to step S302.

  In step S307, the CPU (search path generation unit 14) sets the generated optimum route as a search path. Thereafter, the CPU ends the process, and shifts the process to the next step after the step shifts to the present process.

FIG. 8 is a diagram illustrating a data extraction example of the data extraction device 10 according to an embodiment of the present invention. In the example of FIG. 8, the HTML document is composed of “how to make curry and rice” and “how to make beef grated radish sauce” as follows.
<Html>
<Head>
<Title> Cooking with beef </ title>
</ Head>
<Body>
<H1> How to make curry and rice </ h1>
<Dl>
<Dt> beef </ dt><dd> 200 g </ dd>
<Dt><img src = “curry.jpg”> Carreau </ dt><dd> 1/2 pack </ dd>
<Dt> carrot </ dt><dd> 1 </ dd>
<Dt> Onion </ dt><dd> 2 </ dd>
<Dt> potato </ dt><dd> 2 </ dd>
</ Dl>
</ Body>
<Body>
<H1> How to make beef grated radish sauce </ h1>
<Dl>
<Dt><img src = “tokusan_gyuu.jpg”> loin beef shoulder </ dt><dd> 400 g </ dd>
<Dt> radish </ dt><dd> 150 g </ dd>
<Dt> sprouts </ dt><dd> 200 g </ dd>
<Dt> peppers </ dt><dd> 2 </ dd>
</ Dl>
</ Body>
</ Html>

  In the example of FIG. 8A, for example, “how to make curry and rice”, “beef” as the target node that is a hint for extracting the extraction target node in the image displayed on the display device of the data extraction device 10. , “200 g”, “Carrero”, and “1/2 pack” are designated by the mouse pointer 601 and are accepted. The example of FIG. 8A is an example in which data extraction processing is started by the extraction button 610.

  In the example of FIG. 8B, the items to be extracted from the HTML document are estimated by the target node which is the hint received by the data extraction device 10, and “carrot” and “one” to “potato” and “ “2 pieces”, “How to make beef grated radish sauce”, “Beef shoulder loin” and “400 g” to “Peppers” and “2 pieces”, image data 611 (“curry.jpg”) and image It is an example showing that the data 612 (“tokusan_gyuu.jpg”) is extracted regardless of the presence or absence.

  According to the present embodiment, the data extraction device 10 has a web data DB 31 that stores an HTML document. Then, the data extraction device 10 accepts designation of two or more target nodes among the nodes constituting the HTML document stored in the web data DB 31 from the user, and is an upper node common to all the accepted target nodes. Identify a common ancestor node. Next, the data extraction device 10 extracts all paths from the specified common ancestor node to the designated target node, and performs DP matching based on the extracted path and the repetition structure of the nodes constituting the HTML document. A search path having an expression partially including wildcards that match a plurality of patterns from the common ancestor node to the extraction target node, which indicates an extraction rule of the extraction target node to be extracted, is generated. Then, the data extraction device 10 extracts an extraction target node to be searched according to the generated search path from the HTML document including the common ancestor node. Therefore, the data extraction apparatus 10 can accept designation of a target node as a hint from the user, and can easily and quickly extract data desired by the user from the HTML document.

  As mentioned above, although embodiment of this invention was described, this invention is not restricted to embodiment mentioned above. The effects described in the embodiments of the present invention are only the most preferable effects resulting from the present invention, and the effects of the present invention are limited to those described in the embodiments of the present invention. is not.

DESCRIPTION OF SYMBOLS 10 Data extraction device 11 Target node reception part 12 An ancestor node specification part 13 Path extraction part 14 Search path generation part 15 Data extraction part 31 Web data DB
50 Web server 70 Internet

Claims (5)

HTML document storage means for storing HTML documents;
A destination node accepting means for accepting designation of two or more destination nodes among nodes constituting the HTML document stored in the HTML document storage means;
An ancestor node specifying means for specifying a common ancestor node that is a common upper node to all the target nodes received by the target node receiving means;
Path extracting means for extracting all paths from the common ancestor node specified by the ancestor node specifying means to the designated target node;
A search path from the common ancestor node to the extraction target node, which indicates an extraction rule of the extraction target node to be extracted based on the path extracted by the path extraction means and the repetition structure of the nodes constituting the HTML document Search path generation means for generating
Data extraction means for extracting the extraction target node extracted according to the search path generated by the search path generation means from the HTML document including the common ancestor node;
A data extraction device.   2. The search path generation unit according to claim 1, wherein the search path generation unit generates the search path having an expression partially including a wild card that matches a plurality of patterns, based on all the paths extracted by the path extraction unit. Data extraction device.   3. The search path generation unit performs DP (Dynamic Programming) matching on all the paths extracted by the path extraction unit to generate a search path having an expression including a part of the wild card. Data extraction device. The ancestor node specifying means includes:
Means for obtaining a first common node that is a common upper node of the two target nodes by combining in order from the lower target node of the HTML document among the two or more target nodes that have received the designation;
Means for obtaining a second common node which is a common upper node of the two first common nodes;
Repeating this until there is one common upper node, and specifying the one common upper node as the common ancestor node,
The path extraction means includes
Means for extracting a path from the common ancestor node to a common upper node one level lower;
Means for extracting a path from the one-stage lower common upper node to the one-stage lower common upper node if the one-stage lower common upper node is not the extraction target node;
And repeating means for extracting a path to the extraction target node.
The search path generation means includes
Means for generating a search path from the common ancestor node to a common upper node one level lower by performing DP matching of the extracted path;
If the lower common upper node is not the extraction target node, the search path from the lower upper common upper node to the lower common upper node is subjected to DP matching of the extracted path. Means for generating, and
And repeating this to generate a search path to the extraction target node.
The data extraction device according to claim 1. A method executed by a data extraction apparatus having an HTML document storage means for storing an HTML document,
A destination node accepting step for accepting designation of two or more destination nodes among nodes constituting the HTML document stored in the HTML document storage means;
An ancestor node specifying step for specifying a common ancestor node that is a common upper node for all the target nodes received in the target node receiving step;
A path extraction step of extracting all paths from the common ancestor node identified in the ancestor node identification step to the designated target node;
A search path from the common ancestor node to the extraction target node indicating an extraction rule of the extraction target node to be extracted based on the path extracted in the path extraction step and the repetition structure of the nodes constituting the HTML document A search path generation step for generating
A data extraction step of extracting the extraction target node extracted according to the search path generated in the search path generation step from the HTML document including the common ancestor node.

Images