JP5701830B2 - Document structure analysis apparatus and program - Google Patents

Description

  The present invention relates to a document structure analysis apparatus and program, and more particularly to a document structure analysis apparatus and program for analyzing a hierarchical structure of elements of a structured document.

  Non-Patent Document 1 proposes a method called a web wrapper that automatically generates a document structure extraction program using a machine learning method when a combination of an HTML document and a portion to be extracted is given as a learning example.

I. Muslea, S. Minton, and C. Knoblock, “A Hierarchical Approach to Wrapper Induc-tion,” pp.190-197, AGENTS '99, 1999.

  However, in the method of Non-Patent Document 1 described above, even if the page includes similar information, the web page has a different HTML structure for each site, so in order to comprehensively acquire information from various sites, A large amount of learning examples consisting of a set of HTML documents and parts to be extracted are required, and the work by manpower is large. In addition, if the template is updated, such as when the EC site is renewed, the HTML structure of the product page changes, and the learned rules can no longer be applied. There is a need.

  The present invention has been made in view of the above circumstances, and by documenting a plurality of elements in a structured document, it is possible to cluster the elements of the structured document with high accuracy and obtain a portion to be extracted. An object is to provide an apparatus and a program.

  In order to achieve the above object, a document structure analyzing apparatus according to the present invention comprises a hierarchical structure analyzing means for analyzing a hierarchical structure of elements of the structured document for each of at least one structured document to be analyzed, Based on the analysis result analyzed by the hierarchical structure analyzing means, the positional information analyzing means for analyzing the display position of each element of the structured document when displaying the structured document analyzed by the hierarchical structure analyzing means , For each of a plurality of elements in the at least one structured document between a first element and a second element, the first element and a descendant element of the first element; the second element; Based on the structural similarity calculation means for calculating the structural similarity regarding the structure of the element by comparing with the descendant elements of the second element, and the analysis result analyzed by the positional information analysis means The position similarity calculation means for calculating the position similarity regarding the display position of each element between two elements of the at least one structured document, and the structure similarity calculation means. And clustering means for clustering a plurality of elements in the at least one structured document based on the structural similarity and the positional similarity calculated by the positional similarity calculation means.

  According to the present invention, the hierarchical structure of the elements of the structured document is analyzed for each of at least one structured document to be analyzed by the hierarchical structure analyzing means. The position information analyzing unit analyzes the display position of each element of the structured document when the structured document analyzed by the hierarchical structure analyzing unit is displayed.

  And based on the analysis result analyzed by the hierarchical structure analysis means by the structure similarity calculation means, for each between the first element and the second element of the plurality of elements in at least one structured document, The first element and the descendant element of the first element are compared with the second element and the descendant element of the second element, and the structural similarity related to the structure of the element is calculated.

  Then, based on the analysis result analyzed by the position information analysis means by the position similarity calculation means, the position regarding the display position of the element for each of the two elements of the plurality of elements in at least one structured document Calculate similarity.

  Then, the clustering unit clusters a plurality of elements in at least one structured document based on the structural similarity calculated by the structural similarity calculation unit and the position similarity calculated by the position similarity calculation unit. .

  As a result, the hierarchical structure of the elements of the structured document is analyzed, the display position of each element of the structured document when the analyzed structured document is displayed, and two of the plurality of elements in the structured document are analyzed. For each of the two elements, the structural similarity regarding the structure of the element and the positional similarity regarding the display position of the element are calculated, and a plurality of elements in the structured document are clustered based on the structural similarity and the positional similarity. As a result, the elements of the structured document can be clustered with high accuracy, and the portion to be extracted can be obtained.

  Further, the structural similarity calculation means, for each element in the at least one structured document, based on the analysis result analyzed by the hierarchical structure analysis means, the attribute name of the element and the descendant element of the element A histogram can be calculated, and the similarity between the histogram of the first element and the histogram of the second element can be calculated as the structural similarity for each between the first element and the second element.

  In addition, the position similarity calculation unit is configured to display a display position of the two elements when the structured document is displayed for each of the two elements based on the analysis result analyzed by the position information analysis unit. The similarity based on the size can be calculated as the position similarity.

  Further, for each of the clusters clustered by the clustering means, further includes a priority calculation means for calculating the priority of the cluster based on a word included in each element belonging to the cluster and a predetermined keyword. be able to.

  Further, a plurality of cluster candidates including two elements clustered in the same cluster among a plurality of elements in the at least one structured document based on the structure similarity calculated by the structure similarity calculation means. Candidate searching means for searching, further comprising: the clustering means based on the structural similarity and the position similarity between elements included in each cluster candidate searched by the candidate searching means. Multiple elements in a document can be clustered.

  In addition, the candidate search means, with respect to each of the plurality of elements in the at least one structured document, the element included in the cluster candidate and the element having the maximum structural similarity among the elements having a parent-child relationship It can be.

  Further, the candidate search means searches for the plurality of cluster candidates including two elements whose display position distance when the structured document is displayed is equal to or smaller than a predetermined value and clustered into the same cluster. be able to.

  The program according to the present invention is a program for causing a computer to function as each unit of the document structure analysis apparatus.

  As described above, according to the document structure analysis apparatus and program of the present invention, the hierarchical structure of the elements of the structured document is analyzed, and each element of the structured document is displayed when the analyzed structured document is displayed. Analyzing the position and calculating the structural similarity related to the structure of the element and the positional similarity related to the display position of the element for each of the two elements of the plurality of elements in the structured document. By clustering a plurality of elements in the structured document based on the degree, it is possible to cluster the elements of the structured document with high accuracy and to obtain a portion to be extracted.

It is the schematic which shows the structure of the document structure analysis apparatus which concerns on embodiment of this invention. It is the schematic of the keyword list KW ^ used by embodiment of this invention. It is the schematic of the hierarchical structure information table K ^ used by embodiment of this invention. It is the schematic of the positional information table P ^ used by embodiment of this invention. It is the schematic of the cluster table C ^ used by embodiment of this invention. It is the schematic of the structure similarity table ^Lstr ^ used by embodiment of this invention. It is the schematic of the position similarity table ^Lpos ^ used by embodiment of this invention. It is a figure which shows the web page of the operation example 1 used by description of embodiment of this invention. It is a figure which shows the HTML tag structure of the web page of the operation example 1 used by description of embodiment of this invention. It is a figure which shows the DOM tree structure of the operation example 1 used by description of embodiment of this invention. It is a figure which shows the hierarchical structure information table of the operation example 1 used by description of embodiment of this invention. It is a flowchart which shows the content of the structured document information extraction process routine in the document structure analysis apparatus which concerns on embodiment of this invention. It is a flowchart which shows the content of the first half part of the candidate search process routine called within a structured document information extraction process routine. It is a flowchart which shows the content of the second half part of the candidate search process routine called within a structured document information extraction process routine. It is an algorithm which shows the first half part of the candidate search process routine called within the structured document information extraction process routine. It is an algorithm which shows the latter half part of the candidate search process routine called within a structured document information extraction process routine. It is an algorithm showing the contents of the firstChild routine called in the candidate search processing routine. This is an algorithm showing the contents of the nextSibling routine called in the candidate search processing routine. This is an algorithm showing the contents of the nextNode routine called in the candidate search processing routine. It is a flowchart which shows the content of the clustering process routine called within a structured document information extraction process routine. This is an algorithm of a clustering processing routine called in the structured document information extraction processing routine. It is an algorithm which shows the content of the priority calculation process routine called within a structured document information extraction process routine. It is a figure which shows the keyword list KW ^ of the operation example 1 used by description of embodiment of this invention. It is a figure which shows the positional information table P ^ of the operation example 1 used by description of embodiment of this invention. It is a figure which shows the structure similarity table ^Lstr ^ of the operation example 1 used by description of embodiment of this invention. It is a figure which shows the position similarity table ^Lpos ^ of the operation example 1 used by description of embodiment of this invention. It is a figure which shows the cluster table C ^ of the operation example 1 used by description of embodiment of this invention. It is a figure which shows the clustering result of the operation example 1 used by description of embodiment of this invention. It is a figure which shows the output result of the operation example 1 used by description of embodiment of this invention. It is a figure which shows the web page of the operation example 2 used by description of embodiment of this invention. It is a figure which shows the HTML tag structure of the web page of the operation example 2 used by description of embodiment of this invention. It is a figure which shows the keyword list KW ^ of the operation example 2 used by description of embodiment of this invention. It is a figure which shows the output result of the operation example 2 used by description of embodiment of this invention. It is an algorithm which shows the other structural example 1 of the priority calculation process routine called within a structured document information extraction process routine. It is an algorithm which shows the other structural example 2 of the priority calculation processing routine called within a structured document information extraction processing routine. It is an algorithm which shows the other structural example 3 of the priority calculation process routine called within a structured document information extraction process routine.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<System configuration>

  The document structure analysis apparatus according to the present embodiment is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing a structured document information extraction apparatus processing routine described later. Is configured as follows.

  As shown in FIG. 1, the document structure analysis apparatus according to the present embodiment includes an input unit 1, a storage unit 2, a calculation unit 3, and an output unit 4. In addition, the calculation unit 3 includes a hierarchical structure analysis unit 10, a position information analysis unit 12, a structure similarity calculation unit 14, a position similarity calculation unit 16, a candidate search unit 18, a clustering unit 20, and a priority level. And a calculation unit 22.

  Note that the storage unit 2 may be provided outside and connected to the document structure analysis apparatus via a network.

The input unit 1 receives an input from the outside. For example, the input unit 1 receives a web page group, parameters described later (structure similarity threshold θ _str , position similarity threshold θ _pos , search range thresholds γ ₀ , γ ₁ ), An input of a keyword list KW ^ to be described later is accepted.

The storage unit 2 stores various data input from the input unit 1. Specifically, a web page group input by the input unit 1, parameters (structure similarity threshold θ _str , position similarity threshold θ _pos , search range thresholds γ ₀ , γ ₁ ), and a keyword list KW shown in FIG. Remember ^. Further, the storage unit 2 stores the results of each processing described later, specifically, the hierarchical structure information table K ^ shown in FIG. 3, the position information table P ^ shown in FIG. 4, and the cluster table shown in FIG. C, the structural similarity table L ^str ^ shown in FIG. 6, the position similarity table L ^pos ^ shown in FIG. 7, and the like are stored. Note that “＾” attached to a symbol indicates that the symbol is a matrix, a multidimensional array, or a vector. The storage unit 2 also stores the results of other processes described later.

The hierarchical structure analysis unit 10 analyzes the hierarchical structure of the element indicated by the HTML tag for each of the web pages input from the input unit 1 and stored in the storage unit 2. Specifically, a DOM (Document Object Model) tree including each node corresponding to each element indicated by the HTML tag is generated from the web page, and for each node in the DOM tree, the node of the own node and each of the descendant nodes. A histogram is created by name (attribute name), and the hierarchical structure information table K ^ shown in FIG. 3 is created.
For example, when the web page of operation example 1 shown in FIG. 8 is input, the hierarchical structure of the element indicated by the HTML tag as shown in FIG. 9 is analyzed, and the tree shown in FIG. 10 is generated. Then, a histogram as shown in FIG. 11 is created for each node in the DOM tree.

  The position information analysis unit 12 receives the display position on the web browser and the width and height of the display area of the element indicated by the HTML tag for each of the web pages input from the input unit 1 and stored in the storage unit 2. And the position information table P ^ shown in FIG. 4 is created. The width and height of the element display area are examples of the size when the element is displayed.

Based on the histogram of each node calculated by the hierarchical structure analysis unit 10, the structural similarity calculation unit 14 calculates the similarity related to the hierarchical structure between two elements for each pair of two nodes. For example, when node A and node B are given, their structural similarity S ^str (A, B) is calculated according to the following equation (1).

Here, hist ^A and hist ^B are histograms of the nodes A and B calculated by the hierarchical structure analysis unit 10.

The position similarity calculation unit 16 calculates the similarity of the positional relationship between two elements for each pair of two nodes based on the position information calculated by the position information analysis unit 12. For example, when node A and node B are given, the position similarity S ^pos ^ (A, B) of the nodes A and B is calculated according to the following equation (2).

Here, (x _A , y _A , w _A , h _A ) and (x _B , y _B , w _B , h _B ) are the position information of the nodes A and B calculated by the position information analysis unit 12. . In addition, (W _A , H _A ) and (W _B , H _B ) are the width and height of the display area (including scrolling) of the web page including A and B on the web browser.

The candidate search unit 18 receives the parameters (structure similarity threshold θ _str , search range thresholds γ ₀ , γ ₁ ) input by the input unit 1, the structure similarity calculated by the structure similarity calculation unit 14, and position information analysis Based on the node position information calculated by the unit 12, a set of nodes serving as cluster candidates is searched from each node of the DOM tree generated by the hierarchical structure analysis unit 10. Then, as a result of the candidate search, the structural similarity S ^str (A, B) calculated by the above equation (1) is stored for each cluster candidate node set (i, j), as shown in FIG. A structural similarity table L ^str ^ is created. Further, the position similarity table L ^pos shown in FIG. 7 in which the position similarity S ^pos (A, B) calculated by the above equation (2) is stored for each cluster candidate node set (i, j). Create ^. Then, the cluster table C ^ shown in FIG. 5 is created in which each node belonging to at least one cluster candidate is each cluster.
Specifically, the nodes of the DOM tree are traced, the structure similarity calculated by the structure similarity calculation unit 14 is equal to or greater than a threshold value, and the display position of the node calculated by the position information analysis unit 12 is within the search range. Detect a pair of two nodes. And a structure between the detected node pair and the one of the nodes having a parent-child relationship so that there is no parent-child relationship in the node group that is the other node. The node having the maximum similarity is selected as the other node and is set as a cluster candidate. Then, the structure similarity and position similarity of each cluster candidate are stored, and a structure similarity table L ^str ^ and a position similarity table L ^pos ^ are created and stored in the storage unit 2.

The clustering unit 20 integrates the clusters of the cluster table C ^ based on the structure similarity table L ^str ^, the position similarity table L ^pos ^, and the cluster table C ^ created by the candidate search module 18 to obtain a cluster. The nodes are clustered by repeatedly updating the table C ^. Here, among the cluster candidates whose positional similarity calculated by the positional similarity calculation unit 16 and the structural similarity calculated by the structural similarity calculation unit 14 are equal to or greater than the specified threshold, the cluster candidate having the maximum structural similarity is selected. It is repeated that the two clusters corresponding to the node pair are integrated to update the cluster table C ^.

  The priority calculation unit 22 calculates the priority of the cluster generated by the clustering unit 20. Specifically, based on the keyword list KW ^ input by the input unit 1, the hierarchical structure information table K ^ created by the hierarchical structure analysis unit 10, and the cluster table C ^ obtained by the clustering unit 20, For each node belonging to the cluster, a sum of keyword weights included in the node is calculated, and the maximum value is calculated as the priority of the cluster.

  The output unit 4 outputs the clustering results generated by the clustering unit 20 in order of priority based on the priority of each cluster calculated by the priority calculation unit 22.

<Action>
Next, the operation of the document structure analysis apparatus according to this embodiment will be described. First, a web page group, parameters (structure similarity threshold θ _str , position similarity threshold θ _pos , search range thresholds γ ₀ , γ ₁ ) and a keyword list KW ^ are input to the document structure analysis apparatus and stored in the storage unit 2. Stored. Then, the structured document information extracting device processing routine shown in FIG. 12 is executed in the document structure analyzing device.

First, in step S100, a web page group, parameters (structure similarity threshold θ _str , position similarity threshold θ _pos , search range thresholds γ ₀ , γ ₁ ) and keywords input from the storage unit 2 by the input unit 1 and keywords. Read the list KW ^.

  Next, in step S102, the hierarchical structure is analyzed for the web page. Specifically, one web page input from the input unit 1 is read from the storage unit 2, and a DOM (Document Object Model) tree including each node corresponding to each element indicated by the HTML tag is generated. For each node in the tree, a histogram is created with the node name (attribute name) of each of its own node and descendant nodes, and the hierarchical structure information table K ^ shown in FIG. 3 is created and stored in the storage unit 2.

  Next, in step S104, position information is analyzed for the web page. Specifically, one web page input from the input unit 1 is read from the storage unit 2, the display positions of each element indicated by the HTML tag on the web browser are calculated, and the positions shown in FIG. An information table P ^ is created and stored in the storage unit 2.

  In step S106, it is determined whether or not the structural information analysis in step S102 and the positional information analysis in step S104 have been completed for all the web page groups stored in the storage unit 2. If the analysis has not been completed for all of the web page groups, the process proceeds to step S102. If the analysis has been completed for all of the web page groups, the process proceeds to step S108.

  In step S108, candidate search processing routines shown in FIGS. 13 and 14 are executed in order to search for a set of nodes that are cluster candidates from each node. FIGS. 15 and 16 show algorithms corresponding to the candidate search processing routines shown in FIG. 13 and FIG.

<Candidate search processing routine>
In the candidate search processing routine, a process of searching for a set of nodes that are cluster candidates from each node is executed. The candidate search processing routine is executed for each pair of web pages obtained from the web page group. In addition, a candidate search processing routine is executed for each pair of the same web page.

First, in step S200, a web page pair (pageA, pageB) to be processed, parameters (threshold value θ _str of structure similarity, parameters γ ₀ , γ ₁ specifying a search range), and the above-described step S104 are created. The position information table P ^ is read again.

  Next, in step S202, it is determined whether or not pageA and pageB are the same. If pageA and pageB are the same, the process proceeds to step S204. If pageA and pageB are not the same, the process proceeds to step S206.

In step S204, a value of γ ₀ is input to the threshold value γ that specifies the search range. In step S206, the value of γ ₁ is input to γ. Here, γ ₀ and γ ₁ are parameters for designating search ranges on the same page and different pages, respectively.

  Next, in step S208, the firstChild routine shown in FIG. 17 is called and pageA is input to the firstChild routine. Then, the value returned by the firstChild routine is assigned to the node number n1 indicating one node of the set of nodes that can be cluster candidates. The firstChild routine returns the node number of the first child node of pageA when pageA has a child node. If pageA has no child nodes, it returns φ (empty).

  Next, in step S210, it is determined whether or not the value of n1 calculated in step S208 is φ (empty). If the value of n1 is not φ (empty), the process proceeds to step S212. When the value of n1 is φ (empty), the process proceeds to step S306.

  Next, in step S212, φ (empty) is substituted for the node number bnode indicating the candidate node of the other cluster candidate, and −1 is substituted for score.

Next, in step S214, the position information corresponding to the n1st node number of the position information table P ^A ^ for pageA is substituted into (x ₁ , y ₁ , w ₁ , h ₁ ).

  In step S216, it is determined whether pageA and pageB are the same. If pageA and pageB are the same, the process proceeds to step S218. If pageA and pageB are not the same, the process proceeds to step S220.

  In step S218, the nextSibling routine shown in FIG. 18 is called and n1 is input to the nextSibling routine. Then, the value returned by the nextSibling routine is substituted into the node number n2 indicating the other node of the set of nodes that can be cluster candidates. The nextSibling routine returns the node number of the first sibling node of the node corresponding to the node number n1, when the node corresponding to the input node number n1 has a sibling node. If the node corresponding to the node number n1 does not have a sibling node, φ (empty) is returned.

  In step S220, the firstChild routine shown in FIG. 17 is called, and pageB is input to the firstChild routine. The value returned by the firstChild routine is assigned to n2.

  Next, in step S222, it is determined whether or not the value of n2 calculated in step S218 or step S220 or the value of n2 previously calculated in step S240 described later is φ (empty). If the value of n2 is not φ (empty), the process proceeds to step S224. When the value of n1 is φ (empty), the process proceeds to step S300.

Next, in step S224, the structural similarity S ^str is based on n1 previously calculated in step S208 or step S304 described later and n2 previously calculated in step S218 or step S220 or step S240 described later. (N1, n2) is calculated, and the structural similarity S ^str (n1, n2) is substituted into the structural similarity value sim for comparison.

Next, in step S226, it is determined whether or not sim calculated in step S224 is equal to or greater than the parameter (structural similarity threshold θ _str ) input in step S200. If sim is equal to or greater than the structural similarity threshold θ _str , the process proceeds to step S228. If sim is less than the structural similarity threshold θ _str , the process proceeds to step S240.

Next, in step S228, the position information corresponding to the n2nd node number of the position information table P ^B ^ for pageB is substituted into (x ₂ , y ₂ , w ₂ , h ₂ ).

Next, in step S230, the value of γ input in step S204 or step S206, (x ₁ , y ₁ , w ₁ , h ₁ ) substituted in step S214, and substituted in step S228 above. A determination is made based on (x ₂ , y ₂ , w ₂ , h ₂ ). Specifically, x ₂ −x ₁ ≦ γw ₁ , x ₁ −x ₂ ≦ γw ₂ , y ₂ −y ₁ ≦ γh ₁ , and y ₁ −y ₂ ≦ If it is determined that γh ₂ , the process proceeds to step S232. Otherwise, the process proceeds to step S240.

  Next, in step S232, the bnode in which the value is substituted in step S212, the bnode in which the previous value is substituted in step S234 or step S238 described later, and n2 in which the value is substituted in step S218 or step S220 described later In step S240, determination is performed based on n2 to which the previous value is substituted. Specifically, when it is determined that bnode ≠ φ (empty) and n2 is not in a parent-child relationship with bnode, the process proceeds to step S234. Otherwise, the process proceeds to step S236.

In step S234, the maximum structural score value of the structural similarity in the bnode having the parent-child relationship calculated in the previous step S234 is stored in L ^str ({n1, bnode}) of the structural degree similarity table L ^str ^ or described later. In step S238, the previously calculated score value is substituted. Then, the value of the position similarity S ^pos (n1, bnode) is substituted into L ^pos ({n1, bnode}) of the position similarity table L ^pos ^. Further, n1 into which the value is substituted in step S208 or the value of n1 previously calculated in step S304 described later is substituted into C (n1) of the cluster table C ^. Then, the value of bnode previously calculated in step S234 or the value of bnode previously calculated in step S238 described later is substituted into C (bnode) of the cluster table C ^. Further, the value of n2 calculated in step S218 or step S220 or the value of n2 calculated last time in step S240 described later is substituted into bnode. Then, the value of sim calculated in step S224 is substituted for score.

  In step S236, the score value calculated in step 212 above, the score value previously calculated in step S234, or the score value previously calculated in step S238 described later, and the sim value calculated in step S224 above. A determination is made based on the value. Specifically, if it is determined that the score value is smaller than the sim value, the process proceeds to step S238. When it is determined that the score value is equal to or greater than the sim value, the process proceeds to step S240.

  Next, in step S238, the value of n2 calculated in step S218 or step S220 or the value of n2 calculated in the previous step in step S240 described later is substituted for bnode. Then, the value of sim calculated in step S224 is substituted for the maximum value score of the structural similarity in the bnode having the parent-child relationship.

  Next, in step S240, the nextNode routine shown in FIG. 19 is called, and pageB and n2 are input to the nextNode routine. The value returned by the nextNode routine is assigned to n2. The nextNode routine returns a node number indicating the node of the first child node of the node n2 when the node n2 in the input pageB has a child node. When the node n2 has no child node, a node number indicating the first sibling node of the node n2 is returned. If the node n2 has no sibling node, a node number indicating the next sibling node of the parent node of the node n2 is returned. Otherwise, returns φ (empty).

  In step S300, it is determined whether or not the bnode value calculated in step S212 or the bnode value calculated in step S234 or S238 is φ (empty). If the value of bnode is not φ (empty), the process proceeds to step S302. When the value of bnode is φ (empty), the process proceeds to step S304.

In step S302, the score value calculated in step S234 or the score value calculated in step S238 is substituted into L ^str ({n1, bnode}) of the structural degree similarity table L ^str ^. Then, the value of the position similarity S ^pos (n1, bnode) is substituted into L ^pos ({n1, bnode}) of the position similarity table L ^pos ^. Further, the value of n1 calculated in step S208 or the value of n1 calculated last time in step S304 described later is substituted into C (n1) of the cluster table C ^. Then, the value of bnode calculated in step S234 or step S238 is substituted into C (bnode) of the cluster table C ^.

  Next, in step S304, the nextNode routine shown in FIG. 19 is called, and pageA and n1 are input to the nextNode routine. Then, the value returned by the nextNode routine is substituted for n1.

In step S306, the cluster table C ^, the structure similarity table ^Lstr ^, and the position similarity table ^Lpos ^ are output as results and stored in the storage unit 2 as processing results of the candidate search processing routine.

<Clustering processing routine>
After the candidate search process routine is completed, the process returns to step S110 of the structured document information extraction process routine shown in FIG. In step S110, the clusters of the cluster table C ^ are integrated based on the cluster table C ^, the structural similarity table ^Lstr ^, and the position similarity table ^Lpos ^ created by the candidate search processing routine in step S108. Repeat. In step S110, the clustering processing routine shown in FIG. 20 is executed. FIG. 21 shows an algorithm corresponding to the clustering processing routine shown in FIG.

In step S400, the cluster table C ^, structure similarity table ^Lstr ^, position similarity table ^Lpos ^ calculated in step S108, and parameters (structure similarity threshold _θstr , position similarity threshold _θpos ) are calculated. Is read from the storage unit 2.

Next, in step S402, −1 is assigned as an initial value to the cluster numbers p and q indicating candidate clusters to be integrated, and the maximum value maxS ^str of the structural similarity in a pair of candidate clusters to be integrated.

In step S403, a cluster pair ({i, j}, S ^str ) to be processed is set based on the structure similarity table L ^str ^.

Next, in step S404, maxS ^str is compared with the structural similarity S ^str . If maxS ^str is less than S ^str , the process proceeds to step S406. If maxS ^str is greater than S ^{str, the} process proceeds to step S412.

In step S406, the value L ^pos ({i, j}) of the position similarity table L ^pos ^ read in step S400 is substituted for the position similarity S ^pos ^. Here, the position similarity S ^pos ^ = (Sx ^pos , Sy ^pos , Sw ^pos , Sh ^pos ).

In step S408, the position similarity S ^pos ^ = (Sx ^pos , Sy ^pos , Sw ^pos , Sh ^pos ) substituted in step S406 is compared with the position similarity threshold θ _pos . Then, a Sx _^pos ≧ ^{θ pos,} and whether it is Sw _^pos ≧ ^{θ pos,} or a Sy ^pos ≧ ^θ _po, and in the case of Sh _^pos ≧ θ ^pos, the process proceeds to step S410. Otherwise, the process proceeds to step S412.

In step S410, the values of (i, j, S ^str ) are substituted for p, q, and maxS ^str for which the initial values are substituted in step S402.

Then determines, in step S412, all the data contained in the structural similarity table ^{L str ^ ({i, j} }, S str) for, whether the process of steps S404~ step S410 has ended. If completed, the process proceeds to step S414. If not completed, the process proceeds to step S403.

Next, in step S414, maxS ^str whose value is updated in step S410 is compared with the structural similarity threshold θ _str . If maxS ^str is less than θ _str , the process proceeds to step S434. If maxS ^str is greater than or equal to θ _{str, the} process proceeds to step S416.

  In step S416, the cluster C ^ (q) stored in the cluster table C ^ is integrated with the cluster C ^ (q), and the cluster C ^ (p) stored in the storage unit 2 is updated.

  In step S418, φ (empty) is stored in C (q) of the cluster table C ^.

Next, in step S420, φ (empty) is set in L ^str ({p, q}) of the structural degree similarity table L ^str ^ stored in the storage unit 2 for the integrated cluster pair p, q. Store.

In step S422, φ (empty) is stored in L ^pos ({p, q}) of the position similarity table L ^pos ^ stored in the storage unit 2 for the integrated cluster pair p, q. To do.

In step S423, the cluster C _i to be processed is set based on the cluster table C ^.

Next, in step S424, the larger value of the values L ^str ({p, i}) and L ^str ({q, i}) of the structure similarity table L ^str ^ is stored in the storage unit 2. Stored in L ^str ({p, i}) of the similarity table L ^str ^.

In step S426, φ (empty) is stored in L ^str {q, i}) of the structural similarity table L ^str ^ stored in the storage unit 2.

Next, in step S428, the larger value of the values L ^pos ({p, i}) and L ^pos ({q, i}) of the position similarity table L ^pos ^ is stored in the storage unit 2. Stored in L ^pos ({p, i}) of the similarity table L ^pos ^.

In step S430, φ (empty) is stored in L ^str ({q, i}) of the position similarity table L ^str ^ stored in the storage unit 2.

In step S432, it determines whether the processing of steps S424~ step S430 for all the clusters _{C i} is not empty is completed. If completed, the process proceeds to step S402. If not completed, the process proceeds to step S423.

  In step S434, the cluster table C ^ stored in the storage unit 2 is output as a result as the processing result of the clustering processing routine.

  After the clustering processing routine is completed, the process returns to step S112 of the structured document information extraction processing routine shown in FIG. In step S112, based on the keyword list KW ^ input in step S100, the hierarchical structure information table K ^ obtained in step S102, and the cluster table C ^ obtained in the clustering processing routine in step S110, Calculate cluster priority. FIG. 22 shows a priority calculation algorithm executed in step S112.

  In step S112, for each cluster, based on the keyword list KW ^ input in step S100, the hierarchical structure information table K ^ obtained in step S102, and the cluster table C ^ created in step S110, For each node belonging to the cluster, the sum of keyword weights included in the node is calculated, and the maximum value is calculated as the priority of the cluster.

  Next, in step S114, the output unit 4 outputs the clustering results calculated in step S110 in the priority order calculated in step S112.

  Next, an actual operation example of the document structure analysis apparatus according to the present embodiment will be described below.

<Operation example 1>
In the operation example 1, an example of extracting repeated portions in the same page will be described. In the first operation example, the web page shown in FIG. 8 is input to the document structure analysis apparatus. Also, FIG. 9 shows the HTML tag structure below the body of the web page shown in FIG. The keyword list shown in FIG. 23 is input to the input unit 1. As parameters input to the input unit 1, data of the structural similarity threshold θ _str = 0.7, the position similarity threshold θ _pos = 0.8, and the search range threshold γ ₀ = 1.5 are input to the input unit 1. Entered.

  In step S102, the DOM (Document Object Model) tree shown in FIG. 10 is generated, and the hierarchical structure information table K ^ shown in FIG. 11 is created.

  In step S104, the position information table P ^ shown in FIG. 24 is created.

  Next, in step S108, the candidate search processing routine shown in FIG. 13 and FIG. 14 is executed.

  In the first step S208, n1 ← 1 is substituted.

  Next, in step S216, it is determined whether or not the input web page pair is the same. In the operation example 1, the paired web pages are the same, and the process proceeds to step S218.

  In the next step S218, since there is no sibling (sibling node) of n1, n2 ← φ (empty) is substituted.

  In step S304, n1 ← 2 is substituted.

  In the second step S218, n2 ← 14 is substituted.

  In step S224, the structural similarity calculation unit 16 is called. In the structural similarity calculation unit 16, from the hierarchical structure information table K ^ shown in FIG. 11 calculated by the hierarchical structure analysis unit 10 in step S102, K ^ (2) = (0, 10, 2), K ^ ( 14) = (0, 9, 2) is read. When the value read from the hierarchical structure information table K ^ is substituted into the above equation (1) and calculated, the structural similarity is calculated as follows.

The structural similarity S ^str calculated by the above equation (3) is substituted into sim.

Then, in the next step S226, it is determined whether or not the structural similarity ^Sstr calculated by the above equation (3) is equal to or _larger than the structural similarity threshold _θstr . Since the structural similarity S ^str (2,14) ≈0.92 calculated by the above equation (3) and the structural similarity threshold θ _str = 0.7, the structural similarity S ^str (2,14) The relationship between sim and θ _str into which the value is substituted is sim ≧ and θ _str . Accordingly, it is determined to proceed to step S228.

Next, in step S228, since n2 ← 14 is substituted in step S218, the position information corresponding to the 14th node number of the position information table P ^B ^ is (x ₂ , y ₂ , w ₂ , h ₂ ).

Next, in step S230, (x ₁ , y ₁ , w ₁ , h ₁ ) = (0.56, 0.37, 12, 8,. 5), (x ₂ , y ₂ , w ₂ , h ₂ ) = (0.56, 9.25, 4, 0.7), and γ = 1.5 is substituted in step S204. Therefore, according to the determination formula of step S230, it is determined that the value is within the range.

  Since the candidate bnode ← φ (empty) is substituted, the candidate bnode ← 14 is substituted, the value of score ← 0.92 is substituted in step S238, and the candidate is updated.

  Next, in step S240, n2 ← 15 is substituted.

Next, in the second step S224, the above processing is performed for n1 (= 2) and n2 (= 15). Here, since the structural similarity S ^str (2,15) ≈0.41 between n1 (= 2) and n2 (= 15) and less than the structural similarity threshold θ _str = 0.7, nothing is done. Check the next node. Up to the node 24, the structure similarity S ^str is less than the structure similarity threshold θ _str = 0.7, so nothing is done.

When n2 becomes node 25, the structural similarity S ^str (2,25) ≈0.92 between n1 = 2 and n2 = 25, and the display position determined in step S230 is within the range. The process proceeds to step S232.

  In step S232, the current candidate bnode (= 14) is compared with the node 25. Since the node 25 is not in a parent-child relationship with the node 14, the process proceeds to step S234.

In step S234, the value of the score into which the value of the structural similarity of the pair of the node 2 and the node 14 is substituted is stored in the structural similarity table L ^str ^. Further, the position similarity of the set of the node 2 and the node 14 is calculated and stored in the position similarity table L ^str ^. The current candidate bnode ← 25 is updated.

  Similarly, when the above process is advanced, when the process proceeds to n1 = 5 and n2 = 8, the condition of the structural similarity in step S226 and the condition of the display position in step S230 are satisfied. Update.

  Next, when n1 = 5 and n2 = 10, the structural similarity condition in step S226 and the display position condition in step S230 are satisfied again, but nodes 8 and 10 are in a parent-child relationship, and the node 10 Since the structural similarity is high, the process proceeds to step 236, and the pair of the node 5 and the node 8 is not saved.

As a result of the processing as described above, as the execution result of the candidate search processing routine in step S108 shown in FIG. 12, the structure similarity table L ^str ^ as shown in FIG. 25 and the position similarity as shown in FIG. A table L ^pos ^ and a cluster table C ^ as shown in Fig. 27 are created.

  Next, in step S110 shown in FIG. 12, the clustering processing routine shown in FIG. 20 is executed.

For example, for the nodes 2 and 25, as shown in FIG. 25, the structural similarity S ^str (2,25) ≈0.92, which is equal to or greater than the threshold (θ _str = 0.7), As shown in FIG. 26, the position similarity S ^pos (2,25) ≈ (0.36, 1,0.67,0.67), and the condition of step S408 of the clustering processing routine shown in FIG. Since it does not satisfy, it is not integrated.

  As a result of the above processing, the clustering result is as shown in FIG.

  After the clustering processing routine is completed, the process returns to step S112 of the structured document information extraction processing routine shown in FIG. In step S112, the cluster priority is calculated based on the cluster table C ^ created by the clustering processing routine in step S110.

  Accordingly, in step S112, the algorithm processing shown in FIG. 22 is repeated for each cluster shown in FIG. For example, the cluster 2 shown in FIG. 28 includes {2, 14} nodes. As shown in FIG. 11, the node 2 has the keywords “title” and “author” shown in FIG. The priority is 2. Similarly, the priority of the node 14 is 2. Since the priority of the cluster is the maximum value of the priority of each node, the priority of cluster 2 is 2. Further, since the node 6 includes the “title” and the node 7 includes the “author”, the maximum value thereof is the cluster priority, so the priority is 1.

In the next step S114, the clustering results calculated in step S110 are output in the order of priority calculated in step S112.
Therefore, in step S112, clusters are output in order of priority as shown in FIG.

  Referring to FIG. 8 and FIG. 29 above, it can be seen that a certain repetition having a hierarchical structure is correctly extracted.

<Operation example 2>
In the operation example 2, an example of extracting repeated portions of different pages will be described. In the operation example 2, pageA and pageB shown in FIG. 30 are input to the document structure analysis apparatus as web pages. FIG. 31 shows an HTML tag structure below the body of the web page shown in FIG. In addition, the keyword list shown in FIG. As parameters input to the input unit 1, data of the structural similarity threshold θ _str = 0.7, the position similarity threshold θ _pos = 0.8, and the search range threshold γ ₁ = 0.5 are input to the input unit 1. Entered.

  When each data in the operation example 2 is input, the structured document information extraction processing routine shown in FIG. 12 is executed, and the result shown in FIG. 33 is output.

  Referring to FIG. 30 and FIG. 33, it can be seen that the correspondence of elements between pages is correctly extracted.

  As described above, according to the document structure analysis apparatus according to the present embodiment, the hierarchical structure of the elements of the structured document is analyzed, and each element of the structured document when the analyzed structured document is displayed is analyzed. Analyzing the display position and calculating the structural similarity regarding the structure of the element and the positional similarity regarding the display position of the element for each of the two elements of the plurality of elements in the structured document. By clustering a plurality of elements in the structured document based on the similarity, the elements of the structured document can be clustered with high accuracy.

  Further, according to the document structure analysis apparatus according to the present embodiment, it is possible to extract a repetitive structure that is likely to be meaningful to humans (that is, arranged adjacently like a table or a list).

  Also, according to the present embodiment, the candidate search unit can limit the cluster candidates to those that do not have a parent-child relationship, so that even if there is a hierarchical repetitive structure, it can be extracted efficiently.

  In addition, since the priority is given to the repetitive structure, it can be efficiently presented even when a large number of clusters are generated.

  In addition, by using a histogram of node names in the structure similarity calculation, it is possible to compare structures at a high speed with a constant calculation amount (number of tag names) regardless of the number of child nodes.

  In addition, when learning data consisting of a combination of an HTML document and a portion to be extracted is created from scratch, a keyword can be changed as assumed data to change the order of presentation of clustering results.

  Further, in the relearning by updating the site, elements already serving as learning data can be efficiently acquired from the clustering result by setting the already extracted data as keywords.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  In the present embodiment, the structural similarity calculation unit 14 calculates the similarity related to the hierarchical structure between two elements for each pair of two nodes based on the node histogram calculated by the hierarchical structure analysis unit 10. However, other configurations may be used.

  For example, when the node A and the node B are given to the structural similarity calculation unit 14, the HTML start tag and end tag of the subtree having these nodes as roots are set as one character, and two character strings are It is good also as a structure which calculates normalized edit distance (ND) and returns 1-ND as structural similarity. For example, normalized edit distance is described in B. Liu, "Web Data Mining," pp.384-386, Springer, 2007.

  Further, in the present embodiment, the priority calculation unit 22 sets the priority of each node as the sum of the weights of the keywords included in each node for the nodes included in one cluster based on the keyword list KW ^. Although the maximum value is calculated as the priority, other configurations may be used.

  For example, the priority calculation unit 22 specifies 1 if a keyword list and a cluster are specified, and all keywords are included in any one of the constituent elements (nodes) of the cluster, and 0 otherwise. Can be returned. An example of this algorithm is shown in FIG.

  Further, the priority calculation unit 22 can specify a cluster and return the number of components (nodes) of the cluster as a priority. An example of this algorithm is shown in FIG.

  Further, the priority calculation unit 22 designates a cluster, and for each node included in the cluster, the number of leaf nodes on the web page is N, and the number of leaf nodes included in the subtree having the node included in the cluster as a root is included. n, the priority of each node

The average value can be returned as the cluster priority. An example of this algorithm is shown in FIG.

  In addition, the document structure analysis apparatus described above has a computer system inside, but the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. .

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10 Hierarchical structure analysis part 12 Position information analysis part 14 Structure similarity calculation part 16 Position similarity calculation part 16 Structure similarity calculation part 18 Candidate search part 20 Clustering part 22 Priority calculation part

Claims (8)

Hierarchical structure analyzing means for analyzing a hierarchical structure of elements of the structured document for each of at least one structured document to be analyzed;
Position information analyzing means for analyzing the display position of each element of the structured document when the structured document analyzed by the hierarchical structure analyzing means is displayed;
Based on the analysis result analyzed by the hierarchical structure analyzing means, the first element and the second element for each of the first element and the second element among the plurality of elements in the at least one structured document. A structural similarity calculation means for comparing the second element and the descendant element of the second element with each other and calculating a structural similarity related to the structure of the element;
Based on the analysis result analyzed by the position information analysis means, the position similarity for calculating the position similarity regarding the display position of the element for each of two elements among the plurality of elements in the at least one structured document Degree calculation means,
Clustering means for clustering a plurality of elements in the at least one structured document based on the structure similarity calculated by the structure similarity calculation means and the position similarity calculated by the position similarity calculation means; Document structure analysis device including   The structural similarity calculation means, based on the analysis result analyzed by the hierarchical structure analysis means, for each element in the at least one structured document, a histogram of attribute names of the element and descendant elements of the element 2. The document structure according to claim 1, wherein the document structure is calculated and a similarity between the histogram of the first element and the histogram of the second element is calculated as the structure similarity for each of the first element and the second element. Analysis device.   The position similarity calculation unit is configured to display a display position and a size of the two elements when the structured document is displayed for each of the two elements based on the analysis result analyzed by the position information analysis unit. The document structure analysis apparatus according to claim 1, wherein a similarity based on the height is calculated as the position similarity.   The system further includes priority calculation means for calculating a priority of the cluster based on a word included in each element belonging to the cluster and a predetermined keyword for each cluster clustered by the clustering means. The document structure analysis apparatus according to any one of claims 1 to 3. Based on the structural similarity calculated by the structural similarity calculation means, a plurality of cluster candidates including two elements clustered in the same cluster among the plurality of elements in the at least one structured document are searched. Further including candidate searching means,
The clustering unit clusters a plurality of elements in the at least one structured document based on the structural similarity and the position similarity between elements included in each cluster candidate searched by the candidate searching unit. The document structure analysis apparatus according to any one of claims 1 to 4.   The candidate search means sets, for each of a plurality of elements in the at least one structured document, an element included in the cluster candidate as an element having the maximum structural similarity among elements having a parent-child relationship. The document structure analyzing apparatus according to claim 5.   The candidate search means searches for the plurality of cluster candidates including two elements whose display position distance when the structured document is displayed is not more than a predetermined value and clustered into the same cluster. 5. The document structure analysis apparatus according to 5 or 6.   The program for functioning a computer as each means of the document structure analysis apparatus of any one of Claims 1-7.