TWI764491B - Text information automatically mining method and system - Google Patents

Abstract

A text information automatic mining method and a system are provided. The text information automatic exploration method can perform a depth-first search algorithm after constructing source codes of a web page as a document object tree to retrieve content of a DOM tree node with high possibility of text information based on the depth and distance.

Description

Text information automatic mining method and text information automatic mining system

The present invention relates to an information mining method and system, in particular to a textual information automatic mining method and a textual information automatic mining system.

The website may display various articles and various other content such as advertisements, links, share buttons, printing, emailing articles and other similar content. However, for readers of web pages, such as on news web pages, additional content such as advertisements can be distracting and tend to obscure the content the reader wants to read.

Therefore, how to quickly and effectively obtain important content on a web page has become one of the urgently needed goals in the art.

The technical problem to be solved by the present invention is to provide an automatic text information mining method and a text information automatic mining system aiming at the shortcomings of the prior art.

In order to solve the above-mentioned technical problems, one of the technical solutions adopted by the present invention is to provide an automatic text information mining method, which includes: configuring a web page capture module to obtain web page source code data of a target web page; configuring a model tree construction The module parses the obtained webpage source code data, obtains a plurality of document object model (Document Object Model, DOM) tree nodes according to a parsing result, and constructs a document object model (Document Object Model, DOM) according to the DOM tree nodes Object Model, DOM) tree; configure a first search module, execute a depth-first search algorithm for the DOM tree nodes of the DOM tree, to proceed in the DOM tree according to a depth-first search order, and for the DOM tree Each of these DOM tree nodes: judge whether the current DOM tree node is a text node, if not, enter the next DOM tree node, if so, judge whether the current DOM tree node is the same as the previous one. Whether a node depth between the DOM tree nodes determined to be text nodes is greater than a predetermined depth and whether a node distance is greater than a predetermined distance, wherein the node depth is determined by the DOM tree node and the previous text node. The node distance is determined by the number of node travels between the DOM tree node and the previous text node; and in response to determining the distance between the current DOM tree node and the previous text node (text node) If the depth of the node and the distance between the nodes are respectively greater than the predetermined depth range and the predetermined distance, the current context of the DOM tree node is recorded, and the next DOM tree node is entered according to the depth-first search sequence; and A storage unit is configured to store the content recorded after executing the depth-first search algorithm as a text information of the target webpage.

In some embodiments, determining whether the current DOM tree node is a text node is to determine whether the format of the current DOM tree node conforms to a text node defined in Hyper Text Markup Language (HTML).

In some embodiments, the predetermined depth is in the range of 3 to 7, the predetermined distance is in the range of 8 to 12, and both the predetermined depth and the predetermined range are positive integers.

In some embodiments, the automatic text information mining method further includes configuring a checking module to check whether the text recorded on a single DOM tree node exceeds a predetermined length, and if so, configuring the storage unit to store the checked data inner text.

In some embodiments, the automatic text information mining method further includes: configuring a second search module to determine whether the DOM tree nodes in the DOM tree both conform to the element nodes defined in HTML and conform to . A specific description tag of the DOM tree node, if so, directly obtain the content of the DOM tree node for recording.

In some embodiments, the automatic text information mining method further includes configuring the storage unit to store the text recorded by the second search module as the text information of the target webpage.

In some embodiments, the specific description tag is used to describe a material type inserted into the material placed on the target web page.

In some embodiments, the specific description tag is a <style> tag, a <script> tag or an <iframe> tag defined under DataAtom of Golang.

In order to solve the above technical problem, another technical solution adopted by the present invention is to provide a text information automatic determination and exploration system, which includes a webpage capture module, a model tree construction module, a first search module and a storage unit. The webpage capture module is configured to obtain webpage source code data of a target webpage. The model tree construction module is configured to parse the obtained web page source code data, obtain a plurality of document object model (Document Object Model, DOM) tree nodes according to a parsing result, and construct a tree node according to the DOM tree nodes Document Object Model (DOM) tree. The first search module is configured to execute a depth-first search algorithm for the DOM tree nodes of the DOM tree to travel through the DOM tree according to a depth-first search order, and for each of the DOM tree nodes One process: judge whether the current DOM tree node is a text node, if not, enter the next DOM tree node, if so, judge the current DOM tree node and the previous one judged to be a text node. Whether a node depth between the DOM tree nodes is greater than a predetermined depth and a node distance is greater than a predetermined distance, wherein the node depth is determined by the number of node levels between the DOM tree node and the previous text node, The node distance is determined by the number of node travels between the DOM tree node and the previous text node; and in response to determining the node depth and the node depth between the current DOM tree node and the previous text node If the node distance is greater than the predetermined depth range and the predetermined distance, respectively, the content of the current DOM tree node is recorded, and the next DOM tree node is entered according to the depth-first search order. The storage unit is configured to store the content recorded after executing the depth-first search algorithm as a text information of the target webpage.

One of the beneficial effects of the present invention is that the automatic text information mining method and text information automatic mining system provided by the present invention can construct a web page source code into a DOM tree, and then perform a depth-first search method to search for target data according to depth and distance. There is a high probability that the content of a DOM tree node with textual information is retrieved.

On the other hand, the text information automatic mining method and text information automatic mining system provided by the present invention can judge whether the content of the target web page has data that meets the specific format conditions through a specific node type and a specific description tag, and then can directly This data is obtained to save the time and cost of manual data mining.

For a further understanding of the features and technical content of the present invention, please refer to the following detailed descriptions and drawings of the present invention. However, the drawings provided are only for reference and description, and are not intended to limit the present invention.

The following are specific specific examples to illustrate the implementation of the "text information automatic mining method and text information automatic mining system" disclosed in the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. . The present invention can be implemented or applied through other different specific embodiments, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the concept of the present invention. In addition, the drawings of the present invention are merely schematic illustrations, and are not drawn according to the actual size, and are stated in advance. The following embodiments will further describe the related technical contents of the present invention in detail, but the disclosed contents are not intended to limit the protection scope of the present invention. In addition, the term "or", as used herein, should include any one or a combination of more of the associated listed items, as the case may be.

FIG. 1 is a block diagram of an automatic text information mining system according to an embodiment of the present invention. Referring to FIG. 1 , an embodiment of the present invention provides an automatic text information determination and exploration system 1 , which includes a web page capture module 100 , a model tree construction module 102 , a first search module 104 and a storage unit 106 . The webpage capture module 100 is configured to obtain webpage source code data of a target webpage 2 .

The model tree construction module 102 is configured to parse the obtained web page source code data, obtain a plurality of document object model (Document Object Model, DOM) tree nodes according to the analysis results, and construct document objects according to the DOM tree nodes Model (Document Object Model, DOM) tree.

In detail, the DOM is not a part of the Hyper Text Markup Language (HTML) file, but the data created by the browser in the memory. The browser can use the DOM tree structure to build this model, because this model It is constructed from objects, so it is called an object model.

Generally speaking, a DOM tree can be composed of four kinds of nodes, including Document node, Element node, Attribute node and Text node. The document node is the top of the DOM tree, representing the entire web page, also known as the document object (Document object). The element node describes the structure of the HTML page, which is composed of HTML tags (Tag). The way to access the data of the DOM tree is to find an element, and after finding it, you can access the attribute or text of the element. The attribute node is the attribute placed in the opening tag of the HTML element, and the text node is the text between the opening and closing tags of the HTML element.

According to the above conditions, the model tree construction module 102 can analyze the source code information of the webpage to generate a plurality of DOM tree nodes, so as to construct the DOM tree. For example, please refer to FIG. 2 , which is a schematic diagram of a DOM tree according to an embodiment of the present invention.

Next, the first search module 104 is configured to perform a depth-first search algorithm for the DOM tree nodes of the DOM tree. In detail, a depth-first search method can be used for the algorithm that traverses the DOM tree described above. Start exploring from the root of the DOM tree (that is, the topmost file node DN in Figure 2), first explore the edge (Edge), for example, the connection between the file node DN and the first element node EN1 below can be called an edge) A DOM tree node that has not been searched, and search as deep as possible until all the edge nodes of the node have been explored, backtracking to the previous node, and repeatedly exploring the unsearched DOM tree node until it finds Destination node or traverse all nodes. In other words, the above order may be referred to as a depth-first search order, and the first search module 104 may proceed in the DOM tree according to this order, thereby searching for DOM tree nodes with a high probability of being associated with the context.

Please refer to FIG. 3 , which is a flowchart of a depth-first search algorithm according to an embodiment of the present invention. As shown, the first search module 104 performs the following steps for each of the DOM tree nodes:

Step S30: Proceed to the next DOM tree node according to the depth-first search order.

Step S31: Determine whether the current DOM tree node is a text node. If not, go back to step S30 to enter the next DOM tree node. If yes, go to step S32.

More specifically, in this step, determining whether the current DOM tree node is a text node is to determine whether the format of the current DOM tree node conforms to the text node defined in HTML.

Step S32: Determine whether the node depth between the current DOM tree node and the last DOM tree node judged to be a text node is greater than a predetermined depth, and whether the current DOM tree node and the previous DOM tree node judged to be a text node are not deep. Whether the distance between nodes is greater than the predetermined distance.

Specifically, the node depth is determined by the number of node levels between the DOM tree node and the previous text node, and the node distance is determined by the number of node travels between the DOM tree node and the previous text node. For example, as shown in FIG. 2, when proceeding from the leftmost attribute node AN1 to the attribute node AN2, the node depth between the two is 0. On the other hand, for the node distance, before reaching the attribute node AN1, the ancestor (Ancestor) node of the attribute node AN1 has already traveled, and there is no need to repeat it, so the attribute node AN1 directly travels to the attribute node AN2 The parent (Parent) node, and then to attribute node AN2. Therefore, according to the above process, the node distance between the attribute node AN1 and the attribute node AN2 is 2, and it can be deduced to all DOM tree nodes in this way.

In a preferred embodiment, the predetermined depth may be in the range of 3 to 7, the predetermined distance may be in the range of 8 to 12, and both the predetermined depth and the predetermined range are positive integers. In the preferred embodiment, the predetermined depth may be 5 and the predetermined range may be 12. Specifically, this mechanism corresponds to the writing logic of the news webpage. Therefore, according to this mechanism to assist the above-mentioned depth-first search algorithm to search, the text node corresponding to the main text information of the target webpage 2 can be obtained with a high probability.

In response to judging in step S32 that the node depth and node distance between the current DOM tree node and the previous text node are respectively greater than the predetermined depth and the predetermined distance, then enter step S33: record the content of the current DOM tree node, And go back to step S30, enter the next DOM tree node according to the depth-first search order. If in step S32, the node depth is not greater than the predetermined depth and/or the node distance is not greater than the predetermined distance, then go back to step S30.

Thereafter, the storage unit 106 may be configured to store the content recorded after executing the depth-first search algorithm as the text information of the target webpage.

In addition, as shown in FIG. 1 , the text information automatic mining system 1 may further include a checking module 110, which is configured to check whether the text recorded on a single DOM tree node exceeds a predetermined length, and if so, the storage unit 106 stores it. Checked text. In a specific embodiment, the checking module 110 may remove the blank (space) characters in the content to be recorded, and then calculate whether the length exceeds a predetermined length. In a preferred embodiment, the predetermined length is in the range of 30 to 80 characters, and in a preferred embodiment, the predetermined length is 50 characters of the text after removing the blank characters.

Please refer to FIG. 1 again, the text information automatic mining system 1 may further include a second search module 108, which is configured to determine whether the DOM tree nodes in the DOM tree have element nodes that also conform to the definition in HTML. A DOM tree node that matches a specific description tag. If so, directly obtain the content of this DOM tree node for recording. The storage unit 106 can store the content recorded by the second search module 108 as the text information of the target webpage 2 .

For example, a specific description tag is used to describe the type of material inserted into the material placed on the target web page 2 . For example, the specific description tag is the <style> tag, <script> tag or <iframe> tag defined under Golang's DataAtom. Taking the <iframe> tag as an example, it defines an iframe as a material to embed another HTML file in the current HTML file. The <script> tag is used to define user-side scripts, such as JavaScript. The <script> tag can either contain script statements or point to an external command file through the src attribute.

It should be noted that the difference in the structure of the webpage may cause the crawling rate of the first search module 104 and the second search module 108 to fluctuate when obtaining the text information of the target webpage 2 . However, in most cases, the first search module 104 and the second search module 108 can be complementary in crawl rate. In other words, the content respectively obtained by the first search module 104 and the second search module 108 can be integrated to serve as the final obtained text information of the target webpage 2 .

In view of the above, further reference may be made to FIG. 4 , which is a flowchart of an automatic text information mining method according to an embodiment of the present invention.

As shown in Figure 4, the automatic text information mining method may include the following steps:

Step S40: Configure the webpage capture module to obtain webpage source code data of the target webpage.

Step S41 : Configure a model tree construction module, parse the acquired webpage source code data, obtain a plurality of DOM tree nodes according to the analysis results, and construct a DOM tree according to the DOM tree nodes.

Step S42 : Configure a first search module to execute a depth-first search algorithm for the DOM tree nodes, so as to proceed in the DOM tree according to the depth-first search order, and obtain the context.

Step S43: Configure a second search module to determine whether there is a DOM tree node in the DOM tree that both conforms to the element node defined in HTML and conforms to a specific description tag, and if so, directly obtains the content of the DOM tree node for recording .

Step S44: Configure a storage unit to store the contents recorded by the first search module and the second search module as text information of the target webpage.

The textual information automatic mining method of the foregoing embodiment can be executed by the textual information automatic mining system 1 of the preceding embodiment, and the textual information automatic mining system 1 can be realized by a computer system (such as a desktop computer, a server, etc.), which, for example, has a central Electronic components such as processors, north-south bridges, volatile memory, storage units, and network chips.

The storage unit 106 can be, for example, a logical disk array such as a Redundant Array of Independent Disks (RAID) or a Just a Bunch Of Disks (JBOD) system. Alternatively, the storage unit 106 may also be a non-volatile storage device such as a hard disk drive (Hard Disk Drive, HDD).

The text information automatic determination and exploration system 1 , the webpage capture module 100 , the model tree construction module 102 , the first search module 104 , the second search module 108 and the inspection module 110 are stored in the storage unit 106 A computer program product that can be executed by a central processing unit to perform specific functions.

[Advantageous effects of the embodiment]

One of the beneficial effects of the present invention is that the automatic text information mining method and text information automatic mining system provided by the present invention can construct a web page source code into a DOM tree, and then perform a depth-first search method to search for target data according to depth and distance. There is a high probability that the content of a DOM tree node with textual information is retrieved.

On the other hand, the text information automatic mining method and text information automatic mining system provided by the present invention can judge whether the content of the target web page has data that meets the specific format conditions through a specific node type and a specific description tag, and then can directly This data is obtained to save the time and cost of manual data mining.

The contents disclosed above are only preferred feasible embodiments of the present invention, and are not intended to limit the scope of the present invention. Therefore, any equivalent technical changes made by using the contents of the description and drawings of the present invention are included in the application of the present invention. within the scope of the patent.

1: Text information automatic judgment and exploration system

100: Web Capture Module

102: Model Tree Building Blocks

104: The first search module

106: Storage unit

108: Second search module

110: Check the mod

2: Landing page

DN: file node

AN2: Attribute Node

AN1: Attribute Node

EN1: Element Node

FIG. 1 is a block diagram of an automatic text information mining system according to an embodiment of the present invention.

FIG. 2 is a schematic diagram of a document object model tree according to an embodiment of the present invention.

FIG. 3 is a flowchart of a depth-first search algorithm according to an embodiment of the present invention.

FIG. 4 is a flowchart of an automatic text information mining method according to an embodiment of the present invention.

1: Text information automatic judgment and exploration system

100: Web Capture Module

102: Model Tree Building Blocks

104: The first search module

106: Storage unit

108: Second search module

110: Check the mod

2: Landing page

Claims (16)

An automatic text information mining method, comprising: configuring a web page capture module to obtain web page source code data of a target web page; configuring a model tree construction module to parse the obtained web page source code data, and based on a The parsing result obtains a plurality of document object model (Document Object Model, DOM) tree nodes, and constructs a document object model (Document Object Model, DOM) tree according to the DOM tree nodes; configures a first search module for the DOM The DOM tree nodes of the tree execute a depth-first search algorithm to travel through the DOM tree according to a depth-first search order, and for each of the DOM tree nodes: determine whether the current DOM tree node is It is a text node. If not, enter the next DOM tree node. If so, judge whether the depth of a node between the current DOM tree node and the previous DOM tree node judged to be a text node is not. greater than a predetermined depth and whether a node distance is greater than a predetermined distance, wherein the node depth is determined by the number of node levels between the DOM tree node and the previous text node, and the node distance is determined by the DOM tree node and the previous text node. Determined by the number of node travels between a text node; and in response to judging that the node depth and the node distance between the current DOM tree node and the previous text node (text node) are greater than the predetermined depth range and the predetermined depth, respectively. distance, record the current content of the DOM tree node, and enter the next DOM tree node according to the depth-first search sequence; and configure a storage unit to store the recorded content after executing the depth-first search algorithm , as a text message for the landing page. The automatic text information mining method according to claim 1, wherein the current Whether the DOM tree node is a text node is to determine whether the current format of the DOM tree node conforms to the text node defined in the Hyper Text Markup Language (Hyper Text Markup Language, HTML). The automatic text information mining method according to claim 1, wherein the predetermined depth is in the range of 3 to 7, the predetermined distance is in the range of 8 to 12, and both the predetermined depth and the predetermined range are positive integers . The automatic text information mining method according to claim 1, further comprising configuring a checking module to check whether the text recorded on the single DOM tree node exceeds a predetermined length, and if so, configuring the storage unit to store the checked 's content. The automatic text information mining method according to claim 1, further comprising: configuring a second search module to determine whether the DOM tree nodes in the DOM tree have element nodes that also conform to the definition in HTML And match a specific description tag, if so, directly obtain the content of the DOM tree node for recording. The automatic text information mining method according to claim 5, further comprising configuring the storage unit to store the content recorded by the second search module as the text information of the target webpage. The automatic text information mining method according to claim 5, wherein the specific description tag is used to describe a material type of the material inserted and placed on the target web page. The automatic text information mining method according to claim 7, wherein the specific description tag is a <style> tag, a <script> tag or an <iframe> tag defined under DataAtom of the Golang programming language. An automatic text information exploration system, comprising: a webpage capture module configured to obtain webpage source code data of a target webpage; A model tree construction module configured to parse the obtained web page source code data, obtain a plurality of document object model (DOM) tree nodes according to a parsing result, and according to the DOM tree nodes constructing a Document Object Model (DOM) tree; a first search module configured to execute a depth-first search algorithm for the DOM tree nodes of the DOM tree to proceed according to a depth-first search order In the DOM tree, and for each of the DOM tree nodes: determine whether the current DOM tree node is a text node (text node), if not, enter the next DOM tree node, if so, then Determine whether a node depth between the current DOM tree node and the last DOM tree node judged to be a text node is greater than a predetermined depth and a node distance is greater than a predetermined distance, wherein the node depth is determined by the DOM The number of node levels between the tree node and the previous text node is determined, and the node distance is determined by the number of node travels between the DOM tree node and the previous text node; and in response to judging that the current DOM tree node and the If the node depth and the node distance between the previous text nodes are respectively greater than the predetermined depth range and the predetermined distance, the current content of the DOM tree node is recorded, and the depth-first search order is performed. Entering the next DOM tree node; and a storage unit configured to store the content recorded after executing the depth-first search algorithm as a text information of the target webpage. The automatic text information mining system according to claim 9, wherein judging whether the current DOM tree node is a text node is judging whether the format of the current DOM tree node conforms to the Hyper Text Markup Language (Hyper Text Markup Language) A literal node defined in Language, HTML). The text information automatic exploration system according to claim 9, wherein the predetermined depth is in the range of 3 to 7, the predetermined distance is in the range of 8 to 12, and both the predetermined depth and the predetermined range are positive integers . The automatic text information mining system according to claim 9, further comprising a checking module configured to check whether the text recorded on the single node of the DOM tree exceeds a predetermined length, and if so, the storage unit is configured to Save the checked body. The automatic text information mining system according to claim 9, further comprising a second search module configured to determine whether the DOM tree nodes in the DOM tree have element nodes that also conform to the HTML definition. ) and match a specific description tag, if so, directly obtain the content of the DOM tree node for recording. The automatic text information mining system of claim 13, wherein the storage unit is further configured to store the text recorded by the second search module as the text information of the target webpage. The automatic text information mining system of claim 13, wherein the specific description tag is used to describe a material type of the material inserted and placed on the target web page. The automatic text information mining system according to claim 15, wherein the specific description tag is a <style> tag, a <script> tag or an <iframe> tag defined under DataAtom of the Golang programming language.

Images