CN111444407A - Automatic extraction method and system for page list information of web crawler - Google Patents
Automatic extraction method and system for page list information of web crawler Download PDFInfo
- Publication number
- CN111444407A CN111444407A CN202010222132.4A CN202010222132A CN111444407A CN 111444407 A CN111444407 A CN 111444407A CN 202010222132 A CN202010222132 A CN 202010222132A CN 111444407 A CN111444407 A CN 111444407A
- Authority
- CN
- China
- Prior art keywords
- page
- document object
- extracted
- determining
- markup language
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000605 extraction Methods 0.000 title claims abstract description 40
- 238000000034 method Methods 0.000 claims abstract description 24
- 238000005516 engineering process Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 2
- 101100171060 Caenorhabditis elegans div-1 gene Proteins 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/957—Browsing optimisation, e.g. caching or content distillation
- G06F16/9577—Optimising the visualization of content, e.g. distillation of HTML documents
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to a method and a system for automatically extracting page list information of a web crawler. The method comprises the following steps: acquiring a hypertext markup language document of a page to be extracted; determining a set of hypertext markup language objects according to elements in the hypertext markup language document; traversing the hypertext markup language object set to determine a document object model; determining a webpage structure of the page to be extracted according to the document object model; the webpage structure comprises list items and list item attributes; determining an extraction mode according to the webpage structure of the page to be extracted; and extracting the page list information to be extracted by utilizing the extraction mode. The method and the system for automatically extracting the web crawler page list information provided by the invention realize the automatic capture of the web crawler page list information.
Description
Technical Field
The invention relates to the field of web crawlers, in particular to a method and a system for automatically extracting page list information of a web crawler.
Background
With the rapid development of information technology, a large amount of data information of the features is brought. How to extract useful information from massive information quickly, efficiently and accurately puts higher demands on the information acquisition technology of the new era. Under the background, the web crawler technology is rapidly developed and widely applied due to the characteristics of convenience in information acquisition, various acquisition modes, semi-automation of information acquisition and the like. However, in the conventional web crawler technology, a crawler script needs to be written according to the characteristics of a web page, the internet is full of massive information publishing platforms, systems and websites, and the way and format of displaying information on each page are varied, which causes the development cost of the web crawler technology to be high, and after the web page is changed, the crawler program needs to be updated pertinently, and the capturing stability is also influenced. Moreover, automatic page grabbing can be realized only through manual intervention. Therefore, the prior art cannot realize automatic capture of web crawler page list information.
Disclosure of Invention
The invention aims to provide a method and a system for automatically extracting web crawler page list information, which are used for automatically capturing the web crawler page list information.
In order to achieve the purpose, the invention provides the following scheme:
a method for automatically extracting page list information of a web crawler comprises the following steps:
acquiring a hypertext markup language document of a page to be extracted;
determining a set of hypertext markup language objects according to elements in the hypertext markup language document; the elements include tags, attributes, and text of a hypertext markup language document;
traversing the hypertext markup language object set to determine a document object model;
determining a webpage structure of the page to be extracted according to the document object model; the webpage structure comprises list items and list item attributes;
determining an extraction mode according to the webpage structure of the page to be extracted;
and extracting the page list information to be extracted by utilizing the extraction mode.
Optionally, the acquiring a hypertext markup language document of a page to be extracted further includes:
judging whether the page to be extracted is reprinted or not to obtain a first judgment result;
if the first judgment result shows that the page to be extracted is changed, executing the step of acquiring the hypertext markup language document of the page to be extracted;
and if the first judgment result shows that the page to be extracted is not reprinted, extracting the page list information to be extracted directly according to the corresponding extraction mode when the page is not reprinted.
Optionally, the traversing the hypertext markup language object set to determine a document object model specifically includes:
traversing the hypertext markup language object set to determine the relationship among all objects; all the relationships among the objects comprise the height and the depth of the peer relationship, the inclusion relationship, the parent relationship, the child relationship and the hierarchical relationship among the objects;
and determining the document object model according to the relation among all the objects.
Optionally, the determining, according to the document object model, a web page structure of the page to be extracted specifically includes:
determining the number of the tags of each attribute and the number of the tags of all the attributes in the page to be extracted according to the document object model;
determining the proportion of the occurrence times of each label of each attribute in the page to be extracted to obtain a single proportion set;
determining the proportion of the label attribute combination in the page to be extracted to obtain a combination proportion set;
arranging the single proportion set and the combined proportion set in a descending order according to the frequency of the tag attributes to obtain list items; the attribute of the list item is a label attribute with the same appearance proportion of a plurality of single label attributes as the appearance proportion of the combined label attribute in the combined proportion set;
determining all document object chains in the page to be extracted according to the document object model;
intercepting each document object chain by taking the tag attribute combination corresponding to the list item as the starting point;
calculating the frequency of occurrence of each intercepted document object chain;
and performing descending order arrangement on the frequency of the occurrence of each intercepted document object chain, and determining the attribute of the list item.
Optionally, the intercepting each document object chain starting with the tag attribute combination corresponding to the list item further includes:
judging whether the document object chain contains a tag attribute combination corresponding to the list item to obtain a second judgment result;
if the second judgment result indicates that the document object chain contains the tag attribute combination corresponding to the list item, the document object chain is reserved, and each document object chain is intercepted by taking the tag attribute combination corresponding to the list item as an initial point;
and if the second judgment result shows that the document object chain does not contain the tag attribute combination corresponding to the list item, rejecting the document object chain.
An automatic extraction system for web crawler page list information, comprising:
the hypertext markup language document acquisition module is used for acquiring a hypertext markup language document of a page to be extracted;
a hypertext markup language object set determining module, configured to determine a hypertext markup language object set according to elements in the hypertext markup language document; the elements include tags, attributes, and text of a hypertext markup language document;
the document object model determining module is used for traversing the hypertext markup language object set and determining a document object model;
the webpage structure determining module is used for determining the webpage structure of the page to be extracted according to the document object model; the webpage structure comprises list items and list item attributes;
the extraction mode determining module is used for determining an extraction mode according to the webpage structure of the page to be extracted;
and the page list information extraction module is used for extracting the page list information to be extracted by utilizing the extraction mode.
Optionally, the method further includes:
the first judgment module is used for judging whether the page to be extracted is reprinted or not to obtain a first judgment result;
the execution module is used for executing the step of acquiring the hypertext markup language document of the page to be extracted if the first judgment result shows that the page to be extracted is changed;
and the extraction module is used for extracting the page list information to be extracted directly according to the corresponding extraction mode when the version is not changed if the first judgment result shows that the page to be extracted is not changed.
Optionally, the document object model determining module specifically includes:
the relation determining unit among all the objects is used for traversing the hypertext markup language object set and determining the relation among all the objects; all the relationships among the objects comprise the height and the depth of the peer relationship, the inclusion relationship, the parent relationship, the child relationship and the hierarchical relationship among the objects;
and the document object model determining unit is used for determining the document object model according to the relation among all the objects.
Optionally, the web page structure determining module specifically includes:
the label attribute determining unit is used for determining the number of labels of each attribute and the number of labels of all attributes in the page to be extracted according to the document object model;
the single proportion set determining unit is used for determining the proportion of the occurrence times of each label of each attribute in the page to be extracted to obtain a single proportion set;
a combined proportion set determining unit, configured to determine a proportion of a tag attribute combination in the page to be extracted, so as to obtain a combined proportion set;
the list item determining unit is used for performing descending order arrangement on the single proportion set and the combined proportion set according to the frequency of the tag attributes to obtain list items; the attribute of the list item is a label attribute with the same appearance proportion of a plurality of single label attributes as the appearance proportion of the combined label attribute in the combined proportion set;
the document object chain determining unit is used for determining all document object chains in the page to be extracted according to the document object model;
a document object chain intercepting unit, configured to intercept each document object chain starting with a tag attribute combination corresponding to the list item;
the frequency calculating unit is used for calculating the frequency of the occurrence of each intercepted document object chain;
and the list item attribute determining unit is used for performing descending order arrangement on the frequency of the document object chain after each interception to determine the list item attribute.
Optionally, the web page structure determining module further includes:
the first judging unit is used for judging whether the document object chain contains a tag attribute combination corresponding to the list item to obtain a second judging result;
a document object chain retaining unit, configured to retain the document object chain if the second determination result indicates that the document object chain contains the tag attribute combination corresponding to the list item, and intercept each of the document object chains starting with the tag attribute combination corresponding to the list item;
and the document object chain removing unit is used for removing the document object chain if the second judgment result shows that the document object chain does not contain the tag attribute combination corresponding to the list item.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
according to the method and the system for automatically extracting the web crawler page list information, provided by the invention, the hypertext markup language document of the page to be extracted is obtained, the hypertext markup language object set is determined, the document object set is determined according to the hypertext markup language object set, and then the web page structure of the page to be extracted is determined through the document object set, namely, the web page structure to be extracted is automatically identified, so that the problem that the information display mode and the format of each page are changeable is solved, manual intervention is avoided, automatic capture of the web crawler page list information is realized, and the stability of a web crawler is ensured.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
FIG. 1 is a schematic flow chart of a method for automatically extracting page list information of a web crawler according to the present invention;
fig. 2 is a schematic structural diagram of an automatic web crawler page list information extraction system according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention aims to provide a method and a system for automatically extracting web crawler page list information, which are used for automatically capturing the web crawler page list information.
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
Fig. 1 is a schematic flow chart of a method for automatically extracting page list information of a web crawler according to the present invention, and as shown in fig. 1, the method for automatically extracting page list information of a web crawler according to the present invention includes:
s101, obtaining a hypertext markup language document of a page to be extracted.
Before S101, the method further includes:
and judging whether the page to be extracted is reprinted or not to obtain a first judgment result.
And if the first judgment result shows that the page to be extracted is changed, executing the step of acquiring the hypertext markup language document of the page to be extracted.
And if the first judgment result shows that the page to be extracted is not reprinted, extracting the page list information to be extracted directly according to the corresponding extraction mode when the page is not reprinted.
S102, determining a set of hypertext markup language objects according to elements in the hypertext markup language document, wherein the elements comprise tags, attributes and texts of the hypertext markup language document, the elements in the hypertext markup language document are all regarded as one object, and the determined set of hypertext markup language objects is HTM L L ═ Obj1,Obj2,Obj3,...,Objn}。
S103, traversing the hypertext markup language object set and determining a document object model.
Traversing the hypertext markup language object set to determine the relationship among all objects; the relationships among all the objects comprise the peer relationships among the objects, the inclusion relationships, the parent relationships, the child relationships and the height and the depth of the hierarchical relationship. Height is the length of the longest path of a leaf and depth is the length of the path to its root.
And determining the document object model Dom according to the relation among all the objects.
The method comprises the steps of S104, determining a webpage structure of a page to be extracted according to a document object model, wherein the webpage structure comprises list items and list item attributes, and according to the characteristics of the document object model, the number of all elements and attributes and the incidence relation among the elements and the attributes can be rapidly calculated, after statistical data and the incidence relation are obtained, the statistical data and the incidence relation are analyzed, contents such as the list items, the list item attributes and irrelevant information are extracted from the document structure, and the irrelevant information is removed.
And determining the number of the tags of each attribute and the number of the tags of all the attributes in the page to be extracted according to the document object model. Computing all tags, attributes x within a pageiThe amount of (a) is mi; the total number of all labels and attributes in the page is calculated as w,
and determining the proportion of the occurrence times of each label of each attribute in the page to be extracted to obtain a single proportion set. I.e. calculating the proportion p of the number of occurrences of a single label and attribute in the whole pageiForm a single specific gravity set p, p ═ p1,p2,p3,p4,...pw}，
And determining the proportion of the label attribute combination in the page to be extracted to obtain a combination proportion set k. Sometimes the property within a page is more than one composition, a table, by combination of different elementsTo different structural content, k ═ p (b1,2,...),p(1,3,...),p(2,3,...),...,p(i,j,...) Where i, j represent different combinations of label attributes, and i, j are all combinations present within the page.
n(i,j,...) Refers to the number of occurrences of multiple tag attribute combinations in a page.
Arranging the single proportion set and the combined proportion set in a descending order according to the frequency of the tag attributes to obtain list items; the attribute of the list item is a label attribute with the same appearance proportion of a plurality of single label attributes as the appearance proportion of the combined label attribute in the combined proportion set. I.e. k (i,j,...)＝pi＝pj＝...。
And determining all document object chains, namely DOM chains, in the page to be extracted according to the document object model. Where the hierarchical DOM chain is denoted by M and the side-by-side DOM chain is denoted by N.
And intercepting each document object chain by taking the tag attribute combination corresponding to the list item as the start.
And calculating the occurrence frequency of the document object chain after each interception.
And performing descending order arrangement on the frequency of the occurrence of each intercepted document object chain, and determining the attribute of the list item.
S105, determining an extraction mode according to the webpage structure of the page to be extracted;
and S106, extracting the page list information to be extracted by utilizing the extraction mode.
Intercepting each document object chain by taking the tag attribute combination corresponding to the list item as a start, wherein the method further comprises the following steps:
and judging whether the document object chain contains a tag attribute combination corresponding to the list item to obtain a second judgment result.
If the second judgment result shows that the document object chain contains the tag attribute combination corresponding to the list item, the document object chain is reserved, and each document object chain is intercepted by taking the tag attribute combination corresponding to the list item as an initial point.
And if the second judgment result shows that the document object chain does not contain the tag attribute combination corresponding to the list item, rejecting the document object chain.
In a specific embodiment, HM L ═ html, head, body, h3, div, p, a, span }, traverses the HTM L, records the Height (Height) and Depth (Depth) of the interrelation (sibling, including, parent, child, etc.) and hierarchical relationship among all objects, the Height is the length of the longest path of the leaf, and the Depth is the length of the path to its root.
1) Calculating the number of all tags in the page to be 25 (since the tags in the head do not participate in data extraction, useless tags in the head are removed, and the tags do not participate in calculation), and calculating the number of the attributes to be 11;
2) calculating the total number of all labels and attributes in the page to be 25+ 11-36;
3) calculating the proportion p of the occurrence times of the single label and the attribute in the whole page; as shown in table 1:
TABLE 1
| Label/Property | p | Label/Property | p |
| html | 1/36 | a | 5/36 |
| head | 1/36 | span | 10/36 |
| body | 1/36 | main | 5/36 |
| div | 1/36 | list | 5/36 |
| p | 5/36 | content | 1/36 |
In the above table, html, head, body, div, p, a, and span are labels, and main, list, and content are attributes.
4) And calculating the proportion of the label attribute combination in the page. Sometimes the attributes in a page are more than one composition, and different structural contents are expressed through the combination of different elements. The combined tag attribute set is denoted as k. The p-tag in this example has two class attributes, main and list;
k＝{pmain,list}
5) and (3) arranging the sets p and k in a descending order according to the occurrence frequency of the label attributes, acquiring the label attributes with the same occurrence ratio of the single element set and the combined element set and the same occurrence ratio of a plurality of single elements and the combined label attributes in the combined element set, and finding out through comparison:
p(mian,list)＝pmain＝plist
then the tags P with attributes main and list are list items, and the tags html, head, body, div and attribute content are irrelevant tags and irrelevant attributes.
② decision algorithm for list item attribute
Through analysis, it is found that in this example, the list item and the list item attribute are in a containment relationship, and are in a hierarchical form M, and a parallel form N exists between the list item attributes, then the process of determining the list item attribute is as follows:
1) all DOM chains within the page are listed, with the hierarchical DOM chain represented by M and the side-by-side DOM chain represented by N. In this example, only one layer contains the relationship, i.e., p contains a, span.
M＝{P→a|span|span}
2) And intercepting the DOM chains by taking the attribute (combination) of the list item tag as an initial point, directly abandoning the DOM chains without the attribute of the list item tag, calculating the frequency of occurrence of each DOM chain, and constructing a set R as a result. In this example, there is only one chain { P → a | span | span }, which occurs 5 times, then:
3) and arranging DOM chains in a descending order according to the frequency, recording the DOM chains as R', and determining the attribute of the list item to be selected with higher frequency. And then according to the webpage rule obtained above, the span can be judged as the attribute of the list item.
As a result: the list item is P, and the attributes of the list item are main and list; the list item content is an a label, and the list item attribute is a span label.
Fig. 2 is a schematic structural diagram of an automatic extraction system for web crawler page list information provided by the present invention, and as shown in fig. 2, the automatic extraction system for web crawler page list information provided by the present invention includes: a hypertext markup language document acquisition module 201, a hypertext markup language object set determination module 202, a document object model determination module 203, a web page structure determination module 204, an extraction mode determination module 205, and a page list information extraction module 206.
The hypertext markup language document acquisition module 201 is configured to acquire a hypertext markup language document of a page to be extracted.
The hypertext markup language object set determining module 202 is configured to determine a hypertext markup language object set according to elements in the hypertext markup language document; the elements include tags, attributes, and text of a hypertext markup language document.
The document object model determining module 203 is configured to traverse the set of hypertext markup language objects to determine a document object model.
The web page structure determining module 204 is configured to determine a web page structure of the page to be extracted according to the document object model; the web page structure includes list items and list item attributes.
The extraction mode determining module 205 is configured to determine an extraction mode according to a web page structure of the page to be extracted.
The page list information extraction module 206 is configured to extract the page list information to be extracted by using the extraction mode.
The system for automatically extracting the page list information of the web crawler provided by the invention further comprises: the device comprises a first judgment module, an execution module and an extraction module corresponding to the non-version-change time.
The first judgment module is used for judging whether the page to be extracted is reprinted or not to obtain a first judgment result.
And the execution module is used for executing the step of acquiring the hypertext markup language document of the page to be extracted if the first judgment result shows that the page to be extracted is changed.
And the extraction module according to the extraction mode corresponding to the non-version-change condition is used for directly extracting the page list information to be extracted according to the extraction mode corresponding to the non-version-change condition if the first judgment result shows that the page to be extracted is not version-changed.
The document object model determining module 203 specifically includes: a relation determining unit between all the objects and a document object model determining unit.
The relation determining unit among all the objects is used for traversing the hypertext markup language object set and determining the relation among all the objects; the relationships among all the objects comprise the peer relationships among the objects, the inclusion relationships, the parent relationships, the child relationships and the height and the depth of the hierarchical relationship.
The document object model determining unit is used for determining the document object model according to the relation among all the objects.
The web page structure determining module 204 specifically includes: the system comprises a tag attribute determining unit, a single proportion set determining unit, a combined proportion set determining unit, a list item determining unit, a document object chain intercepting unit, a frequency calculating unit and a list item attribute determining unit.
The label attribute determining unit is used for determining the number of labels of each attribute and the number of labels of all attributes in the page to be extracted according to the document object model.
The single proportion set determining unit is used for determining the proportion of the occurrence times of each label of each attribute in the page to be extracted to obtain a single proportion set.
And the combined proportion set determining unit is used for determining the proportion of the label attribute combination in the page to be extracted to obtain a combined proportion set.
The list item determining unit is used for performing descending order arrangement on the single proportion set and the combined proportion set according to the frequency of the tag attributes to obtain list items; the attribute of the list item is a label attribute with the same appearance proportion of a plurality of single label attributes as the appearance proportion of the combined label attribute in the combined proportion set.
And the document object chain determining unit is used for determining all document object chains in the page to be extracted according to the document object model.
And the document object chain intercepting unit is used for intercepting each document object chain by taking the tag attribute combination corresponding to the list item as the starting point.
The frequency calculating unit is used for calculating the frequency of the document object chain after each interception.
And the list item attribute determining unit is used for performing descending order arrangement on the frequency of the document object chain after each interception to determine the list item attribute.
The web page structure determining module 204 further comprises: the device comprises a first judging unit, a document object chain retaining unit and a document object chain removing unit.
The first judging unit is used for judging whether the document object chain contains a tag attribute combination corresponding to the list item to obtain a second judging result.
And the document object chain retaining unit is used for retaining the document object chain if the second judgment result indicates that the document object chain contains the tag attribute combination corresponding to the list item, and intercepting each document object chain by taking the tag attribute combination corresponding to the list item as a start.
And the document object chain removing unit is used for removing the document object chain if the second judgment result shows that the document object chain does not contain the tag attribute combination corresponding to the list item.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the system disclosed by the embodiment, the description is relatively simple because the system corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the method part for description.
The principles and embodiments of the present invention have been described herein using specific examples, which are provided only to help understand the method and the core concept of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed. In view of the above, the present disclosure should not be construed as limiting the invention.
Claims (10)
1. A method for automatically extracting page list information of a web crawler is characterized by comprising the following steps:
acquiring a hypertext markup language document of a page to be extracted;
determining a set of hypertext markup language objects according to elements in the hypertext markup language document; the elements include tags, attributes, and text of a hypertext markup language document;
traversing the hypertext markup language object set to determine a document object model;
determining a webpage structure of the page to be extracted according to the document object model; the webpage structure comprises list items and list item attributes;
determining an extraction mode according to the webpage structure of the page to be extracted;
and extracting the page list information to be extracted by utilizing the extraction mode.
2. The method according to claim 1, wherein the obtaining of the hypertext markup language document of the page to be extracted further comprises:
judging whether the page to be extracted is reprinted or not to obtain a first judgment result;
if the first judgment result shows that the page to be extracted is changed, executing the step of acquiring the hypertext markup language document of the page to be extracted;
and if the first judgment result shows that the page to be extracted is not reprinted, extracting the page list information to be extracted directly according to the corresponding extraction mode when the page is not reprinted.
3. The method according to claim 1, wherein the traversing the set of html objects to determine a document object model specifically includes:
traversing the hypertext markup language object set to determine the relationship among all objects; all the relationships among the objects comprise the height and the depth of the peer relationship, the inclusion relationship, the parent relationship, the child relationship and the hierarchical relationship among the objects;
and determining the document object model according to the relation among all the objects.
4. The method according to claim 1, wherein the determining a web page structure of the page to be extracted according to the document object model specifically includes:
determining the number of the tags of each attribute and the number of the tags of all the attributes in the page to be extracted according to the document object model;
determining the proportion of the occurrence times of each label of each attribute in the page to be extracted to obtain a single proportion set;
determining the proportion of the label attribute combination in the page to be extracted to obtain a combination proportion set;
arranging the single proportion set and the combined proportion set in a descending order according to the frequency of the tag attributes to obtain list items; the attribute of the list item is a label attribute with the same appearance proportion of a plurality of single label attributes as the appearance proportion of the combined label attribute in the combined proportion set;
determining all document object chains in the page to be extracted according to the document object model;
intercepting each document object chain by taking the tag attribute combination corresponding to the list item as the starting point;
calculating the frequency of occurrence of each intercepted document object chain;
and performing descending order arrangement on the frequency of the occurrence of each intercepted document object chain, and determining the attribute of the list item.
5. The method according to claim 4, wherein the intercepting each document object chain starting with the tag attribute combination corresponding to the list item further comprises:
judging whether the document object chain contains a tag attribute combination corresponding to the list item to obtain a second judgment result;
if the second judgment result indicates that the document object chain contains the tag attribute combination corresponding to the list item, the document object chain is reserved, and each document object chain is intercepted by taking the tag attribute combination corresponding to the list item as an initial point;
and if the second judgment result shows that the document object chain does not contain the tag attribute combination corresponding to the list item, rejecting the document object chain.
6. An automatic extraction system for web crawler page list information, comprising:
the hypertext markup language document acquisition module is used for acquiring a hypertext markup language document of a page to be extracted;
a hypertext markup language object set determining module, configured to determine a hypertext markup language object set according to elements in the hypertext markup language document; the elements include tags, attributes, and text of a hypertext markup language document;
the document object model determining module is used for traversing the hypertext markup language object set and determining a document object model;
the webpage structure determining module is used for determining the webpage structure of the page to be extracted according to the document object model; the webpage structure comprises list items and list item attributes;
the extraction mode determining module is used for determining an extraction mode according to the webpage structure of the page to be extracted;
and the page list information extraction module is used for extracting the page list information to be extracted by utilizing the extraction mode.
7. The system according to claim 6, further comprising:
the first judgment module is used for judging whether the page to be extracted is reprinted or not to obtain a first judgment result;
the execution module is used for executing the step of acquiring the hypertext markup language document of the page to be extracted if the first judgment result shows that the page to be extracted is changed;
and the extraction module is used for extracting the page list information to be extracted directly according to the corresponding extraction mode when the version is not changed if the first judgment result shows that the page to be extracted is not changed.
8. The system according to claim 6, wherein the document object model determining module specifically comprises:
the relation determining unit among all the objects is used for traversing the hypertext markup language object set and determining the relation among all the objects; all the relationships among the objects comprise the height and the depth of the peer relationship, the inclusion relationship, the parent relationship, the child relationship and the hierarchical relationship among the objects;
and the document object model determining unit is used for determining the document object model according to the relation among all the objects.
9. The system according to claim 6, wherein the web page structure determining module specifically comprises:
the label attribute determining unit is used for determining the number of labels of each attribute and the number of labels of all attributes in the page to be extracted according to the document object model;
the single proportion set determining unit is used for determining the proportion of the occurrence times of each label of each attribute in the page to be extracted to obtain a single proportion set;
a combined proportion set determining unit, configured to determine a proportion of a tag attribute combination in the page to be extracted, so as to obtain a combined proportion set;
the list item determining unit is used for performing descending order arrangement on the single proportion set and the combined proportion set according to the frequency of the tag attributes to obtain list items; the attribute of the list item is a label attribute with the same appearance proportion of a plurality of single label attributes as the appearance proportion of the combined label attribute in the combined proportion set;
the document object chain determining unit is used for determining all document object chains in the page to be extracted according to the document object model;
a document object chain intercepting unit, configured to intercept each document object chain starting with a tag attribute combination corresponding to the list item;
the frequency calculating unit is used for calculating the frequency of the occurrence of each intercepted document object chain;
and the list item attribute determining unit is used for performing descending order arrangement on the frequency of the document object chain after each interception to determine the list item attribute.
10. The system according to claim 9, wherein the web page structure determining module further comprises:
the first judging unit is used for judging whether the document object chain contains a tag attribute combination corresponding to the list item to obtain a second judging result;
a document object chain retaining unit, configured to retain the document object chain if the second determination result indicates that the document object chain contains the tag attribute combination corresponding to the list item, and intercept each of the document object chains starting with the tag attribute combination corresponding to the list item;
and the document object chain removing unit is used for removing the document object chain if the second judgment result shows that the document object chain does not contain the tag attribute combination corresponding to the list item.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010222132.4A CN111444407B (en) | 2020-03-26 | 2020-03-26 | Automatic extraction method and system for page list information of web crawlers |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010222132.4A CN111444407B (en) | 2020-03-26 | 2020-03-26 | Automatic extraction method and system for page list information of web crawlers |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111444407A true CN111444407A (en) | 2020-07-24 |
| CN111444407B CN111444407B (en) | 2023-05-16 |
Family
ID=71648987
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010222132.4A Active CN111444407B (en) | 2020-03-26 | 2020-03-26 | Automatic extraction method and system for page list information of web crawlers |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111444407B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101630327A (en) * | 2009-08-14 | 2010-01-20 | 昆明理工大学 | Design method of theme network crawler system |
| US20120317089A1 (en) * | 2003-07-03 | 2012-12-13 | Randall Keith H | Scheduler for Search Engine Crawler |
| US20160125087A1 (en) * | 2014-10-31 | 2016-05-05 | Informite Inc. | Systems and methods for semantic keyword analysis |
| CN106202108A (en) * | 2015-05-06 | 2016-12-07 | 阿里巴巴集团控股有限公司 | Web crawlers captures method for allocating tasks and device and data grab method and device |
| EP3107009A1 (en) * | 2015-06-19 | 2016-12-21 | Tata Consultancy Services Limited | Self-learning based crawling and rule-based data mining for automatic information extraction |
| CN107943838A (en) * | 2017-10-30 | 2018-04-20 | 北京大数元科技发展有限公司 | A kind of automatic method and system for obtaining xpath generation reptile scripts |
| WO2018103488A1 (en) * | 2016-12-08 | 2018-06-14 | 腾讯科技（深圳）有限公司 | Webpage scraping method and server |
| CN110309386A (en) * | 2018-02-28 | 2019-10-08 | 腾讯科技（深圳）有限公司 | A kind of method and apparatus of web page crawl |
-
2020
- 2020-03-26 CN CN202010222132.4A patent/CN111444407B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120317089A1 (en) * | 2003-07-03 | 2012-12-13 | Randall Keith H | Scheduler for Search Engine Crawler |
| CN101630327A (en) * | 2009-08-14 | 2010-01-20 | 昆明理工大学 | Design method of theme network crawler system |
| US20160125087A1 (en) * | 2014-10-31 | 2016-05-05 | Informite Inc. | Systems and methods for semantic keyword analysis |
| CN106202108A (en) * | 2015-05-06 | 2016-12-07 | 阿里巴巴集团控股有限公司 | Web crawlers captures method for allocating tasks and device and data grab method and device |
| EP3107009A1 (en) * | 2015-06-19 | 2016-12-21 | Tata Consultancy Services Limited | Self-learning based crawling and rule-based data mining for automatic information extraction |
| WO2018103488A1 (en) * | 2016-12-08 | 2018-06-14 | 腾讯科技（深圳）有限公司 | Webpage scraping method and server |
| CN107943838A (en) * | 2017-10-30 | 2018-04-20 | 北京大数元科技发展有限公司 | A kind of automatic method and system for obtaining xpath generation reptile scripts |
| CN110309386A (en) * | 2018-02-28 | 2019-10-08 | 腾讯科技（深圳）有限公司 | A kind of method and apparatus of web page crawl |
Non-Patent Citations (2)
| Title |
|---|
| 金涛: ""网络爬虫在网页信息提取中的应用研究"" * |
| 陆妍玲等: ""扩展流数据立方体的GIS时空大数据组织方法"" * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111444407B (en) | 2023-05-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Sun et al. | Dom based content extraction via text density | |
| CN102253979B (en) | Vision-based web page extracting method | |
| CN102156737B (en) | Method for extracting subject content of Chinese webpage | |
| CN102663023B (en) | Implementation method for extracting web content | |
| CN102662969B (en) | Internet information object positioning method based on webpage structure semantic meaning | |
| US9449114B2 (en) | Removing non-substantive content from a web page by removing its text-sparse nodes and removing high-frequency sentences of its text-dense nodes using sentence hash value frequency across a web page collection | |
| CN102270206A (en) | Method and device for capturing valid web page contents | |
| CN102915361B (en) | Webpage text extracting method based on character distribution characteristic | |
| KR101780377B1 (en) | Method of automatically extracting food safety event in real time from news and social networking service data | |
| Insa Cabrera et al. | Using the words/leafs ratio in the DOM tree for content extraction | |
| CN104572934B (en) | A kind of webpage key content abstracting method based on DOM | |
| CN103699591A (en) | Page body extraction method based on sample page | |
| Qureshi et al. | Hybrid model of content extraction | |
| Yu et al. | Web content information extraction based on DOM tree and statistical information | |
| CN113849718A (en) | Internet tobacco science and technology information automatic acquisition device, method and storage medium | |
| CN108459936B (en) | Accurate statistical method and device based on content modularization | |
| CN111444407A (en) | Automatic extraction method and system for page list information of web crawler | |
| CN108255895A (en) | A kind of web data acquisition methods using context environmental rule | |
| CN112632421B (en) | Self-adaptive structured document extraction method | |
| Azad et al. | Removal of noisy information in web pages | |
| YesuRaju et al. | A language independent web data extraction using vision based page segmentation algorithm | |
| CN102567463A (en) | Biography automatic extraction method based on Internet | |
| CN112818279A (en) | Method and device for determining similarity of web pages and computer-readable storage medium | |
| Jiang et al. | Noise reduction of web pages via feature analysis | |
| Thanadechteemapat et al. | Automatic web content extraction for generating tag clouds from thai web sites |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20200724 Assignee: Guilin Yidoumi Information Technology Co.,Ltd. Assignor: GUILIN University OF TECHNOLOGY Contract record no.: X2023980044286 Denomination of invention: A Method and System for Automatically Extracting Page List Information from a Web Crawler Granted publication date: 20230516 License type: Common License Record date: 20231027 Application publication date: 20200724 Assignee: Guangxi Biaoyu Technology Development Co.,Ltd. Assignor: GUILIN University OF TECHNOLOGY Contract record no.: X2023980044270 Denomination of invention: A Method and System for Automatically Extracting Page List Information from a Web Crawler Granted publication date: 20230516 License type: Common License Record date: 20231026 |
|
| EE01 | Entry into force of recordation of patent licensing contract |