CN108733405A - The method and apparatus that training webpage distribution indicates model - Google Patents
The method and apparatus that training webpage distribution indicates model Download PDFInfo
- Publication number
- CN108733405A CN108733405A CN201710239759.9A CN201710239759A CN108733405A CN 108733405 A CN108733405 A CN 108733405A CN 201710239759 A CN201710239759 A CN 201710239759A CN 108733405 A CN108733405 A CN 108733405A
- Authority
- CN
- China
- Prior art keywords
- webpage
- node
- model
- tree structure
- dom tree
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/75—Structural analysis for program understanding
- G06F8/751—Code clone detection
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
A kind of method and apparatus that trained webpage distribution indicates model are disclosed, wherein this method includes：Generate DOM Document Object Model (DOM) tree construction of each webpage in multiple webpages；For the DOM tree structure of each webpage, the sequence node of the predetermined length of predetermined number is extracted, wherein the extraction of each sequence node includes：Randomly choose one of breadth first traversal mode and depth-first traversal mode；And a node is randomly selected from DOM tree structure, and using one node as start node, sequence node is extracted from DOM tree structure in a manner of selected traversal；And the webpage distribution is trained to indicate that model, webpage distribution indicate that model is used to generate the expression vector of input webpage based on the sequence node extracted.In accordance with an embodiment of the present disclosure, the text message and structural information of webpage can be merged.
Description
Technical field
This disclosure relates to machine learning field and character representation field.More specifically, this disclosure relates to one kind can merge
The text message of webpage and the training webpage distribution of structural information indicate the method and apparatus of model and generate point of webpage
The method and apparatus that cloth indicates.
Background technology
Webpage similarity calculation can compare the similarity between different web pages.Webpage similarity calculation in the prior art
It usually only calculates the content deltas between different web pages or only calculates the architectural difference between different web pages, and cannot merge
Text message in webpage and structural information.
Invention content
The brief overview about the disclosure is given below, in order to provide the basic of some aspects about the disclosure
Understand.It is understood, however, that this general introduction is not the exhaustive general introduction about the disclosure.It is not intended to for determining
The critical component or pith of the disclosure, nor being intended to limit the scope of the present disclosure.Its purpose is only with letter
The form of change provides certain concepts about the disclosure, in this, as preamble in greater detail given later.
In view of problem above, purpose of this disclosure is to provide a kind of combination depth-first traversal and breadth first traversal with
The training webpage distribution of machine sampling algorithm, the text message to merge webpage and structural information indicates the method and dress of model
Set and generate the distributed method and apparatus indicated of webpage.
According to the one side of the disclosure, a kind of method that trained webpage distribution indicates model is provided, including：It can give birth to
At DOM Document Object Model (DOM) tree construction of each webpage in multiple webpages；It can be directed to the DOM tree structure of each webpage,
The sequence node for extracting the predetermined length of predetermined number, wherein the extraction of each sequence node includes：Range can be randomly choosed
One of first traversal mode and depth-first traversal mode；And a node can be randomly selected from DOM tree structure, and
Using a node as start node, sequence node is extracted from DOM tree structure in a manner of selected traversal；And it can be with base
The webpage distribution is trained to indicate that model, webpage distribution indicate model for generating input net in the sequence node extracted
The expression vector of page.
According to another aspect of the present disclosure, a kind of device of trained webpage distribution expression model is additionally provided, including：Text
Shelves object model generation unit, may be configured to generate the DOM tree structure of each webpage in multiple webpages；Extract node sequence
Column unit may be configured to the DOM tree structure for each webpage, extract the sequence node of the predetermined length of predetermined number,
The extraction of wherein each sequence node includes：Can randomly choose breadth first traversal mode and depth-first traversal mode it
One；And a node can be randomly selected from DOM tree structure, and using one node as start node, with institute
The traversal mode of selection extracts sequence node from DOM tree structure；And training unit, it is configured to be extracted
Sequence node trains the webpage distribution to indicate that model, the webpage distribution indicate that model is used to generate the table of input webpage
Show vector.
According to another aspect of the present disclosure, a kind of distributed method indicated generating webpage is additionally provided, including：It can be with
Generate the DOM tree structure of input webpage；One of breadth first traversal mode and depth-first traversal mode can be randomly choosed；With
And a node can be randomly selected from DOM tree structure, and using one node as start node, with selected
Traversal mode extracts the sequence node of predetermined length from DOM tree structure；And it can be utilized based on the sequence node extracted
Predetermined webpage distribution indicates model to generate the expression vector of input webpage.
According to the other aspects of the disclosure, additionally provide for realizing the above-mentioned computer program according to disclosed method
Code and computer program product and thereon record have this for realizing the above-mentioned computer program according to disclosed method
The computer readable storage medium of code.
The other aspects of the embodiment of the present disclosure are provided in following specification part, wherein be described in detail for abundant
Ground discloses the preferred embodiment of the embodiment of the present disclosure, without applying restriction to it.
Description of the drawings
The disclosure can by reference to being better understood below in association with the detailed description given by attached drawing, wherein
Same or analogous reference numeral has been used in all the appended drawings to indicate same or similar component.The attached drawing is together under
The detailed description in face includes in the present specification and to form part of specification together, for the disclosure is further illustrated
Preferred embodiment and explain the disclosure principle and advantage.Wherein：
Fig. 1 is to show that trained webpage distribution according to an embodiment of the present disclosure indicates the flow example of the method for model
Flow chart；
Fig. 2 is the exemplary figure for showing DOM tree structure according to an embodiment of the present disclosure；
Fig. 3 is the exemplary figure for showing to indicate the parameter of model according to the training webpage distribution of the embodiment of the present disclosure；
Fig. 4 is to show that trained webpage distribution according to an embodiment of the present disclosure indicates that the functional configuration of the device of model is shown
The block diagram of example；
Fig. 5 is the stream for the flow example for showing the distributed method indicated according to an embodiment of the present disclosure for generating webpage
Cheng Tu；
Fig. 6 is the exemplary figure for showing the similarity to two webpages according to the embodiment of the present disclosure and being compared；
Fig. 7 is the functional configuration example for showing the distributed device indicated according to an embodiment of the present disclosure for generating webpage
Block diagram；And
Fig. 8 is the example knot for being shown as the personal computer of adoptable information processing equipment in embodiment of the disclosure
The block diagram of structure.
Specific implementation mode
The exemplary embodiment of the disclosure is described hereinafter in connection with attached drawing.For clarity and conciseness,
All features of actual implementation mode are not described in the description.It should be understood, however, that developing any this actual implementation
Much decisions specific to embodiment must be made during example, to realize the objectives of developer, for example, symbol
Restrictive condition those of related to system and business is closed, and these restrictive conditions may have with the difference of embodiment
Changed.In addition, it will also be appreciated that although development is likely to be extremely complex and time-consuming, to having benefited from the disclosure
For those skilled in the art of content, this development is only routine task.
Herein, it is also necessary to which explanation is a bit, in order to avoid having obscured the disclosure because of unnecessary details, in the accompanying drawings
It illustrate only with according to the closely related device structure of the scheme of the disclosure and/or processing step, and be omitted and the disclosure
The little other details of relationship.
The present invention proposes a kind of method that trained webpage distribution indicates model, and this method, which uses, combines depth-first time
Go through the webpage distribution of the text message and structural information with the random sampling algorithms of breadth first traversal, to form fusion webpage
Formula indicates, form new semantic feature vector, the semantic feature vector can be used as webpage it is similar calculate, classified calculating
Basis.
It is described in detail below in conjunction with the accompanying drawings in accordance with an embodiment of the present disclosure.
First, the method 100 of trained webpage distribution expression model according to an embodiment of the present disclosure will be described referring to Fig.1
Flow example.Fig. 1 is to show that trained webpage distribution according to an embodiment of the present disclosure indicates the flow of the method 100 of model
Exemplary flow chart.As shown in Figure 1, trained webpage distribution according to an embodiment of the present disclosure indicates that the method 100 of model is wrapped
Include DOM tree structure generation step S102, extraction sequence node step S104 and training step S106.
In DOM tree structure generation step S102, the DOM tree structure of each webpage in multiple webpages can be generated.
As a specific example, can utilize well known to a person skilled in the art technology, for a large amount of auto-building html files
The DOM tree structure of each webpage.
Preferably, the generation of DOM tree structure includes removing the node for not including text information in webpage.
As a specific example, when generating the DOM tree structure of each webpage, it can remove and not include word in webpage
The node (that is, functional code) of information, such as meaningless html tag is removed, such as<style>,</style>,<script
>,</script>Deng.
Preferably, the generation of DOM tree structure further includes carrying out word segmentation processing to text node.
As a specific example, in the case where text node is Chinese, which can be divided
Word processing, and in the case where text node is English without carrying out word segmentation processing.
Fig. 2 is the exemplary figure for showing DOM tree structure according to an embodiment of the present disclosure.As shown in Fig. 2, in dom tree knot
There are multiple branches in structure, there are multiple layers in each branch, and the leaf node of each branch is text node.
In extracting sequence node step S104, it can be directed to the DOM tree structure of each webpage, extract the pre- of predetermined number
The sequence node of measured length, wherein the extraction of each sequence node includes：It randomly chooses breadth first traversal mode and depth is excellent
First one of traversal mode；And a node is randomly selected from DOM tree structure, and saved using one node as starting
Point extracts sequence node in a manner of selected traversal from DOM tree structure.
As a specific example, when extracting each node series of predetermined length from the DOM tree structure of each webpage,
One of breadth first traversal mode and depth-first traversal mode are randomly choosed first.Breadth first traversal mode is for DOM
The mode that tree construction is successively traversed.For example, for DOM tree structure as shown in Figure 2, if randomly selecting top layer
" div " node proceeds by breadth first traversal from the start node and extracts the node that length is 11 as start node
Sequence, then obtained breadth first traversal sequence be：div,tr,ul,td,td,td,li,this,is,my,job.Range is excellent
First traversal mode can more reflect the structural information of webpage.Depth-first traversal mode is for DOM tree structure by branch's progress time
The mode gone through.For example, for DOM tree structure as shown in Figure 2, if " this " node for randomly selecting the most lower left corner is used as
Beginning node proceeds by depth-first traversal from the start node and extracts the sequence node that length is 11, then obtained
Depth-first traversal sequence is：this,is,td,td,my,td,tr,job,li,ul,div.Depth-first traversal mode more can
Reflect the content information of webpage.
The predetermined number and the predetermined length can rule of thumb be predefined.For example, can be from each webpage
It extracts 100 sequence nodes in DOM tree structure, the length of window for choosing node can be set as 100, i.e., each sequence node
Length be 100.
Preferably, breadth first traversal mode and depth-first are randomly choosed using random number way or Alias algorithms
One of traversal mode.
As a specific example, when extracting each node series of predetermined length from the DOM tree structure of each webpage,
Use random number way or Alias algorithms in such a way that the probability of P chooses breadth first traversal, otherwise selected depth first traversal
Mode.
Preferably, when choosing one node, the probability for choosing list node is more than the general of selection text node
Rate.
As a specific example, a node is being randomly selected from DOM tree structure and is being made with one node
For start node when, choose one node according to certain probability, the probability for choosing list node is more than and chooses text
The probability of node.For example, choosing the list tubercle in Fig. 2<td>,<li>Deng probability be more than choose Fig. 2 in text node
<this>,<is>Deng probability.
In training step S106, model, webpage can be indicated based on the sequence node training webpage distribution extracted
Distribution indicates that model is used to generate the expression vector of input webpage.
It, can be based on the sequence node while training knot vector and entire HTML extracted as a specific example
Vector defines maximum likelihood function, and the parameter of model is indicated using stochastic gradient descent training webpage distribution, is updated simultaneously
HTML vectors and knot vector.
Fig. 3 is the exemplary figure for showing to indicate the parameter of model according to the training webpage distribution of the embodiment of the present disclosure.Under
Face illustrates to indicate the training of the parameter of model according to the webpage distribution of the embodiment of the present disclosure in conjunction with Fig. 3.
In calculating forward, input is and node<tr>,<ul>,<td>And<td>Corresponding knot vector and
" htmlv " vector, wherein " htmlv " indicates the document vector of entire HTML, random initializtion knot vector and " htmlv " to
Amount；By these knot vectors and " htmlv " addition of vectors to DUAL PROBLEMS OF VECTOR MAPPING layer, it is X to obtain mapping layer vector；Exporting node layer is
<tr>,<td>And<ul>, calculate Y=WX in output layer, wherein W=(w1,w2,…,wn) be parameter layer parameter, more specifically
Ground, can be by w1,w2,…,wnReferred to as Connecting quantity, Connecting quantity are the parameters that webpage distribution indicates model.
It preferably, can be for each node point in all nodes that the DOM tree structure of the multiple webpage includes
Do not calculate and occur the probability of occurrence of the node in the case of current context, and be directed to each node institute it is calculated go out
The sum of existing probability is up to target to train webpage distribution to indicate the parameter of model.
Defining maximum likelihood function is：
In formula (1), indicate all nodes in set of node (that is, being wrapped in the DOM tree structure of the multiple webpage with l
All nodes included) number, nodeiIndicate that the knot vector of i-th (i=1,2 ..., l) a node, htmlv indicate HTML page
The document vector in face, Context () indicate that current context, Average () expressions are averaged, and formula (1) shows：For section
Each node that point is concentrated, which calculates separately in the case of current context, there is the probability of occurrence of the node, and to make needle
To the sum of the calculated probability of occurrence of each node institute maximum.All output node layers are traversed, i.e., in output layer meter
Y=WX is calculated, and calculates cumulative errors, uses stochastic gradient descent (SGD) undated parameter W.
Then, using cumulative errors, the document vector of the vectorial and entire html page of each node is updated.It can root
Update times are defined according to experience, general update times are 5 times or so.
More clearly illustrate to indicate mould according to the webpage distribution of the embodiment of the present disclosure in conjunction with Fig. 3 with specific example below
The training of the parameter of type.
For simplified description, it is assumed that X and W are one-dimensional vector.Assuming that initial vector X is 3, W 0, then Y=WX=0.
If the actual value of Y is 1, error should be 1-0=1, it is, for example, possible to use parameter W is updated to 0+0.01x by SGD from 0
(1-0) x3=0.03.Then, using cumulative errors, initial vector X can be updated to 3+0.01x (1-0) x3=3.03.This
Sample, it is 0.0909 to calculate WX next time, is no longer 0.Above-mentioned update for several times can be carried out, until error is less than scheduled threshold value
Until.
In conjunction with Fig. 3 and above description it is found that can calculate separately each node in set of node in current context
In the case of there is the probability of occurrence of the node, and be up to mesh to be directed to the sum of calculated probability of occurrence of each node institute
The parameter for marking that webpage distribution is trained to indicate model.
In addition, as seen from the above description, webpage distribution indicates that model can generate the expression vector of input webpage
htmlv.Expression vector htmlv is that the webpage distribution for the text message and structural information for merging webpage indicates, is a kind of language
Adopted feature vector, the semantic feature vector can be used as webpage it is similar calculate, the basis of classified calculating.
Preferably, webpage distribution indicates that model can be linear classifier.
In conclusion trained webpage distribution according to an embodiment of the present disclosure indicate that the method 100 of model uses can be with
The sampling algorithm of the probability of selection breadth first traversal mode and depth-first traversal mode is adjusted, if selection breadth First time
The probability for going through mode is larger, then stresses the structural information of webpage, if the probability of selected depth first traversal mode is larger, side
Weight web page contents and semantic information, and if selection breadth first traversal mode is identical with the probability of depth-first traversal mode,
The structural information and semantic information of webpage can then be taken into account.That is, trained webpage distribution according to an embodiment of the present disclosure indicates
The method 100 of model forms the text message of fusion webpage and the webpage distribution of structural information indicates, it is special to form new semanteme
Sign vector, the semantic feature vector can be used as webpage it is similar calculate, the basis of classified calculating.
The embodiment of the method for model is indicated with above-mentioned trained webpage distribution correspondingly, and the disclosure additionally provides following instruction
Practice the embodiment that webpage distribution indicates the device of model.
Fig. 4 is to show that trained webpage distribution according to an embodiment of the present disclosure indicates that the function of the device 400 of model is matched
Set exemplary block diagram.
As shown in figure 4, trained webpage distribution according to an embodiment of the present disclosure indicates that the device 400 of model may include
DOM tree structure generation unit 402, extraction sequence node unit 404 and training unit 406.It is described below each unit
Functional configuration example.
In DOM tree structure generation unit 402, the DOM tree structure of each webpage in multiple webpages can be generated.
As a specific example, can utilize well known to a person skilled in the art technology, for a large amount of auto-building html files
The DOM tree structure of each webpage.
Preferably, the generation of DOM tree structure includes removing the node for not including text information in webpage.
As a specific example, when generating the DOM tree structure of each webpage, it can remove and not include word in webpage
The node (that is, functional code) of information, such as meaningless html tag is removed, such as<style>,</style>,<script
>,</script>Deng.
Preferably, the generation of DOM tree structure further includes carrying out word segmentation processing to text node.
As a specific example, in the case where text node is Chinese, which can be divided
Word processing, and in the case where text node is English without carrying out word segmentation processing.
Specific example about DOM tree structure may refer to the description of corresponding position in the above correlation method embodiment,
This is not repeated.
In extracting sequence node unit 404, it can be directed to the DOM tree structure of each webpage, extract the pre- of predetermined number
The sequence node of measured length, wherein the extraction of each sequence node includes：It randomly chooses breadth first traversal mode and depth is excellent
First one of traversal mode；And a node is randomly selected from DOM tree structure, and saved using one node as starting
Point extracts sequence node in a manner of selected traversal from DOM tree structure.
It may refer to from the specific example of each node series of the DOM tree structure of each webpage extraction predetermined length above
The description of corresponding position, is not repeated herein in correlation method embodiment.
Preferably, breadth first traversal mode and depth-first are randomly choosed using random number way or Alias algorithms
One of traversal mode.
As a specific example, when extracting each node series of predetermined length from the DOM tree structure of each webpage,
Use random number way or Alias algorithms in such a way that the probability of P chooses breadth first traversal, otherwise selected depth first traversal
Mode.
Preferably, when choosing one node, the probability for choosing list node is more than the general of selection text node
Rate.
As a specific example, a node is being randomly selected from DOM tree structure and is being made with one node
For start node when, choose one node according to certain probability, the probability for choosing list node is more than and chooses text
The probability of node.
In training unit 406, model, webpage point can be indicated based on the sequence node training webpage distribution extracted
Cloth indicates that model is used to generate the expression vector of input webpage.
It, can be based on the sequence node while training knot vector and entire HTML extracted as a specific example
Vector defines maximum likelihood function, and the parameter of model is indicated using stochastic gradient descent training webpage distribution, is updated simultaneously
HTML vectors and knot vector.
It preferably, can be for each node point in all nodes that the DOM tree structure of the multiple webpage includes
Do not calculate and occur the probability of occurrence of the node in the case of current context, and be directed to each node institute it is calculated go out
The sum of existing probability is up to target to train webpage distribution to indicate the parameter of model.
Training webpage distribution indicates that the specific example of the parameter of model may refer to phase in the above correlation method embodiment
The description for answering position, is not repeated herein.
As seen from the above description, webpage distribution indicates that model can generate the expression vector htmlv of input webpage.The table
Show that vectorial htmlv is that the webpage distribution for the text message and structural information for merging webpage indicates, be a kind of semantic feature vector,
The semantic feature vector can be used as webpage it is similar calculate, the basis of classified calculating.
Preferably, webpage distribution indicates that model can be linear classifier.
In conclusion trained webpage distribution according to an embodiment of the present disclosure indicate that the device 400 of model uses can be with
The sampling algorithm of the probability of selection breadth first traversal mode and depth-first traversal mode is adjusted, if selection breadth First time
The probability for going through mode is larger, then stresses the structural information of webpage, if the probability of selected depth first traversal mode is larger, side
Weight web page contents and semantic information, and if selection breadth first traversal mode is identical with the probability of depth-first traversal mode,
The structural information and semantic information of webpage can then be taken into account.That is, trained webpage distribution according to an embodiment of the present disclosure indicates
The device 400 of model forms the text message of fusion webpage and the webpage distribution of structural information indicates, it is special to form new semanteme
Sign vector, the semantic feature vector can be used as webpage it is similar calculate, the basis of classified calculating.
In addition, the disclosure additionally provides a kind of distributed method indicated generating webpage, this method, which uses, combines depth
The random sampling algorithms of first traversal and breadth first traversal, to formed fusion webpage text message and structural information it is defeated
Enter the distributed of webpage to indicate, form new semantic feature vector, which can be used as the similar meter of webpage
It calculates, the basis of classified calculating.
The distributed method 500 indicated according to an embodiment of the present disclosure for generating webpage is described below with reference to Fig. 5
Flow example.Fig. 5 is the flow example for showing the distributed method 500 indicated according to an embodiment of the present disclosure for generating webpage
Flow chart.As shown in figure 5, the distributed method 500 indicated according to an embodiment of the present disclosure for generating webpage includes dom tree
Structural generation step S502, random selection traversal mode step S504, extraction sequence node step S506 and generating indicate to
Measure step S508.
In DOM tree structure generation step S502, the DOM tree structure of input webpage can be generated.
As a specific example, can utilize well known to a person skilled in the art technology, for input auto-building html files DOM
Tree construction.
Preferably, the generation of DOM tree structure includes not including the node of text information in removal input webpage.
As a specific example, when generating the DOM tree structure of input webpage, it can remove and not include in input webpage
The node (that is, functional code) of text information, such as meaningless html tag is removed, such as<style>,</style>,<
script>,</script>Deng.
Preferably, the generation of DOM tree structure further includes carrying out word segmentation processing to text node.
As a specific example, in the case where text node is Chinese, which can be divided
Word processing, and in the case where text node is English without carrying out word segmentation processing.
Specific example about DOM tree structure may refer to the embodiment of the method that the above trained webpage distribution indicates model
The description of middle corresponding position, is not repeated herein.
In randomly choosing traversal mode step S504, breadth first traversal mode and depth-first time can be randomly choosed
Go through one of mode.
Breadth first traversal mode is the mode successively traversed for DOM tree structure.Breadth first traversal mode is more
It can reflect the structural information of webpage.Depth-first traversal mode is the mode traversed by branch for DOM tree structure.Depth
First traversal mode can more reflect the content information of webpage.Tool about breadth first traversal mode and depth-first traversal mode
Body example may refer to the description that the above trained webpage distribution indicates corresponding position in the embodiment of the method for model, herein no longer
It repeats.
Preferably, breadth first traversal mode and depth-first are randomly choosed using random number way or Alias algorithms
One of traversal mode.
It is used as a specific example when extracting the node series of predetermined length from the DOM tree structure of input webpage
Random number way or Alias algorithms are in such a way that the probability of P chooses breadth first traversal, otherwise selected depth first traversal side
Formula.
In extracting sequence node step S506, a node can be randomly selected from DOM tree structure, and with described
One node extracts the sequence node of predetermined length in a manner of selected traversal as start node from DOM tree structure.
Preferably, when choosing one node, the probability for choosing list node is more than the general of selection text node
Rate.
As a specific example, a node is being randomly selected from DOM tree structure and is being made with one node
For start node when, choose one node according to certain probability, the probability for choosing list node is more than and chooses text
The probability of node.
The predetermined length can rule of thumb be predefined.For example, can set choose node length of window as
100, i.e. the length of sequence node is 100.
The sequence node for extracting predetermined length from DOM tree structure in a manner of selected traversal may refer to the above training
The description of corresponding position, is not repeated herein in the embodiment of the method for webpage distribution expression model.
In generating the vectorial step S508 of expression, predetermined webpage distribution can be utilized based on the sequence node extracted
Model is indicated to generate the expression vector of input webpage.
As a specific example, using in the method for indicating model according to the training webpage distribution of the embodiment of the present disclosure
The knot vector for having trained the knot vector and the node in above extracted sequence node that come, again proceeds to and is describing
The process that error and undated parameter are calculated in output layer mentioned in the method for the above training webpage distribution expression model
In.The specific example of update HTML vectors may refer to the above trained webpage distribution and indicate in the embodiment of the method for model accordingly
The description of position, is not repeated herein.It should be noted that indicating mould according to the training webpage distribution of the embodiment of the present disclosure
In the method for type, HTML vectors and knot vector are updated simultaneously at no point in the update process；And in the distributed table of the generation webpage
In the method shown, at no point in the update process not concept transfer vector and only change HTML vector.
As seen from the above description, it can be generated according to the distributed method indicated of the generation webpage of the embodiment of the present disclosure defeated
Enter the expression vector of webpage.The expression vector is that the webpage distribution for the text message and structural information for merging webpage indicates, is
A kind of semantic feature vector, the semantic feature vector can be used as webpage it is similar calculate, the basis of classified calculating.
Preferably, predetermined webpage distribution indicates that model can be linear classifier.
Fig. 6 is the exemplary figure for showing the similarity to two webpages according to the embodiment of the present disclosure and being compared.In order to
Description is convenient, and the webpage in left side in Fig. 6 is known as webpage 1, and the webpage on right side in Fig. 6 is known as webpage 2.As shown in fig. 6, net
The content of page 1 includes commodity " A " and weight " B ", and the content of webpage 2 includes age " C " and height " D ", i.e. webpage 1 and webpage
2 content is dissimilar, but the structure of webpage 1 and webpage 2 is more similar.
As a specific example, the distributed method 500 indicated for generating webpage according to the embodiment of the present disclosure is utilized
The expression vector of the webpage 1 and webpage 2 in Fig. 6 is generated respectively.As described above, it is assumed that with the probability selection breadth first traversal of P
Mode, otherwise selected depth first traversal mode.If P is close to 1, i.e., in a manner of greater probability selection breadth first traversal, extensively
Degree first traversal mode can more reflect the structural information of webpage, since the structure of webpage 1 and webpage 2 is more similar, utilize
According to the expression vector of webpage 1 and webpage 2 that the distributed method 500 indicated of the generation webpage of the embodiment of the present disclosure is generated
Relatively, it is determined that the similarity degree of webpage 1 and webpage 2 is high；And if P close to 0, i.e., it is preferential with greater probability selected depth
Traversal mode, depth-first traversal mode can more reflect the content information of webpage, since the content of webpage 1 and webpage 2 is dissimilar,
Therefore the webpage 1 and webpage 2 generated according to the method 500 of the distributed expression of the generations webpage of the embodiment of the present disclosure is utilized
Indicate that vector difference is larger, it is determined that webpage 1 is low with the similarity degree of webpage 2.
In conclusion the distributed method 500 indicated according to an embodiment of the present disclosure for generating webpage is using can adjust
The sampling algorithm of the probability of breadth first traversal mode and depth-first traversal mode is selected in selected parts, if selection breadth first traversal
The probability of mode is larger, then stresses the structural information of webpage, if the probability of selected depth first traversal mode is larger, stresses
Web page contents and semantic information, and if selection breadth first traversal mode is identical with the probability of depth-first traversal mode,
The structural information and semantic information of webpage can be taken into account.That is, webpage distributed according to an embodiment of the present disclosure that generate indicates
Method 500 formed fusion webpage text message and structural information webpage distribution indicate, formed new semantic feature to
Amount, the semantic feature vector can be used as webpage it is similar calculate, the basis of classified calculating.
With the distributed embodiment of the method indicated of above-mentioned generation webpage correspondingly, the disclosure additionally provides following generation
The embodiment of the distributed device indicated of webpage.
Fig. 7 is the functional configuration for showing the distributed device 700 indicated according to an embodiment of the present disclosure for generating webpage
Exemplary block diagram.
As shown in fig. 7, the distributed device 700 indicated according to an embodiment of the present disclosure for generating webpage may include
DOM tree structure generation unit 702, random selection traversal mode unit 704, extraction sequence node unit 706 and generation indicate
Vector location 708.It is described below the functional configuration example of each unit.
In DOM tree structure generation unit 702, the DOM tree structure of input webpage can be generated.
As a specific example, can utilize well known to a person skilled in the art technology, for input auto-building html files DOM
Tree construction.
Preferably, the generation of DOM tree structure includes not including the node of text information in removal input webpage.
As a specific example, when generating the DOM tree structure of input webpage, it can remove and not include in input webpage
The node (that is, functional code) of text information.
Preferably, the generation of DOM tree structure further includes carrying out word segmentation processing to text node.
As a specific example, in the case where text node is Chinese, which can be divided
Word processing, and in the case where text node is English without carrying out word segmentation processing.
Specific example about DOM tree structure may refer to the embodiment of the method that the above trained webpage distribution indicates model
The description of middle corresponding position, is not repeated herein.
In randomly choosing traversal mode unit 704, breadth first traversal mode and depth-first time can be randomly choosed
Go through one of mode.
Specific example about random selection breadth first traversal mode and depth-first traversal mode may refer to above
The description of corresponding position, is not repeated herein in embodiment of the method.
Preferably, breadth first traversal mode and depth-first are randomly choosed using random number way or Alias algorithms
One of traversal mode.
It is used as a specific example when extracting the node series of predetermined length from the DOM tree structure of input webpage
Random number way or Alias algorithms are in such a way that the probability of P chooses breadth first traversal, otherwise selected depth first traversal side
Formula.
In extracting sequence node unit 706, a node can be randomly selected from DOM tree structure, and with described
One node extracts the sequence node of predetermined length in a manner of selected traversal as start node from DOM tree structure.
Preferably, when choosing one node, the probability for choosing list node is more than the general of selection text node
Rate.
As a specific example, a node is being randomly selected from DOM tree structure and is being made with one node
For start node when, choose one node according to certain probability, the probability for choosing list node is more than and chooses text
The probability of node.
The predetermined length can rule of thumb be predefined.For example, can set choose node length of window as
100, i.e. the length of sequence node is 100.
The sequence node for extracting predetermined length from DOM tree structure in a manner of selected traversal may refer to the above training
The description of corresponding position, is not repeated herein in the embodiment of the method for webpage distribution expression model.
In generating expression vector location 708, predetermined webpage distribution table can be utilized based on the sequence node extracted
Representation model is vectorial come the expression for generating input webpage.
Based on the sequence node extracted, indicate model using predetermined webpage distribution generate the expression of input webpage to
The specific example of amount may refer to the description that the above trained webpage distribution indicates corresponding position in the embodiment of the method for model,
This is not repeated.
Preferably, predetermined webpage distribution indicates that model can be linear classifier.
In conclusion the distributed device 700 indicated according to an embodiment of the present disclosure for generating webpage is using can adjust
The sampling algorithm of the probability of breadth first traversal mode and depth-first traversal mode is selected in selected parts, if selection breadth first traversal
The probability of mode is larger, then stresses the structural information of webpage, if the probability of selected depth first traversal mode is larger, stresses
Web page contents and semantic information, and if selection breadth first traversal mode is identical with the probability of depth-first traversal mode,
The structural information and semantic information of webpage can be taken into account.That is, webpage distributed according to an embodiment of the present disclosure that generate indicates
Device 700 formed fusion webpage text message and structural information webpage distribution indicate, formed new semantic feature to
Amount, the semantic feature vector can be used as webpage it is similar calculate, the basis of classified calculating.
It is noted that although the foregoing describe the devices that trained webpage distribution according to an embodiment of the present disclosure indicates model
The functional configuration of 400 devices 700 indicated with the distribution for generating webpage, but this is only exemplary rather than limitation, and ability
Field technique personnel can modify to above example according to the principle of the disclosure, such as can be to the function mould in each embodiment
Block is added, deletes or combines, and such modification is each fallen in the scope of the present disclosure.
It is furthermore to be noted that device embodiment here is corresponding with above method embodiment, therefore in device reality
The description that the content not being described in detail in example can be found in corresponding position in embodiment of the method is applied, is not repeated to describe herein.
It should be understood that the instruction that the machine in storage medium and program product according to an embodiment of the present disclosure can perform may be used also
The method of model and the distributed method indicated of generation webpage are indicated to be configured to execute above-mentioned trained webpage distribution, because
This content not being described in detail herein can refer to the description of previous corresponding position, be not repeated to be described herein.
Correspondingly, the storage medium of the program product for carrying the above-mentioned instruction that can perform including machine is also included within this
In the disclosure of invention.The storage medium includes but not limited to floppy disk, CD, magneto-optic disk, storage card, memory stick etc..
In addition, it should also be noted that above-mentioned series of processes and device can also be realized by software and/or firmware.?
In the case of being realized by software and/or firmware, from storage medium or network to the computer with specialized hardware structure, such as
The installation of general purpose personal computer 800 shown in Fig. 8 constitutes the program of the software, and the computer is when being equipped with various programs, energy
Enough perform various functions etc..
In fig. 8, central processing unit (CPU) 801 is according to the program stored in read-only memory (ROM) 802 or from depositing
The program that storage part 808 is loaded into random access memory (RAM) 803 executes various processing.In RAM 803, also according to need
Store the data required when CPU 801 executes various processing etc..
CPU 801, ROM 802 and RAM 803 are connected to each other via bus 804.Input/output interface 805 is also connected to
Bus 804.
Components described below is connected to input/output interface 805：Importation 806, including keyboard, mouse etc.；Output par, c
807, including display, such as cathode-ray tube (CRT), liquid crystal display (LCD) etc. and loud speaker etc.；Storage section 808,
Including hard disk etc.；With communications portion 809, including network interface card such as LAN card, modem etc..Communications portion 809 via
Network such as internet executes communication process.
As needed, driver 810 is also connected to input/output interface 805.Detachable media 811 such as disk, light
Disk, magneto-optic disk, semiconductor memory etc. are installed on driver 810 as needed so that the computer journey read out
Sequence is mounted to as needed in storage section 808.
It is such as removable from network such as internet or storage medium in the case of series of processes above-mentioned by software realization
Unload the program that the installation of medium 811 constitutes software.
It will be understood by those of skill in the art that this storage medium be not limited to it is shown in Fig. 8 wherein have program stored therein,
Separately distribute with equipment to provide a user the detachable media 811 of program.The example of detachable media 811 includes disk
(including floppy disk (registered trademark)), CD (comprising compact disc read-only memory (CD-ROM) and digital versatile disc (DVD)), magneto-optic disk
(including mini-disk (MD) (registered trademark)) and semiconductor memory.Alternatively, storage medium can be ROM 802, storage section
Hard disk for including in 808 etc., wherein computer program stored, and user is distributed to together with the equipment comprising them.
Preferred embodiment of the present disclosure is described above by reference to attached drawing, but the disclosure is certainly not limited to above example.This
Field technology personnel can obtain various changes and modifications within the scope of the appended claims, and should be understood that these changes and repair
Changing nature will fall into scope of the presently disclosed technology.
For example, can be realized in the embodiment above by the device separated including multiple functions in a unit.
As an alternative, the multiple functions of being realized in the embodiment above by multiple units can be realized by the device separated respectively.In addition, with
One of upper function can be realized by multiple units.Needless to say, such configuration includes in scope of the presently disclosed technology.
In this specification, described in flow chart the step of includes not only the place executed in temporal sequence with the sequence
Reason, and include concurrently or individually rather than the processing that must execute in temporal sequence.In addition, even in temporal sequence
In the step of processing, needless to say, the sequence can also be suitably changed.
In addition, can also be configured as follows according to the technology of the disclosure.
A kind of 1. methods that trained webpage distribution indicates model are attached, including：
Generate DOM Document Object Model (DOM) tree construction of each webpage in multiple webpages；
For the DOM tree structure of each webpage, the sequence node of the predetermined length of predetermined number is extracted, wherein each node
The extraction of sequence includes：
Randomly choose one of breadth first traversal mode and depth-first traversal mode；And
A node is randomly selected from the DOM tree structure, and using one node as start node, with institute
The traversal mode of selection extracts the sequence node from the DOM tree structure；And
The webpage distribution is trained to indicate that model, the webpage distribution indicate model based on the sequence node extracted
Expression vector for generating input webpage.
The method that training webpage distribution of the note 2. according to note 1 indicates model, wherein use random number way
Or Alias algorithms randomly choose one of the breadth first traversal mode and the depth-first traversal mode.
The method that training webpage distribution of the note 3. according to note 1 indicates model, wherein one choosing
When node, the probability for choosing list node is more than the probability for choosing text node.
The method that training webpage distribution of the note 4. according to note 1 indicates model, wherein the DOM tree structure
Generation include remove webpage in do not include text information node.
It is attached the method that 5. trained webpage distributions described in note 4 indicate model, wherein the DOM tree structure
Generation further include to text node carry out word segmentation processing.
The method that training webpage distribution of the note 6. according to note 1 indicates model, wherein for the multiple net
Each node in all nodes that the DOM tree structure of page includes is calculated separately there is the section in the case of current context
The probability of occurrence of point, and train the webpage to be up to target for the sum of calculated probability of occurrence of each node institute
Distribution indicates the parameter of model.
The method that training webpage distribution of the note 7. according to note 1 indicates model, wherein the webpage is distributed
Indicate that model is linear classifier.
A kind of 8. devices of trained webpage distribution expression model are attached, including：
DOM Document Object Model generation unit is configured to generate the DOM Document Object Model of each webpage in multiple webpages
DOM tree structure；
Sequence node unit is extracted, the DOM tree structure for each webpage is configured to, extracts the pre- fixed length of predetermined number
The sequence node of degree, wherein the extraction of each sequence node includes：
Randomly choose one of breadth first traversal mode and depth-first traversal mode；And
A node is randomly selected from the DOM tree structure, and using one node as start node, with institute
The traversal mode of selection extracts the sequence node from the DOM tree structure；And
Training unit is configured to train the webpage distribution to indicate model based on the sequence node extracted, described
Webpage distribution indicates that model is used to generate the expression vector of input webpage.
Note 9. indicates the device of model according to trained webpage distribution described in note 8, wherein uses random number way
Or Alias algorithms randomly choose one of the breadth first traversal mode and the depth-first traversal mode.
Note 10. indicates the device of model according to trained webpage distribution described in note 8, wherein is choosing described one
When a node, the probability for choosing list node is more than the probability for choosing text node.
Note 11. indicates the device of model according to trained webpage distribution described in note 8, wherein the DOM tree structure
Generation include remove webpage in do not include text information node.
Training webpage distribution of the note 12. according to note 11 indicates the device of model, wherein the dom tree knot
The generation of structure further includes carrying out word segmentation processing to text node.
Note 13. indicates the device of model according to trained webpage distribution described in note 8, wherein for the multiple
Each node in all nodes that the DOM tree structure of webpage includes is calculated separately to be somebody's turn to do in the case of current context
The probability of occurrence of node, and train the net to be up to target for the sum of calculated probability of occurrence of each node institute
The distributed parameter for indicating model of page.
Note 14. indicates the device of model according to trained webpage distribution described in note 8, wherein the webpage distribution
Formula indicates that model is linear classifier.
A kind of 15. distributed methods indicated generating webpage are attached, including：
Generate DOM Document Object Model (DOM) tree construction of input webpage；
Randomly choose one of breadth first traversal mode and depth-first traversal mode；And
A node is randomly selected from the DOM tree structure, and using one node as start node, with institute
The traversal mode of selection extracts the sequence node of predetermined length from the DOM tree structure；And
Based on the sequence node extracted, model is indicated using predetermined webpage distribution to generate the table of the input webpage
Show vector.
The distributed method indicated of generation webpage of the note 16. according to note 15, wherein use random number way
Or Alias algorithms randomly choose one of the breadth first traversal mode and the depth-first traversal mode.
The distributed method indicated of generation webpage of the note 17. according to note 15, wherein one choosing
When node, the probability for choosing list node is more than the probability for choosing text node.
The distributed method indicated of generation webpage of the note 18. according to note 15, wherein the DOM tree structure
Generation include removal it is described input webpage in do not include text information node.
The distributed method indicated of generation webpage of the note 19. according to note 18, wherein the DOM tree structure
Generation further include to text node carry out word segmentation processing.
The distributed method indicated of generation webpage of the note 20. according to note 15, wherein the predetermined webpage point
Cloth indicates that model is linear classifier.
Claims (10)
1. a kind of method that trained webpage distribution indicates model, including：
Generate the DOM Document Object Model DOM tree structure of each webpage in multiple webpages；
For the DOM tree structure of each webpage, the sequence node of the predetermined length of predetermined number is extracted, wherein each sequence node
Extraction include：
Randomly choose one of breadth first traversal mode and depth-first traversal mode；And
A node is randomly selected from the DOM tree structure, and using one node as start node, with selected
Traversal mode the sequence node is extracted from the DOM tree structure；And
The webpage distribution is trained to indicate that model, the webpage distribution indicate that model is used for based on the sequence node extracted
Generate the expression vector of input webpage.
2. the method that trained webpage distribution according to claim 1 indicates model, wherein using random number way or
Alias algorithms randomly choose one of the breadth first traversal mode and the depth-first traversal mode.
3. the method that trained webpage distribution according to claim 1 indicates model, wherein choosing one node
When, the probability for choosing list node is more than the probability for choosing text node.
4. the method that trained webpage distribution according to claim 1 indicates model, wherein the life of the DOM tree structure
At the node including not including text information in removal webpage.
5. the method that trained webpage distribution according to claim 4 indicates model, wherein the life of the DOM tree structure
At further include to text node carry out word segmentation processing.
6. the method that trained webpage distribution according to claim 1 indicates model, wherein for the multiple webpage
Each node in all nodes that DOM tree structure includes is calculated separately there is the node in the case of current context
Probability of occurrence, and train the webpage to be distributed to be up to target for the sum of calculated probability of occurrence of each node institute
Formula indicates the parameter of model.
7. the method that trained webpage distribution according to claim 1 indicates model, wherein the webpage distribution indicates
Model is linear classifier.
8. a kind of trained webpage distribution indicates the device of model, including：
DOM Document Object Model generation unit is configured to generate the DOM Document Object Model dom tree of each webpage in multiple webpages
Structure；
Sequence node unit is extracted, the DOM tree structure for each webpage is configured to, extracts the predetermined length of predetermined number
Sequence node, wherein the extraction of each sequence node includes：
Randomly choose one of breadth first traversal mode and depth-first traversal mode；And
A node is randomly selected from the DOM tree structure, and using one node as start node, with selected
Traversal mode the sequence node is extracted from the DOM tree structure；And
Training unit is configured to train the webpage distribution to indicate model, the webpage based on the sequence node extracted
Distribution indicates that model is used to generate the expression vector of input webpage.
9. trained webpage distribution according to claim 8 indicates the device of model, wherein using random number way or
Alias algorithms randomly choose one of the breadth first traversal mode and the depth-first traversal mode.
10. a kind of distributed method indicated generating webpage, including：
Generate the DOM Document Object Model DOM tree structure of input webpage；
Randomly choose one of breadth first traversal mode and depth-first traversal mode；And
A node is randomly selected from the DOM tree structure, and using one node as start node, with selected
Traversal mode the sequence node of predetermined length is extracted from the DOM tree structure；And
Based on the sequence node extracted, indicate model using predetermined webpage distribution generate the expression of the input webpage to
Amount.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710239759.9A CN108733405A (en) | 2017-04-13 | 2017-04-13 | The method and apparatus that training webpage distribution indicates model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710239759.9A CN108733405A (en) | 2017-04-13 | 2017-04-13 | The method and apparatus that training webpage distribution indicates model |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108733405A true CN108733405A (en) | 2018-11-02 |
Family
ID=63923692
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710239759.9A Pending CN108733405A (en) | 2017-04-13 | 2017-04-13 | The method and apparatus that training webpage distribution indicates model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108733405A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110992194A (en) * | 2019-12-04 | 2020-04-10 | 中国太平洋保险（集团）股份有限公司 | User reference index algorithm based on attribute-containing multi-process sampling graph representation learning model |
| CN111966932A (en) * | 2019-05-20 | 2020-11-20 | 富士通株式会社 | Information processing method and information processing apparatus |
| CN112148943A (en) * | 2020-09-27 | 2020-12-29 | 北京天融信网络安全技术有限公司 | Webpage classification method and device, electronic equipment and readable storage medium |
| CN112347332A (en) * | 2020-11-17 | 2021-02-09 | 南开大学 | XPath-based crawler target positioning method |
| CN113807050A (en) * | 2021-07-01 | 2021-12-17 | 西安华讯科技有限责任公司 | Node interception method, system, equipment and storage medium based on rich text |
| WO2023155303A1 (en) * | 2022-02-16 | 2023-08-24 | 平安科技（深圳）有限公司 | Webpage data extraction method and apparatus, computer device, and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101694668A (en) * | 2009-09-29 | 2010-04-14 | 百度在线网络技术（北京）有限公司 | Method and device for confirming web structure similarity |
| US20100185684A1 (en) * | 2009-01-09 | 2010-07-22 | Amit Madaan | High precision multi entity extraction |
| CN101944094A (en) * | 2009-07-06 | 2011-01-12 | 富士通株式会社 | Webpage information extraction method and device thereof |
| US20110093773A1 (en) * | 2009-10-19 | 2011-04-21 | Browsera LLC | Automated application compatibility testing |
| CN103049562A (en) * | 2012-12-31 | 2013-04-17 | 华为技术有限公司 | Method and device for recognizing similar webpages |
| CN103544210A (en) * | 2013-09-02 | 2014-01-29 | 烟台中科网络技术研究所 | System and method for identifying webpage types |
| CN106227882A (en) * | 2016-08-02 | 2016-12-14 | 浙江大学 | A kind of accessible web page navigation method extracted based on navigation object |
-
2017
- 2017-04-13 CN CN201710239759.9A patent/CN108733405A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100185684A1 (en) * | 2009-01-09 | 2010-07-22 | Amit Madaan | High precision multi entity extraction |
| CN101944094A (en) * | 2009-07-06 | 2011-01-12 | 富士通株式会社 | Webpage information extraction method and device thereof |
| CN101694668A (en) * | 2009-09-29 | 2010-04-14 | 百度在线网络技术（北京）有限公司 | Method and device for confirming web structure similarity |
| US20110093773A1 (en) * | 2009-10-19 | 2011-04-21 | Browsera LLC | Automated application compatibility testing |
| CN103049562A (en) * | 2012-12-31 | 2013-04-17 | 华为技术有限公司 | Method and device for recognizing similar webpages |
| CN103544210A (en) * | 2013-09-02 | 2014-01-29 | 烟台中科网络技术研究所 | System and method for identifying webpage types |
| CN106227882A (en) * | 2016-08-02 | 2016-12-14 | 浙江大学 | A kind of accessible web page navigation method extracted based on navigation object |
Non-Patent Citations (2)
| Title |
|---|
| CHUNYING KANG: "DOM-based Web Pages to Determine the Structure of the Similarity Algorithm", 《PROCEEDINGS OF THE 3RD INTERNATIONAL CONFERENCE ON INTELLIGENT INFORMATION TECHNOLOGY APPLICATION》 * |
| 陈屹: "基于多特征的网页信息抽取技术的研究与应用", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111966932A (en) * | 2019-05-20 | 2020-11-20 | 富士通株式会社 | Information processing method and information processing apparatus |
| CN110992194A (en) * | 2019-12-04 | 2020-04-10 | 中国太平洋保险（集团）股份有限公司 | User reference index algorithm based on attribute-containing multi-process sampling graph representation learning model |
| CN112148943A (en) * | 2020-09-27 | 2020-12-29 | 北京天融信网络安全技术有限公司 | Webpage classification method and device, electronic equipment and readable storage medium |
| CN112347332A (en) * | 2020-11-17 | 2021-02-09 | 南开大学 | XPath-based crawler target positioning method |
| CN113807050A (en) * | 2021-07-01 | 2021-12-17 | 西安华讯科技有限责任公司 | Node interception method, system, equipment and storage medium based on rich text |
| CN113807050B (en) * | 2021-07-01 | 2024-04-09 | 西安华讯科技有限责任公司 | Node interception method, system, equipment and storage medium based on rich text |
| WO2023155303A1 (en) * | 2022-02-16 | 2023-08-24 | 平安科技（深圳）有限公司 | Webpage data extraction method and apparatus, computer device, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108733405A (en) | The method and apparatus that training webpage distribution indicates model | |
| US11475209B2 (en) | Device, system, and method for extracting named entities from sectioned documents | |
| CN104160392B (en) | Semantic estimating unit, method | |
| JP2022541199A (en) | A system and method for inserting data into a structured database based on image representations of data tables. | |
| CN104239300B (en) | The method and apparatus that semantic key words are excavated from text | |
| US7801924B2 (en) | Decision tree construction via frequent predictive itemsets and best attribute splits | |
| CN102129560B (en) | Method and device for identifying characters | |
| CN107943847A (en) | Business connection extracting method, device and storage medium | |
| CN109871491A (en) | Forum postings recommended method, system, equipment and storage medium | |
| CN105512277B (en) | A kind of short text clustering method towards Book Market title | |
| WO2019077405A1 (en) | Method, device, and system, for identifying data elements in data structures | |
| JP2018132969A (en) | Sentence preparation device | |
| Shigarov et al. | TabbyPDF: Web-based system for PDF table extraction | |
| KR20150109447A (en) | Text input system and method | |
| US20230104036A1 (en) | Fast front tracking in eor flooding simulation on coarse grids | |
| CN108304377A (en) | A kind of extracting method and relevant apparatus of long-tail word | |
| CN107193806A (en) | A kind of vocabulary justice former automatic prediction method and device | |
| CN112667940A (en) | Webpage text extraction method based on deep learning | |
| CN110020005A (en) | Symptom matching process in main suit and present illness history in a kind of case history | |
| CN108804472A (en) | A kind of webpage content extraction method, device and server | |
| CN106599280A (en) | Webpage node path information determination method and apparatus | |
| CN107169011B (en) | Webpage originality identification method and device based on artificial intelligence and storage medium | |
| CN117034948B (en) | Paragraph identification method, system and storage medium based on multi-feature self-adaptive fusion | |
| CN111930944B (en) | File label classification method and device | |
| CN104572787A (en) | Method and device for recognizing pseudo original website |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20181102 |
|
| WD01 | Invention patent application deemed withdrawn after publication |