CN102663022B - Classification recognition method based on URL (uniform resource locator) - Google Patents

Abstract

The invention relates to a classification recognition method based on URL (uniform resource locator), comprising the following steps: step 1 of classifying web pages serving advertisements by a classifier to obtain classes of web pages corresponding to all URLs of a website; step 2 of generating an URL tree of the website according to all URLs of the website; step 3 of matching the URL tree according to URL requested by advertisements, and returning a matching result. The classification recognition method based on URL described in the invention can solve the problem that advertisement matching is delayed, URL memory space is large, and pages without indexes are not classified in time.

Description

A kind of classifying identification method based on URL

Technical field

The present invention relates to internet, particularly relate to a kind of classifying identification method based on URL.

Background technology

In Internet advertising, the advertisement relevant to web page contents is embedded in the page.When a user have accessed a webpage, its publisher will consult and request to advertising networks such as such as Google, Microsoft and Yahoo and ask advertisement.Due to the requirement that delay, communication cost and memory space etc. are harsh, full page is issued advertising network business to publisher and advertising network business removes to crawl full page within the millisecond time, and it is all infeasible that parsing content chooses maximally related advertisement again.

Way general is at present all crawl media page under line, and extract its classification and keyword from content of pages, advertisement itself is also like this, to be provided by advertiser with one group and from advertisement title, the classification and keyword of extracting and obtaining are described for feature.In order to quick respond services, URL and the sort key word of all these pages store in the index correspondingly.When there being ad-request, the classified information that index is used to retrieve corresponding URL chooses the advertisement of mating most again.But the access frequency of webpage follows a power-law distribution, the page of the overwhelming majority is seldom accessed, these pages can not crawled and indexed come online advertising, because process and the carrying cost of index make us hanging back under line, and all webpage URL and classification thereof are stored such mode can cause memory space overload.Those have dynamic content or user to generate still more needs the page of authentication cannot capture in advance and index especially.These webpages not in advertising service index are all not indexed webpages, for the ad-request of non-index webpage, by conventional method, because it is not in index, then can not cannot complete online service request.

Summary of the invention

In order to solve above-mentioned technical matters, provide a kind of classifying identification method based on URL.

The invention provides a kind of classifying identification method based on URL, comprising:

Step 1, utilizes sorter to classify to the website and webpage throwing in advertisement, obtains the classification of the corresponding webpage of all URL in this website;

Step 2, generates the URL tree of this website according to all URL of this website;

Step 3, mates described URL according to the URL of ad-request and sets, return matching result.

In one example, step 2 comprises:

Step 21, carries out cutting to URL and obtains its characterization value;

Step 22, according to calculate key word k corresponding to the maximum information ratio of profit increase value of gained as the next node generating URL tree, wherein:

IG(k，C)＝H(C)-H(C|k)；

$H\left(C\right)=-\underset{i}{\mathrm{\Σ}}p\left(c_i\right)\log p\left(c_i\right);$

$H\left(C\right|k)=-\underset{v}{\mathrm{\Σ}}p(k=v)\underset{i}{\mathrm{\Σ}}p\left(c_i\right|k=v)\log p\left(c_i\right|k=v);$

$H\left(k\right)=-\underset{v}{\mathrm{\Σ}}p(k=v)\log p(k=v);$

$p\left(c_i\right)=\frac{{\mathrm{\Σ}}_{u\∈c_i}\mathrm{Imp}\left(u\right)}{{\mathrm{\Σ}}_{u^{\′}\∈U}\mathrm{Imp}\left(u\right)};$

$p(k=v)=\frac{{\mathrm{\Σ}}_{u\∋(u\left(k\right)=v)}\mathrm{Imp}\left(u\right)}{{\mathrm{\Σ}}_{u^{\′}\∈U}\mathrm{Imp}\left(u\right)};$

Imp ^d(u)＝(1-α)Imp ^d-1(u)+αfreq ^d(u)

C represents classification, and Imp (u) is the access frequency of URL u, and α represents smoothing factor, Imp ^du () is by the access frequency freq of particular day d ^dthe Imp of (u) and the previous day ^d-1u () calculates and obtains;

Step 23, when under node, URL belongs to same class, or the access times of URL that contain of node are less than the threshold value that presets and both candidate nodes segmentation does not have statistical significance, or do not have attribute can be used further to segmentation, then split stopping.

In one example, in step 23, the chi-square value of computing node segmentation, exceedes threshold value if fail, and segmentation stops, otherwise continues segmentation.

In one example, step 2 also comprises step 24, adopts pessimistic wrong pruning method to carry out beta pruning to URL tree.

In one example, in step 3, if there is not the path of mating completely with the URL of ad-request, according to formula in URL tree calculate URL and the similarity in path in tree, return the classification of classified information as this ad-request URL in the highest path of Similarity value, wherein, represent url u _iwith path p _iidentical sign K-V is to number; represent path p _isign K-V to total number.

In one example, if there is not the path of mating completely with the URL of ad-request in URL tree, the web data also crawling this URL institutional framework similar carries out parsing classification, and is added in URL tree as subtree by achievement step by these URL.

The classifying identification method based on URL described in the present invention postpones advertising matches, URL memory space greatly, the non-index pages problem such as can not to classify in time has and well improves process.

Accompanying drawing explanation

Fig. 1 is URL Classification and Identification flow process (comprises and build URL tree and the incremental learning set);

Fig. 2 is that website www.rocawear.com sets partial data figure;

Fig. 3 is the part URL Classification and Identification tree-model of website www.rocawear.com.

Embodiment

Throw in coupling in Internet advertising, due to current way be media page is crawled after, by means such as parsing, classification, webpage is classified by its content of text, and classified and to be stored in correspondingly in index file with the URL of webpage, when there being ad-request, then the classified information of searching corresponding URL in index is gone to choose match advertisements again.All webpage URL are stored such mode and can cause memory space overload, and under the necessary line of all pages, treatment classification is complete, the new page of not processed mistake can not be classified in time, so just can not complete online service request.

For these bottlenecks, taxonomic structure information may be there is in the method hypothesis URL that the present invention proposes, cluster forms the possible corresponding similar classification of web page contents at similar URL place, and such cluster URL collection is combined together and gives classification according to prior study by the idea of similar cluster.So main thought of the present invention is that the classification that in tree, each branching pathway is corresponding respective, classified information is stored in leaf node for a URL Classification and Identification tree is set up in each website.When there being ad-request, only the URL of the page need be gone to online path in Match Tree can obtain maximum probability classified information, even newly-increased URL fails to mate Complete Path, also the most similar classified information can be obtained, can extract this URL by the page that its composition crawls similar URL is that URL tree sets up shoot, so method meets incremental learning simultaneously.

The method that the present invention describes comprises: utilize the existing multi-categorizer module based on support vector machine to classify to the website and webpage throwing in advertisement, obtain the classification of the corresponding webpage of all URL in this website, with the tree node dividing method improved and stopping dividing method, all URL are characterized the URL tree of a generation website again by after all URL cutting characterization of this website, a corresponding one tree in website, the leaf node of tree has the classified information of respective paths, also introduce the access frequency of webpage first in the computation process of contributing simultaneously, again redundancy and the little branch of classification contribution are pruned according to its classification error rate after tree generates, finally obtain a website URL Classification and Identification tree, all URL of a website are expressed as such tree, the memory space of URL now will greatly reduce, when there being ad-request, its URL is characterized and goes to path in Match Tree, the existing page can be classified to leaf node by complete match, can perform fast and meet online service demand, if newly-increased URL (non-index pages) does not belong to existing branch, then can compatible portion path, and return similar classified information, taken out by this URL, the webpage crawling this URL institutional framework similar is that URL tree sets up new branch, meets incremental learning again.URL Classification and Identification flow process as shown in Figure 1.

To each website, crawl all pages and classified, setting up a URL tokens as attribute (wherein K is as attribute tags, and V is as property value), webpage classification is set as the URL Classification and Identification of class tag storage in leaf node.After building up tree, carry out a URL request, then remove Match Tree, return the classification that leaf node identifies.Below a few joint content describe in order the characterization of URL, the generation of tree node allocation and stop segmentation, the beta pruning process of tree, the incremental learning of URL Classification and Identification and tree, the Similarity Measure in path in only relating to URL online and setting, all the other are all calculate under line.Except needing to provide URL generic, the access frequency that each webpage in a period of time is provided also is needed in the training corpus of contributing.

URL characterization

URL static part "/", dynamic part with "? ", " & ", "; " etc. cutting recognition property, by "=" number separation parameter pair.Static part { k ₁, k ₂..., k _nrepresent, each parameter name of dynamic part is just defined as a K.Such as URL:

http://shopping.yahoo.com/electronics/cameras/digicams/product.php？prodid＝1000&sort＝price

Obtain it after cutting and characterize K → V

K＝{k ₁，k ₂，...，k ₆，k _prodid，k _sort}

V＝{http，shopping.yahoo.com，electronics，cameras，digicams，product.php，1000，price}

Choose the spliting node building URL tree

Information gain, in the Attributions selection of decision tree, have rated the degree of the quantity of information that attribute embodies, if the information gain of attribute is larger, then also will play larger effect to cutting.Therefore, in the process of carrying out Attributions selection, usually choose the large attribute of information gain value as cutting foundation.Information Gain Method of comparing is partial to the attribute being syncopated as a lot of class, and using information gain-ratio to be used as segmentation criterion to construct decision tree here, is equally also select the maximum attribute of information gain-ratio as cutting node.

For key word K and classification C, information gain IG is defined as:

IG(k，C)＝H(C)-H(C|k)

Wherein, H represents average information.

$H\left(C\right)=-\underset{i}{\mathrm{\Σ}}p\left(c_i\right)\log p\left(c_i\right)$

$H\left(C\right|k)=-\underset{v}{\mathrm{\Σ}}p(k=v)\underset{i}{\mathrm{\Σ}}p\left(c_i\right|k=v)\log p\left(c_i\right|k=v)$

Information gain-ratio Gain Ratio is then defined as:

$\mathrm{GR}(k,C)=\frac{\mathrm{IG}(k,C)}{H\left(k\right)}$

Wherein,

$H\left(k\right)=-\underset{v}{\mathrm{\Σ}}p(k=v)\log p(k=v)$

Consider that the accessed frequency of the different page can differ larger, the error rate of the webpage that access frequency is different should be made low, so introducing the access frequency new probability formula defined in above formula is following form:

$p\left(c_i\right)=\frac{{\mathrm{\Σ}}_{u\∈c_i}\mathrm{Imp}\left(u\right)}{{\mathrm{\Σ}}_{u^{\′}\∈U}\mathrm{Imp}\left(u\right)}$

$p(k=v)=\frac{{\mathrm{\Σ}}_{u\∋(u\left(k\right)=v)}\mathrm{Imp}\left(u\right)}{{\mathrm{\Σ}}_{u^{\′}\∈U}\mathrm{Imp}\left(u\right)}$

Here Imp (u) represents access frequency or the display of URL u in training set.Consider that again the impact of access is in the recent period greater than impact in the past.The calculating of Imp (u) can be expressed as

Imp ^d(u)＝(1-α)Imp ^d-1(u)+αfreq ^d(u)

Wherein, α represents smoothing factor, between value 0-1.Imp ^du () is by the access frequency freq of d day ^dthe Imp of (u) and the previous day ^d-1u () calculates and obtains.The data of Imp (u) obtain by the web-page requests recording (90 days) in a period of time continuously.

Stop segmentation URL tree node

When URL all under node belongs to same class, or the access times of URL that contain of node are less than the threshold value that presets and the statistical significance of both candidate nodes segmentation is little, or do not have attribute can be used further to segmentation, then split stopping.

Chi is used to quantize the deviation between the result that expected result and node allocation S obtain at this.

$x^2=\underset{i}{\mathrm{\Σ}}\underset{j}{\mathrm{\Σ}}\frac{{(O_{\mathrm{ij}}-E_{\mathrm{ij}})}^2}{E_{\mathrm{ij}}}$

Chi-square value is more large more relevant, if the significantly segmentation of a node fails to exceed selected confidence level threshold, then splits stopping.Here O _ijthe access times of the URL of class i are belonged to, E under representing node j _ijbe then expectation value, formula is as follows:

$E_{\mathrm{ij}}=\frac{{\mathrm{\Σ}}_j{O}_{\mathrm{ij}}{\mathrm{\Σ}}_i{O}_{\mathrm{ij}}}{{\mathrm{\Σ}}_j{\mathrm{\Σ}}_i{O}_{\mathrm{ij}}}$

When choosing the attribute of maximum information ratio of profit increase value, calculating its corresponding chi-square value, exceeding threshold value if fail, segmentation stops, otherwise continues segmentation.

The beta pruning of URL tree

After generating the URL tree of website, cut operator (if classification error rate own does not comparatively carry out beta pruning in higher position) is carried out to tree, here pessimistic wrong beta pruning PEP method is adopted, it adopts top-down method order traversal to set, divide sample number to judge whether beta pruning by the mistake before comparing beta pruning and after beta pruning, both generate URL tree with training set and also carry out beta pruning with it.To each internal node t, when meeting following formula, cutting off with t is the subtree T of root node _tone of generation leaf node, the classification that leaf node identifies is determined by most of principle.

e′(t)≤e′(T _t)+SE(e′(T _t))

Wherein,

E ' (t)=e (t)+1/2 (represent and divide sample number to the mistake after node t carries out beta pruning)

(representing that the mistake of non-beta pruning divides sample number)

SE (e ' (T _t))=[e ' (T _t) (n (t)-e ' (T _t))/n (t)] ^1/2(represent subtree T _tstandard error)

E (t) represents node t place error (classification error number)

I is for covering subtree T _tleaf node;

N (t) is had sample number by node t place;

The incremental learning that URL Classification and Identification and URL set

When if new URL judges online coupling less than, then calculate the similarity in URL and path in tree, in most Similar Track, choose the node setting up branch, then crawl after related web page classification according to preceding method structure subtree.

Process: according to this URL of sign URL method cutting when contributing, finds the URL tree of coupling root node, carrying out K-V to mating, finding out identical K-V couple with path in tree, then this URL equals with the similarity of certain paths in tree

Wherein,

represent url u _iwith path p _iidentical sign K-V is to number;

represent path p _isign K-V to total number;

If value equals 1, be mate to return the classification of this route classification as this URL completely, otherwise obtain the path that URL similarity is the highest therewith, return corresponding classified information, and crawl the webpage of analogous tissue's structure (K-V is to identical) URL, according to preceding method this path therewith URL have identical K but different V near the node of root node under set up branch.

Such as set up a web site www.rocawear.com URL Classification and Identification tree, build URL Classification and Identification tree after part branch data as shown in Figure 2.The flow process of the URL Classification and Identification that sets up a web site under line tree comprises: the URL in training corpus carries out characterization, is characterized by TOKEN sequence, sets up URL tree (comprising tree node choose and stop spliting node), the Pruning Away Branches of URL tree.On line, under URL Classification and Identification and line, incremental learning flow process comprises: treat point URL and carry out characterization, be characterized by TOKEN sequence, searches coupling path in URL tree, if existed, return classification results, otherwise return approximate possible outcome, again similar URL is collected, for tree adds shoot under line.

Tree-model after building up (only launches the branch under demonstration men node here) as shown in Figure 3, following 8 URL is removed Match Tree according to its tokens, can correspond to fullpath, return the classification of respective path mark.

www.rocawear.com/landingpages/landing.php？dept＝men，1

www.rocawear.com/nshop/product.php？view＝listing&groupName＝wjeans&dept＝women&both＝yes，2

www.rocawear.com/nshop/product.php？view＝listing&groupName＝mshorts&dept＝men&both＝yes，3

www.rocawear.com/nshop/product.php？view＝listing&groupName＝mjeans&dept＝men&both＝yes，3

www.rocawear.com/nshop/product.php？view＝detail&productid＝RW-R0609J55&groupName＝mshorts&dept＝men，4

www.rocawear.com/nshop/product.php？view＝detail&productid＝RW-R00j225&groupName＝mjeans&dept＝men，5

www.rocawear.com/nshop/product.php？view＝detail&productid＝RW-R0909J03&groupName＝mjeans&dept＝men，5

www.rocawear.com/nshop/product.php？view＝detail&productid＝RW-R0809J17&groupName＝mjeans&dept＝men，5

Those skilled in the art, under the condition not departing from the spirit and scope of the present invention that claims are determined, can also carry out various amendment to above content.Therefore scope of the present invention is not limited in above explanation, but determined by the scope of claims.

Claims (5)

1. based on a classifying identification method of URL, it is characterized in that, comprising:

Step 1, utilizes sorter to classify to the website and webpage throwing in advertisement, obtains the classification of the corresponding webpage of all URL in this website;

Step 2, generates the URL Classification and Identification tree of this website according to all URL of this website;

Step 3, mates described URL Classification and Identification tree according to the URL of ad-request, returns matching result;

Step 2 comprises:

Step 21, carries out cutting to URL and obtains its characterization value;

Step 22, according to calculate key word k corresponding to the maximum information ratio of profit increase value of gained as the next node generating URL Classification and Identification tree, wherein:

IG(k,C)＝H(C)-H(C|k)；

$H\left(C\right)=-\underset{i}{\mathrm{\Σ}}p\left(c_i\right)\log p\left(c_i\right);$

$H\left(C\right|k)=-\underset{v}{\mathrm{\Σ}}p(k=v)\underset{i}{\mathrm{\Σ}}p\left(c_i\right|k=v)\log p\left(c_i\right|k=v);$

$H\left(k\right)=-\underset{v}{\mathrm{\Σ}}p(k=v)\log p(k=v);$

$p\left(c_i\right)=\frac{{\mathrm{\Σ}}_{u\∈c_i}\mathrm{Imp}\left(u\right)}{{\mathrm{\Σ}}_{u^{\′}\∈U}\mathrm{Imp}\left(u^{\′}\right)};$

Imp ^d(u)＝(1-α)Imp ^d-1(u)+αfreq ^d(u)

C represents classification, and Imp (u) is the access frequency of URL u, and α represents smoothing factor, Imp ^du () is by the access frequency freq of particular day d ^dthe Imp of (u) and the previous day ^d-1u () calculates and obtains, c _irepresent i-th classification in C, i=1,2,3 ..., v represents the value that key word k is corresponding, and u (k) represents the value of key word k in URL u, and U represents that URL gathers, and u' represents that URL gathers the example in U;

Step 23, when under node, URL belongs to same class, or the access times of URL that contain of node are less than the threshold value that presets and both candidate nodes segmentation does not have statistical significance, or do not have attribute can be used further to segmentation, then split stopping.

2. classifying identification method as claimed in claim 1, is characterized in that, in step 23, the chi-square value of computing node segmentation, exceedes threshold value if fail, and segmentation stops, otherwise continues segmentation.

3. classifying identification method as claimed in claim 1, it is characterized in that, step 2 also comprises step 24, adopts pessimistic wrong pruning method to carry out beta pruning to URL Classification and Identification tree.

4. classifying identification method as claimed in claim 1, is characterized in that, in step 3, if there is not the path of mating completely with the URL of ad-request, according to formula in URL Classification and Identification tree calculate URL and the similarity in path in tree, return the classification of classified information as this ad-request URL in the highest path of Similarity value, wherein, represent urlu _iwith path p _jidentical sign K-V is to number; represent path p _jsign K-V to total number; K represents the key attribute in URL, and V represents the value of key attribute.

5. classifying identification method as claimed in claim 1, it is characterized in that, if there is not the path of mating completely with the URL of ad-request in URL Classification and Identification tree, the web data also crawling this URL institutional framework similar carries out parsing classification, and is added in URL Classification and Identification tree as subtree by achievement step by these URL.