CN101211339A - Intelligent web page classifier based on user behaviors - Google Patents

Abstract

An intelligent web page classification device based on the user's behaviors: (1) Perform background input with an initial classification sample group for training, so as to gain a clustering center of each classification in the characteristic space. (2) Receive a URL input by the user input before catching and analyzing corresponding pages on the background; then, output texts with index value in the page. Moreover, extract a characteristic set according to the user-input content and web page contents; then, perform feedback modification for the characteristic space in the initial classification sample group and adjust the characteristic weight factor in the vector space. (3) Use the user-selected classification device to perform automatic classification for the texts in the previous step of the created texts and output results. When the user executes a search, the classification device can automatically determine the classification of each result and perform gradual adjustment for the classification device; the more times the user executes the search, the more accurate the classification of the web page classification device will be, so as to help different users effectively reduce the size of the set of the search result before locating necessary information more accurately.

Description

Intelligent web page classifier based on user behavior

Technical field

The present invention relates to a kind of webpage be carried out the technology of intelligent classification, particularly carried out the intelligence learning of sorter in conjunction with user behavior feature and web page contents.

Background technology

Existing Web page classifying device mainly comprises two big classes: the manual sort.Directory search such as YAHOO is to take artificial mode to classify to being kept in the local data base.Though the classification degree of accuracy is higher, efficient is very low, and renewal speed is slow, and maintenance workload is big.Automatically classification.Replace manually webpage being classified with computer system, mainly comprise two kinds of implementation methods: the sorter of KBE and based on the sorter of statistics.The former mainly relies on linguistic knowledge, needs a large amount of inference rules of establishment as classificating knowledge, and Search Results is very accurate, but realizes quite complexity, and the development cost costliness.The latter ignores the linguistics structure of text, and text is considered as the characteristic item set, utilizes the weighted feature item to constitute vector and carries out text representation, the frequency of utilizing word to occur is weighted text feature, realize fairly simplely, classify accuracy is higher, can satisfy general requirement of using.But having shortcoming is not consider with the user search behavior to carry out interaction, and also is not suitable for the very high inquiry of scientific literature and so on claimed accuracy.

Summary of the invention

In order to overcome the above-mentioned deficiency of existing sorter, on traditional basis based on the vector space model Web page classifying device of adding up, the invention provides a kind of novel intelligent Web page classifying device, this sorter can not only satisfy the accurate classificating requirement with huge scale storage webpage, and can dynamically adjust the preliminary classification sample set in conjunction with user behavior, the classification results of meeting consumers' demand is provided for the search engine foreground.

Intelligent web page classifier based on user behavior:

(1) backstage input preliminary classification sample set is trained, and obtains each and is sorted in cluster centre on the feature space.

(2) receive the URL that the user imports, the backstage is grasped and is also analyzed the corresponding page, the text that has index to be worth in the output page.And according to user input content and web page contents, extract characteristic set, the feature space of preliminary classification sample set is carried out feedback revise, adjust the feature weight value of vector space.

(3) sorter that adopts the user to select is classified automatically to the text that previous step generates, and the output result.

After the user has carried out once search, sorter is judged the classification under every result automatically, and sorter is progressively adjusted, the searching times that the user carries out is many more, the classification of Web page classifying device is just accurate more, thereby help different user effectively to dwindle the set of Search Results, find required information more accurately.This method is applicable to the classification of various language texts, only will adopt the segmenting word method that is different from Romance for Asian language.

Specific embodiment:

In vector space model, text is made a general reference various machine-readable records, with D (Document) expression.Characteristic item T (Term) is meant the basic language unit that appears in the document D and can represent the document content, mainly is to be made of speech or phrase, and text can be D (T with the characteristic item set representations ₁, T ₂..., Tn), wherein Tk is a characteristic item, k ∈ 1,2 ..., N.For containing the text of n characteristic item, give certain weight can for usually each characteristic item and represent its significance level.Be D=D (T1, W1; T2, W2; , Tn, Wn), note by abridging into D=D (W1, W2 ..., Wn), this is become the vector representation of text, and wherein Wk is the weight of Tk, k ∈ 1,2 ..., N.In vector space model, two arbitrary text D _iAnd D _jBetween content degree of correlation Sim (D _i, D _j) cosine value of angle represents that formula is between the vector commonly used:

$\mathrm{Sim}(\mathrm{Di},\mathrm{Dj})=\cos \mathrm{\θ}=\frac{\underset{k=1}{\overset{n}{\mathrm{\Σ}}}{W}_{\mathrm{ik}}\×W_{\mathrm{jk}}}{\sqrt{\left(\underset{k=1}{\overset{n}{\mathrm{\Σ}}}{W}_{\mathrm{ik}}^2\right)\left(\underset{k=1}{\overset{n}{\mathrm{\Σ}}}{W}_{\mathrm{jk}}^2\right)}}$

Wherein, W _Ik, W _JkRepresent text D respectively _iAnd D _jThe weights of k characteristic item, k ∈ 1,2 ..., N.If it is Th[i that the threshold value of certain keyword number of times is imported in user's search].

The concrete grammar step is as follows:

(1) selects suitable standard vocabulary.The range of application of combining classification device is selected suitable classed thesaurus, just selects the Dewey Decimal Classification sorted table such as belonging to library's application.Each word in the vocabulary is endowed different serial numbers and Hash functional value.And be the threshold value Th[i of each word designated user input number of times].

(2) set up training sample set.Crawling results in conjunction with new Web Crawler is carried out preliminary analysis to indexed webpage, and setting up training sample set is matrix O[i j]=| W[i j] |, i ∈ 1,2 ..., M, j ∈ 1,2 ..., N.Wherein, each row eigenwert comprises classification number, the initial weight value of a lot of keyword and synon initial weight value.Not corresponding with the standard vocabulary or can't give weighted value the time, the weighted value of these ranks of training sample set is 0, and all weighted values of matrix can not be negative when certain row of certain row.

(3) web page characteristics is extracted and weighting.Indexed webpage is carried out automatic indexing, to the words and phrases in the webpage according to they word frequency and webpage in the position that occurs give weight, obtain the primitive character set D[i j of webpage].If D[i j]=0, then withdraw from; Otherwise changeed for (4) step;

(4) Web page classifying.Then obtain each web page characteristics matrix and training sample set matrix each the row between content degree of correlation Sim (Di, Oj), (Sim (Di, Oj)) then is divided into i webpage in the j class of initial training collection relatively to obtain smallest match value Min.If all webpages are all classified to finish, then withdraw from.

(6) if the user searches for, the searching times of input keyword is less than pre-set threshold Th[i], then changeed for (7) step; Otherwise the combined standard vocabulary is obtained the keyword weighted mean value of all user's inputs, with this weighted value that is worth the keyword of the correspondence position of replacing initial training sample set matrix, obtains new training sample set matrix O[i j], changeed for (2) step.

(7) if the user no longer imports keyword search, then the backstage sorter does not respond this user's searching request, and the user obtains satisfied Search Results, and the training of sample set is stopped.

Claims (2)

1. the intelligent web page classifier based on user behavior is characterized in that,

(1) backstage input preliminary classification sample set is trained, and obtains each and is sorted in cluster centre on the feature space;

(2) receive the URL that the user imports, the backstage is grasped and is also analyzed the corresponding page, the text that has index to be worth in the output page; And according to user input content and web page contents, extract characteristic set, the feature space of preliminary classification sample set is carried out feedback revise, adjust the feature weight value of vector space;

(3) sorter that adopts the user to select is classified automatically to the text that previous step generates, and the output result.

2. according to the described intelligent web page classifier of claim 1, it is characterized in that based on user behavior,

The concrete grammar step is as follows:

(1) selects suitable standard vocabulary; The range of application of combining classification device is selected suitable classed thesaurus, just selects the Dewey Decimal Classification sorted table such as belonging to library's application; Each word in the vocabulary is endowed different serial numbers and Hash functional value; And be the threshold value Th[i of each word designated user input number of times];

(2) set up training sample set; Crawling results in conjunction with new Web Crawler is carried out preliminary analysis to indexed webpage, and setting up training sample set is matrix O[ij]=| W[ij] |, i ∈ 1,2 ..., M, j ∈ 1,2 ..., N; Wherein, each row eigenwert comprises classification number, the initial weight value of a lot of keyword and synon initial weight value; Not corresponding with the standard vocabulary or can't give weighted value the time, the weighted value of these ranks of training sample set is 0, and all weighted values of matrix can not be negative when certain row of certain row;

(3) web page characteristics is extracted and weighting; Indexed webpage is carried out automatic indexing, to the words and phrases in the webpage according to they word frequency and webpage in the position that occurs give weight, obtain the primitive character set D[ij of webpage]; If D[ij]=0, then withdraw from; Otherwise changeed for (4) step;

(4) Web page classifying; Then obtain each web page characteristics matrix and training sample set matrix each the row between content degree of correlation Sim (Di, Oj), (Sim (Di, Oj)) then is divided into i webpage in the j class of initial training collection relatively to obtain smallest match value Min; If all webpages are all classified to finish, then withdraw from;

(6) if the user searches for, the searching times of input keyword is less than pre-set threshold Th[i], then changeed for (7) step; Otherwise the combined standard vocabulary is obtained the keyword weighted mean value of all user's inputs, with this weighted value that is worth the keyword of the correspondence position of replacing initial training sample set matrix, obtains new training sample set matrix O[ij], changeed for (2) step;

(7) if the user no longer imports keyword search, then the backstage sorter does not respond this user's searching request, and the user obtains satisfied Search Results, and the training of sample set is stopped.