CN102254038A - System and method for analyzing network comment relevance - Google Patents

Abstract

The invention relates to a method for analyzing relevance between subject contents and comment contents thereof, which can be used for qualitatively distinguishing whether the comment contents are spam comments and can also be used for quantitatively analyzing the relevance of the comment contents so as to obtain a specific relevance value between 0 and 1, wherein the numerical value is bigger and the relevance is larger, and the relevance between comments and the article can be analyzed according to the relevance value. The invention has one advantage that two aspects of contents of the similarity between the comments and the subject article and the relevance among the comments are comprehensively considered, thus the relevance is analyzed more accurately. A system in the invention is based on a browser mode, is convenient for analyzing and using, and has a friendly interface.

Description

A kind of system and analytical approach thereof of the phase-split network comment degree of correlation

Technical field

The invention belongs to text processing/data mining field, internet, relate between numerous comment contents of utilizing theme article in the data mining correlation technique phase-split network and proposing at these articles aspects such as the degree of correlation.Specifically comprised and utilized vector space model, probability model and language model carry out at theme article and comment content similarity analysis, and the analysis of correlativity between the comment content itself.

Background technology

The Web2.0 epoch are burgeoning epoch of information, and netizens can free make various comments at diverse network news and blog, and these comment data had reached the scale of a magnanimity in recent years.At these data the research of many data minings aspect has been arranged now, extracted and the emotion analysis, the integration of user comment and abstract etc. as user comment.In numerous research fields, whether current to have a research focus to discern comment exactly relevant with theme, i.e. whether comment belongs to rubbish is commented on, and it helps people's better utilization to comment on resource.In research work at present, this identification generally all only is identification qualitatively: uncorrelated then is the rubbish comment, otherwise is that non-rubbish is commented on.In fact, rubbish comment and non-rubbish comment not significantly boundary before are so this identification qualitatively is often fuzzyyer.In addition, even be all non-rubbish comment, their value also often is not quite similar.So far, these deficiencies are not also considered by the general Study personnel more than.

Current, the review information research work mainly concentrates on the User Perspective that utilizes in natural language processing technique and data mining technology extraction and the summary user comment data, and promptly the suggestion in the comment is excavated (positive or negative) field.Specifically comprise: in user comment, sum up some function of this product and user these functional point of view at a certain product, discern the words and phrases that can reflect User Perspective in each comment, the user's emotion that comprises of discerning each comment is positive or passiveness etc.Aspect research comment presenter's individual character and behavior, also at the startup at present and obtain certain achievement in research, similarly also relevant for the research on the comment content reliability.

But in rubbish comment identification still is blank out basically, existingly now also only rests on the kind of the variety of issue of describing the rubbish comment and analysis and the comment of fixed rubbish about the research work in the rubbish comment identification on a small quantity.

Summary of the invention

The present invention is directed to the present situation that lacks review information degree of correlation analysis tool in the existing internet, the degree of correlation analytic system of a kind of network comment and its theme is provided.

For solving the problems of the technologies described above, the degree of correlation analytic system of network comment of the present invention and its theme is characterized in that, comprises webcrawler module, degree of correlation analysis module, and the web page display module,

Described webcrawler module is used for the content of text of intercepting page, generates the data acquisition of theme article and some related commentary, the described degree of correlation analysis module analyzing and processing of described data acquisition confession;

Described degree of correlation analysis module is used for the degree of correlation between quantitative Analysis network comment and the theme article;

Described web page display module is used for network comment degree of correlation result that degree of correlation analysis module is calculated, shows with form web page output.

Described degree of correlation analysis module comprises:

First device, being used for the overall network comment is node, generates non-directed graph;

Second device is used to calculate the similarity between certain network comment and the theme article;

The 3rd device is used to calculate the similarity that the described network comment of second device is adjacent the network comment that node refers to;

The 4th device, be used for the network comment that calculates according to second device and the similarity between the theme article, and the 3rd the network comment that calculates of device be adjacent the similarity of the network comment that node refers to, calculate the degree of correlation between this network comment and the theme article.

Described degree of correlation analysis module also comprises:

Step device is used to select the next network comment of not making relatedness computation, does not if there is no do the network comment of relatedness computation, then returns null value;

Call control device, be used for the selected network comment of described step device as input, and judge whether to be input as null value,

If non-NULL, then call described second, third, the 4th device, calculate the degree of correlation between current network comment and the theme article; Turn back to step device;

If be null value, then stop.

Also comprise degree of correlation judge module, be used for certain network comment and the degree of correlation between the theme article and the size between the preset threshold that more described degree of correlation analysis module calculates; When the described degree of correlation during less than preset threshold, then described web page display module show this network comment for and the irrelevant comment of theme article.

The present invention has proposed a kind of analytical approach of system of the above-mentioned phase-split network comment degree of correlation simultaneously, it is characterized in that, may further comprise the steps:

The content of text of described webcrawler module intercepting page, the data acquisition of generation theme article and some related commentary, described data acquisition is issued described degree of correlation analysis module;

The degree of correlation between described degree of correlation analysis module quantitative Analysis network comment and the theme article;

The network comment degree of correlation result that described web page display module calculates degree of correlation analysis module shows with form web page output.

The degree of correlation between described degree of correlation analysis module quantitative Analysis network comment and the theme article may further comprise the steps:

Step 5-1, calculate the similarity between certain network comment and the theme article;

Step 5-2, be node, generate non-directed graph with the overall network comment;

Step 5-3, calculate the similarity that described network comment is adjacent the network comment that node refers to;

Step 5-4, according to the similarity between this network comment and the theme article, and network comment is adjacent the similarity of the network comment that node refers to, and calculates the degree of correlation between this network comment and the theme article.

The degree of correlation between described degree of correlation analysis module quantitative Analysis network comment and the theme article, further comprising the steps of:

The network comment of relatedness computation is not if there is no done in step 6-1, the next network comment of not making relatedness computation of selection, then returns null value;

Step 6-2, with the selected network comment of described step device as input, and judge whether to be input as null value,

If non-NULL then returns step 5-2,5-3,5-4; Then, turn back to step 6-1;

If be null value, then stop.

Further comprising the steps of:

Certain network comment that calculates according to described degree of correlation analysis module and the degree of correlation between the theme article, when the described degree of correlation during less than preset threshold, then described web page display module show this network comment for and the irrelevant comment of theme article.

The present invention analyzes and draws a concrete relevance degree between 0 to 1 by comment content degree of correlation is carried out quantitative analysis, and the big more then degree of correlation of numerical value is big more, can analyze close and distant relation between comment and this article according to relevance degree.A remarkable advantage of the present invention is that core analysis has partly been taken all factors into consideration the similarity between comment and the theme article, and two the aspect contents of correlativity between the comment, so the analysis of the degree of correlation is more accurate.System of the present invention is based on browser model, analyze easy to use, friendly interface.

Description of drawings

Below in conjunction with the drawings and specific embodiments technical scheme of the present invention is further described in detail.

Fig. 1 is the structured flowchart of the system of the phase-split network comment degree of correlation of the present invention.

Fig. 2 is comment network node figure of the present invention.

Embodiment

The structured flowchart of system as shown in Figure 1, the system of the phase-split network comment degree of correlation of the present invention comprises webcrawler module, degree of correlation analysis module, the web page display module,

Described webcrawler module is used for the content of text of intercepting page, generates the data acquisition of theme article and some related commentary, the described degree of correlation analysis module analyzing and processing of described data acquisition confession;

Described degree of correlation analysis module is used for the degree of correlation between quantitative Analysis network comment and the theme article; The present invention is from the degree of correlation of quantitative angle analysis comment with article, and the degree of correlation here is a linear variable, can be numerical value arbitrarily from 0 to 1 the range of linearity.The degree of correlation that different comment contents calculates tends to difference: the relevance of high more then this comment content of numerical value and theme article is strong more, can clearly represent viewpoint and the view of reviewer for the article theme more; Otherwise the low more practical function that then should comment on of numerical value is worth more little.According to the difference of actual demand, when the degree of correlation is lower than some setting threshold values, can think that this comment is the rubbish comment.Analyzing the comment content with the process of the theme article degree of correlation is: extract (the comment 1 of the theme article A comment content corresponding with it, comment 2, comment 3, comment n), calculate A and the similarity K that comments on 1-n, calculate degree of correlation L between comment and the comment, comprehensive K and L add up by certain weight and draw the degree of correlation P of A and all comments _i(i can be 1 to n between arbitrary integer).P _iBe the relevance degree that finally draws.Can a given degree of correlation threshold values by actual demand, if being lower than this threshold values, any degree of correlation can be judged to be useless comment, P _iHighly more reflect that then this comment is relevant more with theme.This degree of correlation analysis module has different characteristics with present all technology: not only with the statement similarity between comment text and the theme article as the factor that influences the degree of correlation, also the internal relation between numerous comment texts is also taken into account simultaneously.Its core concept is: if one section comment content and those analyzed and the theme article have between the comment of the high degree of correlation and have higher similarity, then this section comment should also have the higher degree of correlation with this theme article, and the degree of correlation of this section comment immediately and theme article is not high.

Described web page display module is used for network comment degree of correlation result that degree of correlation analysis module is calculated, shows with form web page output.The web page display module provides a user oriented interface, is mainly used in by customer requirements to show result, comprises that the degree of correlation of all comment content correspondences is showed, the positive backward of the degree of correlation is arranged functions such as displaying.The result of this module after with degree of correlation analysis module analyzing and processing is organized into the intelligible data structure of user, is shown to user interface with the form of webpage.

Holistic approach method of the present invention is: utilize theme and the comment data collection of web crawlers acquisition module collection at the particular webpage content by demand, afterwards this data set is submitted to degree of correlation analysis module and carries out degree of correlation analysis, at last analysis result is delivered to the web page display module and on web browser, plays frame displaying analysis result by the actual functional capability demand.

Webcrawler module mainly includes but not limited to the website selection mainly based on general web crawlers technique construction, and content of text is selected, and data grasp, the back-end data administrative section.Webcrawler module is an independently necessary preposition module, be mainly used in the content of text intercepting of the webpage that the user browses, and the theme body matter in these texts extracted with the form of comment content by 1:N (corresponding many of one section theme text is commented on contents), be organized into certain data set again for subsequent analysis, as: theme: XXXXX---comment 1:XXX, the form of comment 2:XXX comment 3:XXX.

Degree of correlation analysis module is a nucleus module of the present invention.This module realizes the degree of correlation between the theme article and review information in the automatic phase-split network information, and relevance degree high more then reflection between this comment and the theme text agrees with more, on the contrary then for irrelevant comment, as advertising message, occupy-place information etc.This module mainly is divided into two parts: theme-comment and analysis part, comment-comment and analysis part.Wherein theme-comment and analysis part is mainly considered the degree of correlation between theme text and the review information, and criterion is mainly the vocabulary similarity, aspects such as text repetition rate.Comment-comment part is mainly then mainly analyzed the calculating of the degree of correlation of similarity between all review information.The analysis result of comprehensive two parts can draw final degree of correlation analysis result.

The core analysis model generalization of the present invention's representative has been considered following two factors: similarity between comment and main body article, the internal relation between the comment will provide the implementation procedure of analytical calculation below in order.

(1) calculates similarity between comment and the quilt comment theme

The present invention adopts probabilistic language model to calculate similarity between comment and main body article, and for any one comment R and quilt comment theme article A, definition Sim (R|A) is the similarity between R and the A, can obtain from following formula:

$\mathrm{Sim}(R,A)\≈P\left(R\right|A)=\underset{i=1}{\overset{n}{\mathrm{\Π}}}P\left(q_i\right|A)=\underset{w\∈R}{\mathrm{\Π}}P{\left(w\right|A)}^{c(w,R)}$ Formula (1)

Wherein P (R|A) represents the probability from R to A, and w is an occurring words among the R, c (w, the R) number of times of representing w to occur in R, the frequency probability that on behalf of w, P (w|A) occur in A.

Available maximum likelihood estimate (MLE) calculates P (w|A):

$P\left(w\right|A)=P_{\mathrm{ML}}\left(w\right|A)=\frac{c(w,A)}{\left|A\right|}$

Wherein | A| is all word sums that occur among the A.This method has certain defective, shows that mainly if word w does not have the explicit A of appearing at, then P (w|A) directly can get null value.Do not have fully under the situation of identical word at R and A, the similarity of R and A will be judged as zero.

For the consideration that solves the null value problem, the present invention adopts a kind of improved method: the Jelinek-Mercer smoothing method, and as a typical linear interpolation smoothing method, computing method are as follows:

P(w|A)＝λP _ML(w|A)+(1-λ)P(w|C)

The probability that occurs in corpus C for word w of P (w|C) wherein, λ is a smoothing factor.As preferably, the value that the present invention gets its λ is 0.2.

Be the potential error of avoiding producing because of comment length, the present invention also introduces a length normalization method method and comes the standard original probability:

$P_{\mathrm{norm}}\left(\mathrm{Sim}(R,A)\right)\∞\exp \left(\frac{\log \mathrm{Sim}(R,A)}{\mathrm{len}\left(R\right)}\right)$

Wherein len (R) is the length of R, the total speech number that promptly comprises among the R.

(2) the overall calculation comment and the article degree of correlation

The present invention will be considered as a kind of network node figure at the set of all comments of concrete theme article, i.e. the set of all comments is modeled as a graphic structure, as shown in Figure 2:

By calculating cosine similarity between the comment non-directed graph between can obtaining commenting on, each node represent and is aly commented on the cosine similarity between the weight representation node of internodal line among this figure.Node is successively by subscript mark R ₁, R ₂, R ₃, R ₄, R ₅..., R _nKnow.

Based on above setting, can think that each node contains a correlation, this value can influence the correlation of node on every side of this node, gets node R among the figure arbitrarily _i, R _i∈ (R ₁, R _n), following formula is arranged:

$\mathrm{Pertinence}\left(R_i\right)=\underset{R_j\∈\mathrm{adj}\left[R_i\right]}{\mathrm{\Σ}}\frac{w(R_j,R_i)}{{\mathrm{\Σ}}_{R_k\∈\mathrm{adj}\left[R_j\right]}w(R_i,R_k)}\mathrm{Pertinence}\left(R_j\right)$ Formula (2)

Pertinence (R _i) representative comment R _iAnd the degree of correlation between the theme article A, adj[R _i] represent all and comment on R _iThe set of adjacent comment node, R _jBe set adj[R _i] middle comment.W (R _j, R _i) for commenting on R _jWith comment R _iBetween similarity.

Above-mentioned formula is only considered the influence that concerns between the comment, the formula (1) of comprehensive step 1 and the formula (2) of step 2, and last overall calculation comment is as follows with the formula of the theme article degree of correlation:

$\mathrm{Pertinence}\left(R_i\right)=d\×\frac{\mathrm{sim}(R_i,A)}{{\mathrm{\Σ}}_R\mathrm{sim}(R,A)}+(1-d)\left[\underset{R_j\∈\mathrm{adj}\left[R_i\right]}{\mathrm{\Σ}}\frac{w(R_j,R_i)}{{\mathrm{\Σ}}_{k\∈\mathrm{adj}\left[j\right]}w(R_i,R_k)}\mathrm{Pertinence}\left(R_j\right)\right]$

This formula left half depends on formula (1), the right depends on formula (2), the comment relevance degree partly depends on the similarity of commenting between the theme article, partly depend on the association between the comment, so d represents trade-off value between the two in the following formula, any value between can getting 0 to 1 by actual conditions, the native system acquiescence is got d=0.7.

Provide the treatment scheme of review information analyzing and processing component system at last: may further comprise the steps:

The content of text of webcrawler module intercepting page, the data acquisition of generation theme article and some related commentary, described data acquisition is issued described degree of correlation analysis module;

Calculate the similarity between certain network comment and the theme article;

Steps A, be node, generate non-directed graph with the overall network comment;

Calculate the similarity that described network comment is adjacent the network comment that node refers to;

According to the similarity between this network comment and the theme article, and network comment is adjacent the similarity of the network comment that node refers to, and calculates the degree of correlation between this network comment and the theme article.

Select the next network comment of not making relatedness computation, if there is no do not do the network comment of relatedness computation, then return null value;

The selected network comment of described step device as input, and is judged whether to be input as null value,

If non-NULL then returns steps A; If be null value, then stop.

The network comment degree of correlation result that described web page display module calculates degree of correlation analysis module shows with form web page output.

Certain network comment that calculates according to described degree of correlation analysis module and the degree of correlation between the theme article, when the described degree of correlation during less than preset threshold, then described web page display module show this network comment for and the irrelevant comment of theme article.

It should be noted last that, above embodiment is only unrestricted in order to technical scheme of the present invention to be described, although the present invention is had been described in detail with reference to preferred embodiment, those of ordinary skill in the art is to be understood that, can make amendment or be equal to replacement technical scheme of the present invention, and not breaking away from the spirit and scope of technical solution of the present invention, it all should be encompassed in the middle of the claim scope of the present invention.

Claims (8)

1. the system of a phase-split network comment degree of correlation is characterized in that, comprises webcrawler module, degree of correlation analysis module, and the web page display module,

Described webcrawler module is used for the content of text of intercepting page, generates the data acquisition of theme article and some related commentary, the described degree of correlation analysis module analyzing and processing of described data acquisition confession;

Described degree of correlation analysis module is used for the degree of correlation between quantitative Analysis network comment and the theme article;

Described web page display module is used for network comment degree of correlation result that degree of correlation analysis module is calculated, shows with form web page output.

2. the system of the phase-split network comment degree of correlation according to claim 1 is characterized in that described degree of correlation analysis module comprises:

First device, being used for the overall network comment is node, generates non-directed graph;

Second device is used to calculate the similarity between certain network comment and the theme article;

The 3rd device is used to calculate the similarity that the described network comment of second device is adjacent the network comment that node refers to;

The 4th device, be used for the network comment that calculates according to second device and the similarity between the theme article, and the 3rd the network comment that calculates of device be adjacent the similarity of the network comment that node refers to, calculate the degree of correlation between this network comment and the theme article.

3. the system of the phase-split network comment degree of correlation according to claim 2 is characterized in that described degree of correlation analysis module also comprises:

Step device is used to select the next network comment of not making relatedness computation, does not if there is no do the network comment of relatedness computation, then returns null value;

Call control device, be used for the selected network comment of described step device as input, and judge whether to be input as null value,

If non-NULL, then call described second, third, the 4th device, calculate the degree of correlation between current network comment and the theme article; Turn back to step device;

If be null value, then stop.

4. according to the system of the claim 1 or the 2 or 3 described phase-split networks comment degrees of correlation, it is characterized in that, also comprise degree of correlation judge module, be used for certain network comment and the degree of correlation between the theme article and the size between the preset threshold that more described degree of correlation analysis module calculates; When the described degree of correlation during less than preset threshold, then described web page display module show this network comment for and the irrelevant comment of theme article.

5. the analytical approach of the system of the described phase-split network comment of claim 1 degree of correlation is characterized in that, may further comprise the steps:

The content of text of described webcrawler module intercepting page, the data acquisition of generation theme article and some related commentary, described data acquisition is issued described degree of correlation analysis module;

The degree of correlation between described degree of correlation analysis module quantitative Analysis network comment and the theme article;

The network comment degree of correlation result that described web page display module calculates degree of correlation analysis module shows with form web page output.

6. comment on the analytical approach of the system of the degree of correlation according to the described phase-split network of claim 5, it is characterized in that the degree of correlation between described degree of correlation analysis module quantitative Analysis network comment and the theme article may further comprise the steps:

Step 5-1, calculate the similarity between certain network comment and the theme article;

Step 5-2, be node, generate non-directed graph with the overall network comment;

Step 5-3, calculate the similarity that described network comment is adjacent the network comment that node refers to;

Step 5-4, according to the similarity between this network comment and the theme article, and network comment is adjacent the similarity of the network comment that node refers to, and calculates the degree of correlation between this network comment and the theme article.

7. comment on the analytical approach of the system of the degree of correlation according to the described phase-split network of claim 6, it is characterized in that, the degree of correlation between described degree of correlation analysis module quantitative Analysis network comment and the theme article, further comprising the steps of:

The network comment of relatedness computation is not if there is no done in step 6-1, the next network comment of not making relatedness computation of selection, then returns null value;

Step 6-2, with the selected network comment of described step device as input, and judge whether to be input as null value,

If non-NULL then returns step 5-2,5-3,5-4; Then, turn back to step 6-1;

If be null value, then stop.

8. according to the analytical approach of the system of the claim 5 or the 6 or 7 described phase-split networks comment degrees of correlation, it is characterized in that, further comprising the steps of:

Certain network comment that calculates according to described degree of correlation analysis module and the degree of correlation between the theme article, when the described degree of correlation during less than preset threshold, then described web page display module show this network comment for and the irrelevant comment of theme article.

Images