CN101826090A - WEB public opinion trend forecasting method based on optimal model - Google Patents

Abstract

The invention discloses a WEB public opinion trend forecasting method based on optimal model. The basic thought of the method is to firstly classify historical public opinion events to obtain several categories of opinion events, secondly clustering the time series plot of events in each obtained category to obtain sub-categories and finally obtaining the optimal model of each sub-category while ensuring the sum of mean-square errors is least to obtain the optimal model of each main category. When given an event to be predicted, the predicted event is classified and the optimal models obtained by early training to the category of the event are selected for matching, thus the model which is more suitable for the development trend of the event and the change proportion in matching can be selected; and inverse transformation is performed to the selected models according to the obtained change proportion so as to obtain the long-term development trend of the predicted event. Therefore, the defect that the existing network forecasting method can not forecast the inflection point can be overcome, the government and the supervision department can adopt timely and effective measures and the effects of network supervision can be better realized.

Description

WEB public opinion trend forecasting method based on optimization model

Technical field

The present invention relates to the intelligent information forecasting techniques, more specifically, relate to a kind of forecasting techniques of internet public feelings development trend.

Background technology

Network public-opinion

Along with the fast development of internet, the network media has goed deep into daily life as a kind of new information mode of propagation, and the speech active degree of the public on network also reaches unprecedented stage.No matter be domestic or international major event, can both on network, spread at once and cause the public's very big concern and discuss and then produce huge pressure from public opinion warmly, reach the stage that any department and agency all can't be ignored.We can say that the internet has become the distribution centre of ideology and culture information and the amplifier of public opinion.

To be the public that propagates by the internet have strong influence power and obvious tendentious speech and a viewpoint to what some focus, focal issue in the actual life were held to network public-opinion, mainly by BBS forum, blog, news follow-up post, change realization such as subsides and also strengthened.Current society, information is propagated with suggestion unprecedentedly fast alternately, and the expression demand of network public opinion is also polynary day by day, if guiding is not good at, negative network public-opinion will form bigger threat to social public security.Concerning relevant government department; how to strengthen the timely monitoring of network public opinion and effectively guiding; development trend how to predict network public-opinion in advance is actively to dissolve the network public opinion crisis; to maintaining social stability and promoting national development to have important practical significance, also be create harmonious society intension should be arranged.

The pre existing survey technology

Forecasting techniques can be divided into two classes: 1. quantitative analysis, promptly analyze cause-effect relationship to predict based on statistical data and by mathematical tool.Quantitative analysis prediction concrete grammar is a lot, as trend extrapolation method and regression analysis etc.The trend extrapolation method is the time sequence analysis, it is according to historical and existing data speculation development trend, thereby analyze things development in future situation, so-called time series is promptly arranged the incident that occurs under certain condition in chronological order, and passes through the mathematical model prediction future of trend extrapolation; The cause-effect relationship that regression analysis also claims correlation analysis promptly to change from things goes out to send to be predicted, studies the interaction that causes the following various objective factors that change, the method for pointing out statistical relationship between various objective factors and the to-be.2. qualitatively judge, promptly when data can not utilized fully, can only rely on audio-visual materials to rely on the personal experience and analysis ability is carried out logic determines and to making prediction future.

Forecasting Methodology has four kinds of basic types: qualitative forecasting, time series analysis, causal relation method and simulation.

(1) qualitative forecasting: it be based on estimate and estimate therefore belong to subjective judgement, common qualitative forecasting method comprises: general forecast, market study method, panel discussion method, historical analogy, Delphi method etc.

(2) time series analysis method: it is to be based upon the historical data relevant with the past demand to can be used for predicting on the such setting basis of following demand.Historical data may comprise such as factors such as trend, season, cycles, and common Time series analysis method mainly contains: simple moving average, weight moving average, exponential smoothing, regretional analysis, Bao Kesi Charles Jenkins method, Xi Sijin time series etc.This method is simple, is convenient to grasp, though obtained using widely in reality, its poor accuracy generally only is applicable to short-term forecasting.

(3) causal relation method: it is to be based upon on demand such setting basis relevant with the external factor of some internal factor or surrounding environment, and common causal relation method mainly contains: regretional analysis, economic model, input-output model, row index etc.

(4) model: analogy model allows condition work to a certain degree the hypothesis of prognosticator to prediction.

Technical matters

The prediction of every field at present all adopts above-described pre existing survey technology to predict, these forecasting techniquess have certain effect when carrying out short-term forecasting, and being fit to provides analysis and reference to appropriate authority.But when adopting these forecasting techniquess to carry out long-term forecasting, As time goes on the increase of uncertain factor makes that the deviation that predicts the outcome is very big, and the flex point of the discovery trend development that can't in time do sth. in advance, thereby the effect that causes government and supervision department can't take preventive measure timely and effectively to remove better to realize network supervision.So how finding flex point accurately and the secular trend prediction is made in the development of incident becomes a problem demanding prompt solution.

Summary of the invention

The objective of the invention is to overcome the deficiency of above-mentioned existing Forecasting Methodology, provide a kind of and can find flex point more accurately and the secular trend forecast method is made in the development of incident.

For achieving the above object, the present invention includes following steps:

(1), analyze to specify the URL feature of forum to grasp webpage, information document is saved in local data base with relevant data message;

(2), the information document in the local data base is carried out cluster and classification, obtain all kinds of document databases;

(3), from the existing database of all kinds of documents, obtain the time series of each incident desired parameters (as the document size of unit interval etc.) according to event flag and time mark, or download the Google trend time series of each incident correspondence from Google trends website;

(4), at the time series of all kinds of incidents that obtained in the step (3), set up corresponding optimization model and preserve, as the object of predicted object trend coupling;

(5), when new public sentiment incident takes place, at first obtain this incident some time serieses and affiliated big class accordingly to step (3) by step (1).By with its under some optimization modeles that obtain of training in the big class mate, thereby realize long-term forecasting to new public sentiment.

The present invention is by classifying to historical public sentiment incident and setting up all kinds of optimization model collection, after the classification under kainogenesis public sentiment incident on the network has been determined, use the given data of this public sentiment incident and optimization model collection of class mates under it method is determined the long-run development trend of this incident, not only can and can make prediction in the flex point of the initial stage better prediction outgoing event development of incident development like this to the secular trend of incident development, the deficiency that has so not only remedied the pre existing survey technology can also make government and supervision department adopt measure timely and effectively, better realizes the effect of network supervision.

In order to further specify principle of the present invention and characteristic, the present invention is described in detail below in conjunction with the drawings and specific embodiments.

Description of drawings

Fig. 1 is the overall flow figure that the present invention is based on the WEB public opinion trend forecasting of optimization model;

Fig. 2 is the process flow diagram that step ST4 shown in Figure 1 asks optimization model;

Fig. 3 is the process flow diagram in the cycle that cuts among the step ST2 shown in Figure 2;

Fig. 4 is the process flow diagram of among the step ST5 shown in Figure 1 new public sentiment incident secular trend being predicted;

Fig. 5 connects the broken line graph that break generates in the instantiation part steps 4.21;

Fig. 6 is the experiment effect figure in the cycle that cuts in the instantiation part steps 4.22;

Fig. 7 is the experiment effect figure of cluster result in the instantiation part steps 4.4;

Fig. 8 is the experiment effect figure that sets up optimization model in the instantiation part steps 4.5;

Fig. 9 is the prediction effect figure of swine flu incident in the instantiation part steps 5.

Embodiment

Below the specific embodiment of the present invention is described, overall flow of the present invention can be referring to Fig. 1, detailed content corresponding step 1 described below respectively arrives step 5, what need special prompting is, in the following description, when the detailed description that adopts known function and design can be desalinated main contents of the present invention, these descriptions will be left in the basket.

Step 1: analyze the URL feature of specifying forum and grasp webpage, information document is saved in local data base with relevant data message, the ST1 in this step corresponding diagram 1;

By analyzing the URL feature of specifying forum, filter useless link and the repeated links such as advertisement that does not have this feature during extraction, the structure of web page of analyzing web page extracts the information document of replying and quote number of times, text between the theme numbering in the webpage, the money order receipt to be signed and returned to the sender Customs Assigned Number of respectively posting, user mutually respectively and deposits local data base.

Step 2: the information document in the local data base is carried out cluster and classification, the ST2 in this step corresponding diagram 1, purpose is to be classified as several big classes for the information document with public sentiment;

At first the information document that will describe same incident by the method for cluster is put together, and work is gone up the events corresponding mark, concrete clustering method can be referring to list of references 1 (list of references 1: the application of RBFN model in load prediction of means clustering algorithm step by step, Liu Xiaohua, Liu Pei, Zhang Buhan, ten thousand Jianpings, " Proceedings of the CSEE ") in technology.Then according to the characteristics of public sentiment, information document is divided into several big classes such as criminal case, the attack of terrorism, economic security, disaster, accident disaster, public health event and social security events by classification, concrete sorting technique can adopt existing sorting technique, such as list of references 2 (list of references 2: sorting technique summary in the data mining, Qian Xiaodong, " letter of credence intelligence work ", the 51st the 3rd phase of volume, in March, 2007).

Step 3: from the existing database of all kinds of documents, obtain the time series of each incident desired parameters (as document size of unit interval etc.) according to event flag and time mark, or download the Google trend time series of each incident correspondence, the ST3 in this step corresponding diagram 1 from Google trends website;

Step 4: at the time series of each the big class incident that is obtained in the step 3, set up corresponding optimization model and preservation, as the object of predicted object trend coupling, the ST4 in this step corresponding diagram 1, detailed flow process can be referring to Fig. 2;

Step 401: the event set to class S is { S ₁, S ₂..., S _n, the data acquisition of each incident is { Y _I1, Y _I2..., Y _ImCorresponding curve carries out smoothing processing, and concrete grammar is the one dimension median filtering method, formula is as follows:

I represents the incident that these data are affiliated in the formula (1), and j represents j data of this incident, and median represents to get

Arrive

Intermediate value,

The following integer of m/2 is got in expression.

Step 402: every curve is cut period treatment, cycle of outgoing event development trend, the idiographic flow that cuts the cycle can be referring to Fig. 3:

Step 4021: the traversal primary curve, keep those tangible turning points, link up the formation broken line graph with these turning points of the bundle of lines;

Select the specific practice of these turning points to be: the point of beginning and ending at first is chosen as key point, we are since a key point then, trial connects each point of it and its back with straight line, when a bit surpassing specified value d with the distance of this straight line up to the centre, that off-limits point just is considered to a new key point.Next from this new key point, the process above repeating is up to last point of curve.

Step 4022: seek the position that each cycle begins and finishes on broken line graph, traversal can be avoided the irrelevant interference that rises and falls on broken line graph;

Step 40221: confirm the beginning in cycle

Criterion: when the slope of one section straight line surpasses artificial given threshold value (we are taken as 3 in as instantiation), begin with regard to determination cycles.

Step 40222: confirm the end in cycle

Criterion: after the cycle begins, satisfy one of following two conditions and just judge end cycle:

1, the fluctuating of trend is in a given critical field d, given d when promptly selecting turning point, can suitably adjust in light of the circumstances, and thisly steadily continue a given time span minT at least, the current height of curve should not be higher than cycle 2 times when beginning simultaneously;

2, the length in cycle has surpassed given maxT to greatest extent.

Step 4023: the beginning and the end position in the cycle that obtains according to step 4022 cut the cycle.

Step 403: the cycle that cuts out is carried out the standardization processing of time span and mxm., do not change curve shape;

Need guarantee to measure conforming principle length cycle length of all curves is unified standardization processing is maxT according to setting up data warehouse, at this moment need to carry out the processing of interpolation, to zoom to maxT through the mxm. max of every after interpolation processing curve then, and maxT/max adjusts that all the other put pairing value on the curve in proportion;

Concrete interpolation method illustrates: suppose to cut all after dates and obtain sequence c cycle length, obtain the length l en (c) in this cycle, this length of a curve standard is turned to maxT, through obtaining time series z (z after formula (2) and formula (3) calculating ₁, z ₂..., Z _MaxT).

q＝i*len(c)/maxT(1≤i≤maxT) (2)

Step 404: to the curve for layered cluster after the standardization processing, select means clustering algorithm step by step, this algorithm has solved the local optimum problem of K average, also solved the clusters number problem, the list of references 1 that concrete means clustering algorithm step by step can be mentioned referring to step 2, purpose is in order to obtain to ask the group of optimization model;

Step 405: each group that obtains after the cluster is obtained its optimization model, and this model need guarantee square error and the minimum with such all curves;

Concrete grammar is: the event set of class S is { S ₁, S ₂..., S _n, the data acquisition of each incident is { y _I1, y _I2..., y _Im, 1≤i≤n wherein.Formula (4) is the optimization model of our necessary requirement, wherein x _Ij=j express time mark is [3,20] according to the span of repeatedly testing k, can therefrom choose as the case may be.Formula (5) is the formula of square error, and we regard it as (a ₀, a ₁A _k) the multivariate function, ask the method for extreme value according to the multivariate function, at first to y ' in (5) _IjIn (a ₀, a ₁A _k) carry out differentiate and equal zero and obtain Linear Equations (6), can obtain all stationary point (a by separating this Linear Equations ₀, a ₁A _k), compare mutually with maximal value and minimum value on the boundary value, the pairing stationary point of minimum value be ask coefficient in the optimization model.

$y_{\mathrm{ij}}^{\′}=a_0+a_1{x}_{\mathrm{ij}}^1+a_2{x}_{\mathrm{ij}}^2+...+a_k{x}_{\mathrm{ij}}^k(1\≤i\≤n,1\≤j\≤m)---\left(4\right)$

$e=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}\underset{j=1}{\overset{m}{\mathrm{\Σ}}}{(y_{\mathrm{ij}}^{\′}-y_{\mathrm{ij}})}^2---\left(5\right)$

$\left\{\begin{array}{c}\underset{i=1}{\overset{n}{\mathrm{\Σ}}}\underset{j=1}{\overset{m}{\mathrm{\Σ}}}(a_0+a_1{x}_{\mathrm{ij}}^1+a_2{x}_{\mathrm{ij}}^2+...+a_k{x}_{\mathrm{ij}}^k-y_{\mathrm{ij}})x_{\mathrm{ij}}^0=0\\ \underset{i=1}{\overset{n}{\mathrm{\Σ}}}\underset{j=1}{\overset{m}{\mathrm{\Σ}}}(a_0+a_1{x}_{\mathrm{ij}}^1+a_2{x}_{\mathrm{ij}}^2+...+a_k{x}_{\mathrm{ij}}^k-y_{\mathrm{ij}})x_{\mathrm{ij}}^1=0\\ ......\\ \underset{i=1}{\overset{n}{\mathrm{\Σ}}}\underset{j=1}{\overset{m}{\mathrm{\Σ}}}(a_0+a_1{x}_{\mathrm{ij}}^1+a_2{x}_{\mathrm{ij}}^2+...+a_k{x}_{\mathrm{ij}}^k-y_{\mathrm{ij}})x_{\mathrm{ij}}^k=0\end{array}\right.---\left(6\right)$

Step 5: when new public sentiment incident takes place, at first obtain more corresponding time serieses of this incident and affiliated big class to step 3 by step 1.By with its under some optimization modeles that obtain of training in the big class mate, thereby realize long-term forecasting to new public sentiment, idiographic flow can be referring to Fig. 4.

Step 501: the slope to the time series T of new public sentiment incident is analyzed, if slope more than or equal to threshold value 3 the beginning value predict;

Step 502: be 0.1 to stretch or compressed transform to the curve map horizontal ordinate of time series T correspondence and ordinate with mode from 1 to 100 step-length of traversal respectively, from existing optimization model, take out one with conversion after test data set square error and minimum model S, and keep the horizontal ordinate and the ordinate conversion ratio k of test data set at this moment ₁And k ₂

Step 503: with time series T process k ₁And k ₂Change the curve T ' obtain substitute with S in the curve of the same length of mating with it obtain changing after the model S ' of coupling;

Step 504:, horizontal ordinate and the ordinate of S ' need be pressed 1/k respectively for the prediction curve of the public sentiment incident that obtains newly arriving ₁And 1/k ₂Carry out inverse transformation and obtain the long-term forecasting curve

Step 505: when the given data of the public sentiment incident of newly arriving increased, repeated execution of steps 501 arrived step 504, thereby obtains new long-term forecasting curve.

Instantiation

For further understanding the method for WEB public opinion trend forecasting of the present invention, instantiation of act below us:

Step 1: the URL feature of the forum of several main stream website such as analysis Sina, Netease, Sohu, cat pounce on, Google also grasps webpage, and information document and relevant data message are saved in local data base;

Step 2: after these information documents being carried out cluster and classification, choose the document sets that belongs to the public health class from these information documents, these public health event document sets comprise the document of incidents such as hand-foot-and-mouth disease, bird flu, measles, cholera, the big tiny incident of Guangyuan tangerine, false milk powder;

Step 3: to the time series of each event establishment desired parameters that belongs to the public health class chosen, in the present embodiment part, we adopt the time series of the Google trends that relatively has authority to verify our method;

Step 4: at the time series of each incident that is obtained in the step 3, set up corresponding optimization model and preservation, as the object of predicted object trend coupling;

Step 4.1: each time series is carried out the one dimension median filter smoothness of image, and r gets 2;

Step 4.2: the time series to the Google trends that obtained is carried out the processing in fetch cycle, because incident is more, so our periodogram that only cuts with the bird flu incident is the effect that example illustrates us here;

Step 4.21: we travel through primary curve, and we get d=10 here, keep those and have tangible turning point, link up with these turning points of the bundle of lines again.As shown in Figure 5 in the accompanying drawing, red curve for traversal after the curve that obtains, red circle is turning point that we obtained;

Step 4.22: because the interference that red curve has been removed some minor swings, so we seek the beginning in cycle and the position of end on red trend map, criterion can be referring to 40222 of concrete implementation step, at our data d=10, minT=7, maxT=120.Be the time period in some cycles of judging by standard as the part of Fig. 6 Smalt in the accompanying drawing.

Step 4.3: the top cycle of obtaining is unified to handle, and we are the 180 days the longest time in the cycle with the time span unification, and mxm. all is adjusted into 100;

Step 4.4: the cycle that is obtained in the step 4.3 is carried out hierarchical cluster, because even it is also different according to its evolution of character of incident to belong to of a sort incident, so we have taked it is carried out clustering processing, thereby the incident that will have similar evolution is got together.Shown in Figure 7 in the part design sketch result of hierarchical cluster such as the accompanying drawing;

Step 4.5: to resulting each group of cluster, obtain a curve and guarantee itself and this all group curve square error and minimums, this curve is the optimization model of this group just.Fig. 8 lists the optimization model that part is obtained in the accompanying drawing, and many curves is the dendrogram of curve among the figure, is optimization model that we asked below corresponding;

Step 5: when a new incident arrives such as swine flu, determine by classifying that this incident belongs to and be the public health class.In order to verify optimization model method prediction accuracy, we choose the search data in the swine flu whole world 15 weeks during year July in March, 2009 to 2009 on the Google trends and test, and get the preceding 10 days data in this 15 week and carry out long-term forecasting.Then with get 10 days data and all optimization modeles in the public health class mate, and obtain working as k1=1.5 by traversal, during k2=1, the model of choosing among Fig. 5 can guarantee square error and minimum, and the conversion that the horizontal ordinate and the ordinate of selected model carried out 1/k1 and 1/k2 respectively obtains the long-term forecasting curve among Fig. 9 in the accompanying drawing.Prediction effect this method from figure has predicted the time of flex point basically well, and the secular trend prediction also coincide basically.

Although above the illustrative embodiment of the present invention is described; so that the technician of present technique neck understands the present invention; but should be clear; the invention is not restricted to the scope of embodiment; to those skilled in the art; as long as various variations appended claim limit and the spirit and scope of the present invention determined in, conspicuous when these change, all utilize innovation and creation that the present invention conceives all at the row of protection.

Claims (4)

1. based on the WEB public opinion trend forecasting method of optimization model, it is characterized in that this method may further comprise the steps:

(1), analyze to specify the URL feature of forum to grasp webpage, information document is saved in local data base with relevant data message;

(2), the information document in the local data base is carried out cluster and classification, obtain all kinds of document databases;

(3), from the existing database of all kinds of documents, obtain the time series of each incident desired parameters (as the document size of unit interval etc.) according to event flag and time mark, or download the Google trend time series of each incident correspondence from Google trends website;

(4), at the time series of all kinds of incidents that obtained in the step (3), set up corresponding optimization model and preserve, as the object of predicted object trend coupling;

(5), when new public sentiment incident takes place, at first obtain this incident some time serieses and affiliated big class accordingly to step (3) by step (1).By with its under some optimization modeles that obtain of training in the big class mate, thereby realize long-term forecasting to new public sentiment.

2. WEB public opinion trend forecasting method according to claim 1 is characterized in that all kinds of event establishment optimization modeles.

3. WEB public opinion trend forecasting method according to claim 2 is characterized in that, step (2) is set up specifically may further comprise the steps of optimization model:

A, the pairing curve of each time series of certain class incident is carried out smoothing processing, concrete grammar is the one dimension median filtering method;

B, every curve is cut period treatment, cycle of outgoing event development trend;

C, the cycle that cuts out is carried out the standardization processing of time span and mxm., do not change curve shape;

D, to the curve for layered cluster after the standardization processing, select means clustering algorithm step by step;

E, each group that obtains after the cluster is obtained its optimization model, this model need guarantee square error and the minimum with such all curves.

4. WEB public opinion trend forecasting method according to claim 1 is characterized in that, step (5) is chosen optimization model to new public sentiment and carried out specifically may further comprise the steps of long-term forecasting:

A, the slope of the time series T of new public sentiment incident is analyzed, if slope more than or equal to threshold value 3 the beginning value predict;

B, be 0.1 to stretch or compressed transform to the curve map horizontal ordinate of time series T correspondence and ordinate with mode from 1 to 100 step-length of traversal respectively, from existing optimization model, take out one with conversion after test data set square error and minimum model S, and keep the horizontal ordinate and the ordinate conversion ratio k of test data set at this moment ₁And k ₂

C, with time series T through k ₁And k ₂Change the curve T ' obtain substitute with S in the curve of the same length of mating with it obtain changing after the model S ' of coupling;

D, for the prediction curve of the public sentiment incident that obtains newly arriving, horizontal ordinate and the ordinate of S ' need be pressed 1/k respectively ₁And 1/k ₂Carry out inverse transformation and obtain the long-term forecasting curve

Images