CN103729459A - Method for establishing sentiment classification model - Google Patents

Abstract

The invention provides a sentiment classification method for generating a model deep-convinced-degree network on the basis of the probability of depth study. According to the technical scheme of the method, a plurality of Boltzmann machine layers are stacked, namely, output of this layer is used as input of the next layer. By the adoption of the mode, input information can be expressed in a grading mode, and abstraction can be conducted layer by layer. A multi-layer sensor containing a plurality of hidden layers is the basic study structure of the method. More abstract high layers are formed through combining the characteristics of lower layers and are used for expressing attribute categories or characteristics, so that the distribution type character presentation of data can be discovered. The method belongs to monitoring-free study, and a mainly-used model is the deep-convinced-degree network. The method enables a machine to conduct characteristic abstract better so as to improve the accuracy of sentiment classifications.

Description

A kind of method that builds emotional semantic classification model

Technical field

The application relates to information machine learning areas, a kind of method that builds probability generation model of special design.

Background technology

Along with current cybertimes rise, the degree of depth is learnt to call the revolutionary a kind of new technology of artificial intelligence with the front page of < < New York Times > >.Have reason let us degree of depth study is further understood in depth, as the algorithm of complicated " machine learning ", in identification audio frequency and image accuracy rate, considerably beyond previous technology.But also there is sufficient reason to suspect this viewpoint.Although report, " degree of depth study allows machine carry out the mankind's activity, as sees, listens and think deeply, and can pattern-recognition provide possibility, has promoted the progress of artificial intelligence technology." degree of depth study let us strides forward to the real intelligent machine epoch, is also a small step.Combine with instant practical application, degree of depth study is very important work.

Degree of depth study is a new field in machine learning research, and its motivation is the neural network of setting up, simulation human brain carries out analytic learning, and it imitates the mechanism of human brain and carrys out decryption, for example image, sound and text.Degree of depth study is the one of unsupervised learning.Its concept comes from the research of artificial neural network.Containing the multilayer perceptron of many hidden layers, it is exactly a kind of degree of depth study structure.Degree of depth study forms more abstract high level by combination low-level feature and represents attribute classification or feature, with the distributed nature of finding data, represents.The people such as the Objective Concept Hinton of degree of depth study proposed in 2006.

Study root in depth in traditional " neural network ", " neural network " can trace back to the fifties later stage in 20th century.At that time, Frank's Rosenblat attempted to set up a kind of perceptron of similar mechanical brain, can " memory, makes the machine of response as people's thinking for perception, identification ".This system can be identified some basic shapes within the specific limits, as triangle and square.People place high hopes to its potentiality.

But test finally ends in failure, by Marvin's Ming Siji and he's the co-worker west Palt that rubs, in a book, to be pointed out, the original system of Rosenblat design is very conditional, from literal, blindness is carried out some simple logic functions as distance.As everyone knows, the glamour of " neural network " has disappeared very soon.

But, in the eighties mid-term in last century, when glad of the professor Jeff of Ka Neiji-Mei Long university helps to build more complicated virtual neuroid, can evade the difficult point that some Ming Siji point out, another idea of Rosenblat occurs once again.Han Ding introduces the concept of " hidden layer ", and " hidden layer " neuron allows new generation network to have more complicated learning functionality (the similar XOR function that raw sensed device cannot be processed).But new model also has serious problem.Training time is long, and study is slow, inefficiency, and neural network starts again to lose favour.

But Hinton is unremitting, in 2006, made significant improvement, degree of depth study is proposed, technique is still by Google, Microsoft, and other local institutes apply.A typical setting is such: a computer is faced with a large data set, need to classify to these data, and this allows child classify to toy just as not having in situation of concrete instruction.Child may be to their color, shape or function, or classify in other aspect.Machine learning person attempts doing like this, and for example millions of handwritten forms is learnt on a large scale, these handwritten form data are contrasted mutually, on the basis of similarity to they " cluster ".The important innovations of degree of depth study is to set up model and progressively learns, and attempts to decide the classification of low level, and then the classification of trial learning higher level.

Study in depth and be good at this class problem, be called as unsupervised learning.In some cases, its performance well technology in the past far away.For example, it can be better than early stage system to the study identification of syllable at a new language.But it is still good not, when in the very large situation of the set of classification possibility, in object identification or classification situation, just seem awkward.Google's system that everybody is conventional, but the image that it still can recognition training collection less than 1/6th, but move left and right when the element in image rotation or image, and the result providing can be even worse.

In fact, study in depth and just build huge challenge that intelligent machine a faces part wherein.Such technology lacks causal method for expressing, may face the challenge that obtains abstract concept, as " siblings' relation " or " jointly referring to ".The mode that they do not understand is carried out reasoning from logic, and is having got long long way to go integrating aspect abstract knowledge, as information to as if what, what class is information return, and how to use information.

Based on dark belief network, propose successively training algorithm of non-supervisory greed, for solving the relevant optimization difficult problem of deep structure, bring hope, propose subsequently multilayer autocoder deep structure.In addition convolutional neural networks is first real sandwich construction learning algorithm, and it utilizes space relativeness to reduce number of parameters to improve training performance.

Summary of the invention

In view of this, the object of the present invention is to provide a kind of method that builds probability generation model, the method can improve the accuracy of the abstract calculating of information extraction.

For achieving the above object, technical scheme provided by the invention is:

A method that builds general emotional semantic classification model, the method comprises:

Utilize the feature of artificial neural network, suppose that its output is identical with input, then its parameter is adjusted in training, obtains the weight in every one deck.Naturally, we have just obtained several different expression of input, and these expressions are exactly feature.Autocoder is exactly a kind of neural network that reappears as far as possible input signal.In order to realize this reproduction, need to use autocoder to catch the most important factor that can represent input data, find the principal ingredient that can represent prime information.

When having obtained the coding of ground floor, then need to make the error minimum of reconstruct, just can believe that this coding has been exactly the good representation of original input signal, here we suppose that it and original signal are the same.The training patterns of the second layer and ground floor does not have difference, and we are the input signal as the second layer by the coding of ground floor output, and same minimum reconstructed, will obtain the parameter of the second layer, and obtains the coding of second layer input.Other layer make to use the same method carry out successively just passable.

Through method above, just can obtain multi-layer coding.The number of layers that experiment needs will be according to the concrete effect debugging of oneself experiment.Every one deck all can obtain the different expression of original input, and the vision system of simulating people is here the same.

At present it just study obtained a feature that can well represent input, this feature can represent original input signal to the full extent.So, in order to realize classification, we just can add a sorter at the coding layer on the top of autocoder and return as Rogers is special, and then the supervised training method of the multilayer neural network by standard is gone training.

That is to say, at this time, we need to be input to last sorter by the feature coding of final layer, by there being exemplar, by supervised learning, finely tune.Prototype network by " restriction " be a visual layers and a hidden layer, interlayer exist connect, but layer in unit between there is not connection.Hidden unit is gone to catch the correlativity of the high-order data that show in visual layers by training.

Then, do not consider to push up most and form the two-layer of an associative memory, the connection at degree of a deeply convinceing networking is instructed definite by top-down generation weights, and restriction Boltzmann machine is just as a building block, and it can be easy to connect the study of weights.The most at first, by a non-supervisory greediness successively method go pre-training to obtain the weights of generation model, non-supervisory greediness successively method is proved effectively by Hinton, and is called contrast difference by it.

In this training stage, in visual layers, can produce a vector v, by it, value is delivered to hidden layer.Conversely, the input meeting of visual layers is by random selection, to attempt the original input signal of duplicate removal structure.Finally, these new visual neuronal activation unit by forward direction transmit reconstruct hidden layer activate unit, obtain h, in training process, first by visual vector-valued map to hidden unit; Then visual element is rebuild by Hidden unit; These new visual element are shone upon again to hidden unit, so just obtain new hidden unit.Carry out this step repeatedly and be called gibbs sampler.These retreat and the step of advancing is exactly our familiar Gibbs sampling, and hidden layer activates correlation difference between unit and visual layers input just as the Main Basis of right value update.

Training time can reduce significantly, because only need single step just can approach maximum likelihood study.The every one deck that increases enter the internet all can improve the logarithm probability of training data, and we can be understood as and more and more approach truly expressed.

Accompanying drawing explanation

Fig. 1 is dark belief network Deep Belief Network schematic diagram;

Fig. 2 is limited Bai Ziman machine Restricted Boltzmann Machine schematic diagram;

Fig. 3 is the process flow diagram that the embodiment of the present invention builds the method.

Embodiment

For making object of the present invention, technical scheme and advantage clearer, referring to the accompanying drawing embodiment that develops simultaneously, scheme of the present invention is described in further detail.

X={x in the set of given comment language material ₁, x ₂..., x _n, task, for these documents are carried out to emotional semantic classification, provides each comment its attribute (positive or passive).Wherein every section of comment is all represented as x _i={ x _i3, x _i2..., x _iD.Objective attribute target attribute is also y with vector representation simultaneously _i=1, and-1}, wherein 1 represents that actively ,-1 represents passiveness.Hypothetical target value vector set is Y={y ₁, y ₂..., y _n, object is to find out the mapping function of X to Y.

We use dark belief network as depth model, comment to be expected to train, and dark belief network is a multilayered model, and having an input layer is visible layer, and multiple hidden layers are as Fig. 1, and we set following parameters:

V=h ₀: input layer;

H _i(i=1,2 ... K-1)=i layer hidden layer;

O=h _k: output layer;

H _i: (i=1 ..., K): h _i-1and h _iweight between layer

B _i: (i=1 ..., K): h _iand h _i+1error between layer

C _i: (i=1 ..., K): h _iand h _i-1error between layer

Activation function between wherein two-layer is:

p(h _i-1，s＝1|h _i)＝σ(b _i，s+∑ _jw _i，jh _i，j) (1)

p(h _i，t＝1|h _i-1)＝σ(c _i，t+∑ _jw _i，jh _i-1，j) (2)

Wherein σ (x) is:

σ(x)＝1/(1+e ^-x) (3)

Although gradient descent algorithm can be used for the weights of tuning network, but this algorithm only when initial weight relatively approaches a good solution effect just better.Here by the pre-training pattern of a multilayer, obtain a suitable initial network.From bottom to top, by being regarded as to one deck that visual layers is higher, lower one deck regards hidden layer as, and every a pair of adjacent layer can be regarded limited Bai Ziman machine as.The energy function is here:

E(v，h)＝-∑ _s，tv _sw _sth _t-∑ _sb _sv _s-∑ _tc _th _t (4)

The target of pre-training is the maximum probability that makes to generate training set, and the training set probability of each network allocation can pass through computing formula:

$p\left(v\right)=\underset{h\&Element;H}{\mathrm{\&Sigma;}}p(v,h)=\frac{{\mathrm{\&Sigma;}}_h\exp (-E(v,h))}{{\mathrm{\&Sigma;}}_{u,g}\exp (-E(u,g))}---\left(5\right)$

And its logarithm probability gradient can provide by computing formula:

$\frac{\&PartialD;\log p\left(v\right)}{\&PartialD;w_{\mathrm{st}}}=\&lang;v_s^0{h}_t^0\&rang;-\&lang;v_s^{\&infin;}{h}_t^{\&infin;}\&rang;---\left(6\right)$

Wherein s _v∞ and s _v∞ can, by alternately Gibbs sampling acquisition, can calculate by following formula the renewal of weight:

$\mathrm{\&Delta;}{w}_{\mathrm{st}}=\mathrm{\&epsiv;}(\&lang;v_s^0{h}_t^0\&rang;-\&lang;v_s^1{h}_t^1\&rang;)---\left(7\right)$

In order to make dark belief network better solve emotional semantic classification problem, need to carry out again weights optimization discriminatively, so just make the error minimum of classifying on training set.

$\widehat{\mathrm{\&theta;}}=\underset{\mathrm{\&theta;}}{\arg \min }\left(L(Y,Z,\mathrm{\&theta;})\right)---\left(8\right)$

$L(Y,Z,\mathrm{\&theta;})=\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}\underset{j=1}{\overset{C}{\mathrm{\&Sigma;}}}\exp \left({-y}_{\mathrm{ij}}{z}_{\mathrm{ij}}\right)---\left(9\right)$

Wherein θ=(w ₁..., w _k, c ₁..., c _k), L (Y, Z, θ) is loss function, and for given θ, Y is the desired value of training data reality, and Z is predicted target values, supposes that training set quantity is N.By L2-norm vague generalization:

$C\left(\mathrm{\&theta;}\right)={\mathrm{\&Sigma;}}_i{\mathrm{\&theta;}}_i^2---\left(10\right)$

$\widehat{\mathrm{\&theta;}}=\underset{\mathrm{\&theta;}}{\arg \min }(L(Y,Z,\mathrm{\&theta;})+\mathrm{\&lambda;C}\left(\mathrm{\&theta;}\right))---\left(11\right)$

The above, be only preferred embodiment of the present invention, is not intended to limit protection scope of the present invention.Within the spirit and principles in the present invention all, any modification of doing, be equal to replacement, improvement etc., within all should being included in protection scope of the present invention.

Claims (8)

1. a method that builds learning model, is characterized in that, the method comprises:

Method is to utilize the feature of artificial neural network, and artificial neural network itself is exactly the system with hierarchical structure, if a given neural network, we suppose that its output is identical with input, and then its parameter is adjusted in training, obtains the weight in every one deck.Naturally, we have just obtained several different expression the (every one deck represents a kind of expression) of input, and these expressions are exactly feature.Autocoder is exactly a kind of neural network that reappears as far as possible input signal.In order to realize this reproduction, autocoder just must seizure can represent the most important factor of inputting data, finds the principal ingredient that can represent prime information.

2. the method for structure learning model according to claim 1, is characterized in that, the described method that obtains encoding according to input is:

In general neural network, we have label by the sample of input, have input and desired value, and we go to change the parameter of each layer above according to the difference between current output and desired value like this, until restrain.But we only have without label data now, so, we import a scrambler into input, will obtain a coding, the namely expression of input of this coding, we are exactly input for what know this coded representation so, and we add a demoder, at this time will export an information.If this information and the input signal at the beginning of output are more approaching, we believe that this coding is reliable.So we just, by adjusting the parameter of encoder, make reconstructed error minimum, at this time we just obtained input signal first represented, namely encoded.Because be without label data, so the source of error is exactly to obtain compared with former input after direct reconstruct.

3. the method for structure learning model according to claim 2, is characterized in that, by scrambler, produces feature, successively training;

According to the coding that just can obtain ground floor above, the minimum let us of the error of reconstruct believes that this coding is exactly the good representation of original input signal, and here we suppose that it and original signal are the same.The training patterns of the second layer and ground floor has not just had difference, we are the input signal as the second layer by the coding of ground floor output, and same minimum reconstructed will obtain the parameter of the second layer, and obtain the coding of second layer input, namely second of former input message expression.Other layers make to use the same method and carry out successively.

4. the method for structure network according to claim 3, is characterized in that, disposes the fine setting of supervision;

Through method above, just can obtain multi-layer coding.The number of layers that experiment needs will be according to the concrete effect debugging of oneself experiment.At present it just study obtained a feature that can well represent input, this feature can represent original input signal to the full extent.So, in order to realize classification, we just can add a sorter at the coding layer on the top of autocoder and return as Rogers is special, and then the supervised training method of the multilayer neural network by standard is gone training.

That is to say, at this time, we need to be input to last sorter by the feature coding of final layer, by there being exemplar, by supervised learning, finely tune.

5. the method for structure network according to claim 4, is characterized in that, adds dark belief network;

Dark belief network is comprised of multiple restriction Boltzmann machine layers.These networks are a visual layers and a hidden layer by " restriction ", and interlayer exists and connects, but layer in unit between there is not connection.Hidden unit is gone to catch the correlativity of the high-order data that show in visual layers by training.

First, the connection at degree of a deeply convinceing networking is instructed definite by top-down generation weights, and restriction Boltzmann machine is compared tradition and the sigmoid belief network of Depth Stratification, and it can be easy to connect the study of weights.

The most at first, by a non-supervisory greediness successively method go pre-training to obtain the weights of generation model, in this training stage, in visual layers, can produce a vector v, by it, value is delivered to hidden layer.Conversely, the input meeting of visual layers is by random selection, to attempt the original input signal of duplicate removal structure.Finally, these new visual neuronal activation unit by forward direction transmit reconstruct hidden layer activate unit, obtain h, in training process, first by visual vector-valued map to hidden unit; Then visual element is rebuild by Hidden unit; These new visual element are shone upon again to hidden unit, so just obtain new hidden unit.Carry out this step repeatedly and be called Gibbs sampling.In this patent, we adopt Gibbs sampling, and hidden layer activates correlation difference between unit and visual layers input just as the Main Basis of right value update.

Adopt such method can make the training time to reduce significantly, because only need single step just can approach maximum likelihood study.The every one deck that increases enter the internet all can improve the logarithm probability of training data, and we can be understood as the truly expressed that more and more approaches energy.

6. build degree of deeply convinceing relational network;

The highest two-layer, weights are joined together, and so more the output of low layer will provide the clue of a reference or associated to top layer, and top layer will be related to its memory content like this.After pre-training, dark belief network can be by utilizing tape label data BP algorithm to go to adjust to differentiating performance.Here, a tally set will be affixed to top layer (popularization associative memory), by one bottom-up, the identification weights that study is arrived obtain the classifying face of a network.This performance can be better than the network of simple BP Algorithm for Training.The BP algorithm of dark belief network only need to carry out a local search to weighting parameter space, and this compares feedforward neural network, and training is to want fast, and the time of convergence is also few.

7. the method for structure according to claim 6, is characterized in that, automatic coding and add dark belief network; Bottom is arrived

In order to reach better effect, stacking autocoder can be added to dark belief network, it is the restriction Bai Ziman machine by replace the dark belief network of tradition the inside with stacking autocoder.This can be trained and be produced degree of depth multilayer neural network framework by same rule with regard to making, but it lacks the parameterized strict demand of layer.Different from DBNs, autocoder uses discrimination model, and this structure is just difficult to sampling input sample space like this, and this just makes its internal representations of the more difficult seizure of network.But noise reduction autocoder but can well be avoided this problem, and more excellent than traditional DBNs.Train the process of single noise reduction autocoder the same with the process of RBMs training generation model.

8. add the content about emotion text-processing.According to the method described in claim 1～6, text is carried out to emotional semantic classification, it is characterized in that, text is carried out to pre-service;

First comment document is carried out to cutting, then remove stop words, and represent to be divided into again two parts, training set and test sets with vectorial form.Wherein training set can also be further divided into two parts, is used for respectively training in advance and tuning, uses above-mentioned model.Wherein pre-training is unsupervised, with a kind of greediness successively algorithm obtain initial network.Then the parameter that in evolutionary process, previous step is obtained to network is adjusted by BP algorithm.

Images