CN108984526A - A kind of document subject matter vector abstracting method based on deep learning - Google Patents

Abstract

The invention relates to a method for extracting document topic vectors based on deep learning, and belongs to the technical field of natural language processing. The method of the present invention uses the convolutional neural network to extract local deep semantic information, uses the LSTM model to learn the time series information, makes the semantics of the vector more comprehensive, and uses the implicit co-occurrence relationship between contextual phrases and document topics to avoid For the shortcomings of short texts, some sentence-based topic vector models use the attention mechanism to organically combine CNN and LSTM models to learn the deep semantics, timing information and salient information of the context, and more effectively build a file topic vector. Model.

Description

A Method of Document Topic Vector Extraction Based on Deep Learning

technical field

The invention relates to a method for extracting document topic vectors based on deep learning, and belongs to the technical field of natural language processing.

Background technique

In today's big data era, how to discover the topics of massive Internet text data is a research focus. To analyze the theme of text data, the document theme vector essentially represents the deep semantics of the document, and is the internal combination of theme and semantics. The extracted document topic vector can be widely used in natural language processing tasks, including public opinion analysis of social networks and new media, timely acquisition of news hotspots, etc. Therefore, how to efficiently extract document topic vectors is an important research topic.

For text data, its theme is not necessarily directly reflected in the specific text content, which makes it difficult to mine the hidden theme of the text. It is necessary to extract the content contained in the document according to the relationship between words, sentences, and paragraphs in the text. The theme meaning of the document is combined with the chapter relationship of the document to extract the theme of the document. In recent years, with the enrichment of statistical natural language processing methods and corpora, text topic modeling methods based on "word-topic" and "document-topic" have also been proposed. The basic idea is to assume that the topic of each word and document is Obey a statistical probability distribution, through the training of corpus data, calculate the probability distribution of its document topic, and then cluster according to this document topic.

To correctly analyze the theme of each document, the traditional method is to perform theme analysis on each word of the text, but this method has a big problem: the words that really determine the theme of the text actually only account for a small part of the words in the text , so the traditional method will analyze a large number of words that have nothing to do with the topic. On the one hand, the irrelevant words lead to a large amount of calculation. deep semantic issues.

With the improvement of hardware performance and the continuous expansion of data scale, deep learning has also been widely used in various fields, greatly improving the experimental results on its original basis. Deep learning, with its elegant model and flexible architecture, has been widely used in recent years in the method of combining word embedding and document embedding. Among all deep learning methods, CNN (Convolutional Neural Network, Convolutional Neural Network) and LSTM model (LongShort-Term Memory, long-term short-term memory network model) are the two most mainstream. In natural language processing tasks, text analysis methods based on CNN and LSTM models can well discover the latent semantic features of text, and give natural language processing tasks such as automatic summarization, sentiment analysis, and machine translation a great advantage in semantic analysis calculations. help.

Contents of the invention

The purpose of the present invention is to overcome the defects of the prior art, solve the problem of how to mine the deep semantics of the text in combination with the internal relevance relationship of the text, and propose a document topic vector extraction method based on deep learning. The present invention focuses more on the analysis of document topic feature vectors in the modeling of document topic vectors, digs out the implicit correlation between text features and topic vectors, and thereby learns document topic vectors.

The core idea of the present invention is: use CNN to extract the semantics of contextual phrases, input the extracted semantics into the LSTM model, and use the attention mechanism to extract the importance of different positions of the text and words with different meanings, thereby retaining important information and also The organic combination of CNN and LSTM models has been completed, the intrinsic correlation between contexts has been mined, and document topic vectors with deep semantics and significant features have been learned.

The method of the present invention is realized through the following technical solutions.

A method for extracting document topic vectors based on deep learning, comprising the following steps:

Step 1. Define the relevant definitions, as follows:

Definition 1: Document D, D=[w ₁ ,w ₂ ,..., _wi ,...,w _n ], w _i represents the i-th word of document D;

Definition 2: The predicted word w _d+1 represents the target word that needs to be learned;

Definition 3: Window words are composed of several words that appear consecutively in the text, and there are hidden internal associations between window words;

Definition 4: context phrase, which means the window word that appears before the position of the predicted word, the window length is l, and the context phrase is recorded as w _dl , w _d-l+1 ,...,w _d ;

Definition 5: Document topic mapping matrix, learned by LDA algorithm (Latent Dirichlet Allocation), each row represents the topic of a document;

Definition 6: N _d and doc _id , N _d represents the number of documents in the corpus, and doc _id represents the location of the document; each document corresponds to a unique doc _id , where 1≤doc _{id≤N d} ;

Step 2. Use CNN to learn the semantic vector of the context phrase.

Step 3: Use the LSTM model to learn the semantics of contextual phrases, and obtain hidden layer vectors h _dl , h _d-l+1 ,...,h _d .

Step 4. Through the attention mechanism, combine the CNN and LSTM models organically to obtain the average value of the contextual phrase semantic vector

Step 5. Use the average value of contextual phrase semantic vectors by logistic regression Predict the target word w _d+1 with the document topic information, and obtain the predicted probability of the target word w _d+1 .

Beneficial effect

A method for extracting document topic vectors based on deep learning in the present invention, compared with the prior art, has the following beneficial effects:

1. Use CNN to extract local deep semantic information;

2. Use the LSTM model to learn the timing information, making the semantics of the vector more comprehensive;

3. Select the implicit co-occurrence relationship between contextual phrases and document topics, avoiding the shortcomings of some sentence-based topic vector models for short texts;

4. Using the attention mechanism to organically combine the CNN and LSTM models, learn the deep semantics, timing information and salient information of the context, and more effectively build a model for file topic vector extraction.

Description of drawings

FIG. 1 is a flowchart of a method for extracting document topic vectors based on deep learning in the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the abstracting method described in the present invention will be further described in detail below according to the drawings and embodiments.

A document topic vector extraction method based on deep learning, the basic implementation process is as follows:

Step 1. Define the relevant definitions, as follows:

Definition 1: Document D, D=[w ₁ ,w ₂ ,..., _wi ,...,w _n ], w _i represents the i-th word of document D;

Definition 2: predict word w _d+1 ;, indicating the target word that needs to be learned;

Definition 3: Window words are composed of several words that appear consecutively in the text, and there are hidden internal associations between window words;

Definition 4: The context phrase (w _dl ,w _d-l+1 ,...,w _d ) indicates the window word that appears before the location of the predicted word, and the length of the context phrase is l;

Definition 5: Document topic mapping matrix, learned by LDA algorithm, each row represents the topic of a document;

Definition 6: N _d and doc _id , N _d represents the number of documents in the corpus, and doc _id represents the location of the document; each document corresponds to a unique doc _id , where 1≤doc _{id≤N d} ;

Step 2. Use CNN to learn the semantic vector Context of the context phrase.

The specific implementation process is as follows:

Step 2.1 Use word2vec and other algorithms to train the word vector matrix of document D. The size of the word vector matrix is n×m, where n represents the length of the word vector matrix, and m represents the width of the word vector matrix;

Step 2.2 extracts the word vector corresponding to each word in the context phrase from the word vector matrix obtained in step 2.1, so as to obtain the vector matrix M of the context phrase w _dl , w _d-l+1 ,...,w _d ;

Step 2.3 uses CNN to calculate the semantic vector Context of the context phrase. Specifically, the vector matrix M obtained in step 2.2 and the convolution kernel of the K layer with a size of C _l × C _m are operated;

Wherein, K represents the number of convolution kernels, K is equal to 128 in this specific embodiment, C ₁ represents the length of the convolution kernel, and C ₁ =1, C _m represents the width of the convolution kernel, and C _m =m;

The semantic vector Context of the context phrase is calculated by formula (1):

1≤k≤K

Context＝[Context ₁ ,Context ₂ ,...,Context _K ]

Among them, Context _k represents the k-th dimension of the semantic vector of the context phrase, l represents the length of the context phrase, m represents the width of the word vector matrix, that is, the word vector dimension, d represents the starting position of the first word in the context phrase, c _pq is the weight parameter of the pth row and qth column of the convolution kernel, M _pq represents the pth row and qth column data of the vector matrix M, and b is the bias parameter of the convolution kernel;

Step 3: Use the LSTM model to learn the semantics of contextual phrases, and obtain hidden layer vectors h _dl , h _d-l+1 ,...,h _d .

The specific implementation process is as follows:

Step 3.1 assigns t to d-l, that is, t=d-l, and t represents the tth moment;

Step 3.2 Assign x _t to the word vector of w _t , where x _t represents the word vector input at the t-th moment, and w _t represents the word input at the t-th moment;

Among them, the word vector of w _t is obtained by mapping the word vector matrix output in step 2.1, that is, extracting the word vector of w _t in the corresponding position of the vector matrix M;

Step 3.3 takes x _t as the input of the LSTM model, and obtains the hidden layer vector h t at time _t ;

The specific implementation process is as follows:

Step 3.3.1 Calculate the forgetting gate f _{t at time t} , which is used to control the forgotten information, and is calculated by formula (2);

f _t ＝σ(W _f x _t +U _f h _t-1 +b _f ) (2)

Among them, W _f represents the parameter matrix, x _t represents the word vector input at time t, U _f represents the parameter matrix, h _t-1 represents the hidden layer vector at time t-1, b _f represents the bias vector parameter, when t = dl, h _t-1 = h _dl-1 , and h _dl-1 is a zero vector, σ represents the Sigmoid function, which is the activation function of the LSTM model;

Step 3.3.2 Calculate the input gate i _t at time t, which is used to control the new information that needs to be added at the current time, and is calculated by formula (3);

i _t =σ(W _i x _t +U _i h _t-1 +b _i ) (3)

Among them, W _i represents the parameter matrix, x _t represents the word vector input at time t, U _i represents the parameter matrix, h _t-1 represents the hidden layer vector at time t-1, _bi represents the bias vector parameter, and σ represents The Sigmoid function is the activation function of the LSTM model;

Step 3.3.3 Calculate the updated information at time t Calculated by formula (4);

in, Represents the parameter matrix, x _t represents the word vector input at the tth moment, Represents the parameter matrix, h _t-1 represents the hidden layer vector at time t-1, Represents the bias vector parameter, tanh represents the hyperbolic tangent function, which is the activation function of the LSTM model;

Step 3.3.4 calculates the information at time t, which is obtained by adding the information at the previous time and the updated information at the current time, and calculates by formula (5);

Among them, c _t represents the information at time t, f _t represents the forgetting gate at time t, c _t-1 represents the information at time t-1, and it represents the input gate at time _t , Indicates the updated information at time t, ° indicates the cross product of vectors;

Step 3.3.5 Calculate the output gate o _{t at time t} , which is used to control the input information, calculated by formula (6):

o _t ＝σ(W _o x _t +U ₀ h _t-1 +b _o ) (6)

Among them, W _o represents the parameter matrix, x _t represents the word vector input at time t, U ₀ represents the parameter matrix, h _t-1 represents the hidden layer vector at time t-1, b _o represents the bias vector parameter, σ represents The Sigmoid function is the activation function of the LSTM model; among them, the parameter matrix W _f , U _f , W _i , U _i in steps 3.3.1-3.3.3 and step 3.3.5, The matrix elements of W _o and U _o have different sizes, and the bias vector parameters b _f , b _i , The size of the elements in b _o is different; step 3.3.6 calculates the hidden layer vector h _{t at time t} , calculated by formula (7):

h _t =o _t οc _t (7)

Among them, o _t represents the output gate at time t, and c _t represents the information at time t;

Step 3.4 Determine whether t is equal to d, if not, add 1 to t, and skip to step 3.2; if it is equal, output hidden layer vectors h _dl , h _d-l+1 ,...,h _d , and jump to step 4 ;

Step 4. Use the attention mechanism to combine the CNN and LSTM models to obtain the average value of the context phrase semantic vector The specific implementation process is as follows:

Step 4.1 Use the semantic vector of the context phrase obtained in step 2 to obtain the importance factor α of each word on the semantic vector of the context phrase through the attention mechanism, specifically calculated by formula (8):

d-l≤t≤d

α＝[α _dl ,α _d-l+1 ,...,α _d ] (8)

Among them, α _t represents the importance factor of the word on the semantic vector of the context phrase at time t, Context represents the semantic vector of the context phrase obtained in step 2, x _t represents the word vector input at time t, and x _i represents time i The input word vector; T represents the transposition of the vector; e represents an exponential function based on e, which is a natural constant;

Step 4.2 calculates the weighted hidden layer vector h' based on the attention mechanism, which is calculated by formula (9);

h′ _t =α _t *h _t

d-l≤t≤d

h'=[h' _dl ,h' _d-l+1 ,...,h' _d ] (9)

Among them, h′ _t represents the weight hidden layer vector h′ _{t at time t} , α _t represents the importance factor of each word on the semantic vector of the context phrase at time t, and h _t represents the hidden layer vector at time t;

Step 4.3 uses the mean-pooling operation to calculate the average value of the context phrase semantic vector Calculated by formula (10):

Among them, h′ _t represents the weight hidden layer vector h′ t at time _t ;

Step 5: Predict the target word w _d+1 by using the mean value of the semantic vector of the context phrase and the topic information of the document by means of logistic regression, and obtain the predicted probability of the target word w _d+1 . The specific implementation process is as follows:

Step 5.1 uses the LDA algorithm to learn the document topic mapping matrix, and then maps each document into a -dimensional vector D _z whose length is equal to the width of the word vector matrix in step 2.1 according to the document topic mapping matrix and doc _id ;

Step 5.2 takes the average value of the vector D _z output in step 5.1 and the context phrase semantic vector output in step 4 spliced together to get the spliced vector V _d ,

Step 5.3 uses V _d output from step 5.2 to predict the target word w _d+1 . Specifically, it is classified by the method of logistic regression, and the objective function is as formula (11)

Among them, θ _d+1 is the parameter corresponding to the position of the target word w _d+1 , θ _i corresponds to the parameter corresponding to the word w _i in the vocabulary, |V| represents the size of the vocabulary, and V _d is the splicing vector obtained in step 5.2 , exp represents the exponential function with e as the base, Σ represents the summation; P represents the probability, y represents the dependent variable, and T represents the matrix transposition.

Step 5.4 uses the method of cross entropy to calculate the loss function of the objective function (11) by formula (12):

L=-log(P(y=w _d+1 |V _d )) (12)

Among them, w _d+1 represents the target word, V _d is the splicing vector of step 4.2, and log() represents a logarithmic function with base 10;

The loss function (12) is updated and solved by the Sampled Softmax algorithm and the mini-batch stochastic gradient descent parameter update method to obtain the document topic vector.

So far, from step 1 to step 5, the document topic vector with deep semantics and salience is completed.

Example

This embodiment describes the specific implementation process of the present invention, as shown in FIG. 1 .

As can be seen from Figure 1, the process of a method for extracting document topic vectors based on deep learning in the present invention is as follows:

Step A preprocessing; first remove meaningless symbols in the corpus, such as special characters, etc., and then segment the text. Word segmentation is the process of dividing a continuous sequence of words into separate words according to established lexical rules, thereby decomposing a sentence into several continuous meaningful word strings for subsequent analysis. The word segmentation operation uses the PTB tokenizer for word segmentation processing. After the word segmentation, a vocabulary is constructed for the original text. In this embodiment, the vocabulary is selected from the top 20,000 words in the training text from high to low, that is, the size of the vocabulary V is 20,000. After the vocabulary is selected, the vocabulary index data of the original corpus is constructed according to the vocabulary index, and this text vocabulary index data is used as the input of the model.

Step B uses the word2vec algorithm to learn word vectors. Input the words in the document into the word2vec algorithm to get the word vector, and its objective function is as formula (13):

Among them, k is the window word, i is the current word, Corp is the size of the word in the corpus, and the 128-dimensional word vector is learned by using the gradient descent method;

Step C uses CNN to extract contextual phrase semantic vectors, and uses RNN to learn contextual phrase hidden layer vectors;

Wherein, using CNN to extract contextual phrase semantic vectors, using RNN to learn contextual phrase hidden layer vectors are calculated in parallel, specific to this embodiment:

Use CNN to extract contextual phrase semantic vector; first use Gaussian distribution to randomly initialize a K-layer convolution kernel with a size of C _l ×C _m , for a given contextual phrase w _dl ,w _d-l+1 ,... ,w _d , through the word vector learned in step B, these context phrases are mapped into a matrix of size l×m, where l is the length of the context phrase, m is the dimension of the word vector, and the matrix is randomly initialized in the volume The convolution operation is performed on the product kernel, and the specific operation method is shown in formula (1), so that a vector Context is obtained, which is the semantic vector of the context phrase;

Use RNN to learn the hidden layer vector of contextual phrases; input the word vectors corresponding to contextual phrases w _dl ,w _d-l+1 ,...,w _d into the LSTM model in order, and the hidden layer vector h at time 0 Each dimension of ₀ is set to 0, and then the forgetting gate, input gate, output gate and final result context phrase hidden layer vector are calculated sequentially using formulas (2)-(7), and the dimension size is set to 128;

Step D uses the attention mechanism to calculate the weighted semantic vector and calculate the topic distribution of the document;

Among them, the use of the attention mechanism to calculate the weighted semantic vector and the calculation of the topic distribution of the document are calculated in parallel, specific to this embodiment:

Use the attention mechanism to calculate the weighted semantic vector; according to the word vector obtained in step B and the semantic vector of the context phrase obtained in step C, perform an attention mechanism operation on each word in the context phrase to obtain the attention factor α _t , α _t is a real number between 0 and 1. The larger the number, the more word vector information in its corresponding position will be retained in the last mean-pooling layer, so its size represents the current word in The importance of representing the meaning of the entire phrase, that is, the more important words will receive more attention;

Calculate the document topic distribution; specifically, use the LDA algorithm to calculate, first input the document D into the LDA algorithm, and obtain the topic distribution of each document D, and the topic distribution is directly used as the final result, which is recorded as D _z ;

Step E predicts the target word and learns the document topic vector; directly splices the weighted semantic vector and _Dz , and then maximizes the probability of the target word, which can be obtained by the Sampled Softmax algorithm and the small-batch stochastic gradient descent parameter update method Document topic vector.

Claims (5)

1. A document theme vector extraction method based on deep learning is characterized by comprising the following steps:

step one, performing relevant definition, specifically as follows:

definition 1: document D, D ═ w₁,w₂,...,w_i,...,w_n]，w_iThe ith word representing document D;

definition 2: predicting word w_d+1(ii) a Representing a target word to be learned;

definition 3: window words which are formed by words continuously appearing in the text, and hidden internal association exists between the window words;

definition 4: the contextual phrase: w is a_d-l,w_d-l+1,...,w_dThe word is a window word appearing before the position of the predicted word, and the length of the context phrase is l;

definition 5: the document theme mapping matrix is obtained by learning of an LDA algorithm, and each row represents a theme of a document;

definition 6: n is a radical of_dAnd doc_id，N_dRepresenting the number of documents in the corpus, doc_idRepresenting a location of a document; each document corresponds to a unique doc_idWherein, 1 is less than or equal to doc_id≤N_d；

Learning to obtain a semantic vector of the context phrase by using a Convolutional Neural Network (CNN);

thirdly, learning the semantics of the context phrase by utilizing the long-short term memory network model LSTM to obtain a hidden layer vector h_d-l,h_d-l+1,...，h_d(ii) a The specific implementation process is as follows:

step 3.1, assigning t to d-l, namely t is d-l, and t represents the t-th moment;

step 3.2 reaction of x_tAssignment w_tWord vector of x_tRepresenting the word vector entered at time t, w_tA word indicating input at the t-th time;

wherein, w_tThe word vectors are obtained by mapping the word vector matrix output in step 2.1, i.e. extracting w_tWord vectors at the corresponding positions of the vector matrix M;

step 3.3 reaction of x_tObtaining a hidden layer vector h at the time t as an input of an LSTM model_t；

Step 3.4, judging whether t is equal to d, if not, adding 1 to t, and skipping to step 3.2; if yes, the hidden layer vector h is output_d-l,h_d-l+1,...,h_dJumping to the step four;

step four, organically combining the CNN model and the LSTM model through an attention mechanism to obtain the average value of the semantic vector of the context phraseThe specific implementation method comprises the following steps:

step 4.1, obtaining an importance factor α of each word on the semantic vector of the context phrase through an attention mechanism by using the context phrase semantic vector obtained in the step two, and specifically calculating through the following formula:

d-l≤t≤d

α＝[α_d-l,α_d-l+1,...,α_d]

wherein alpha is_tRepresenting the importance factor of the word at the time t on the semantic vector of the Context phrase, Context representing the semantic vector of the Context phrase obtained in the step two, x_tRepresenting the word vector, x, input at time t_iRepresenting the word vector input at the ith moment; t represents the transposition of the vector; e represents an exponential function with e, a natural constant, as the base;

step 4.2, calculating a hidden layer vector h' with weight based on attention mechanism by the following formula;

h_t′＝α_t*h_t

d-l≤t≤d

h′＝[h′_d-l,h′_d-l+1,...,h_d′]

wherein h is_t' denotes the weighted hidden layer vector h at time t_t′，α_tAn importance factor, h, representing each word at time t on the semantic vector of the context phrase_tRepresenting a hidden layer vector at the moment t;

step 4.3 calculate the mean of the semantic vector of the context phrase using mean-posing operationCalculated by the following equation (10):

wherein h is_t' denotes the weighted hidden layer vector h at time t_t′；

Step five, utilizing the average value of the semantic vector of the context phrase by a logistic regression methodAnd the document topic information prediction target word w_d+1Obtaining the target word w_d+1The prediction probability of (2).

2. The method for extracting document theme vectors based on deep learning of claim 1, wherein the second step is implemented by the following steps:

step 2.1, training a word vector matrix of the document D, wherein the size of the word vector matrix is n multiplied by m, n represents the length of the word vector matrix, and m represents the width of the word vector matrix;

step 2.2, extracting the word vector corresponding to each word in the context phrase from the word vector matrix obtained in step 2.1 to obtain the context phrase w_d-l,w_d-l+1,...,w_dA vector matrix M of (a);

step 2.3, calculating the semantic vector Context of the Context phrase by using the CNN, wherein the vector matrix M and the K layers obtained in the step 2.2 have the size of C_l×C_mThe convolution kernel of (a) operates;

where K denotes the number of convolution kernels, C_lRepresents the length of the convolution kernel, and C_l＝l，C_mRepresents the width of the convolution kernel, and C_m＝m；

The semantic vector Context of a Context phrase is calculated by equation (1):

1≤k≤K

Context＝[Context₁,Context₂,...,Context_K]

wherein, Context_kIndicating short contextThe kth dimension of the semantic vector of a word, l represents the length of the context phrase, m represents the width of the word vector matrix, i.e., the word vector dimension, d represents the starting position of the first word in the context phrase, c_pqIs the weight parameter of the p-th row and q-th column of the convolution kernel, M_pqLine p and column q data representing the vector matrix M, b is the bias parameter for the convolution kernel.

3. The method for extracting document theme vector based on deep learning of claim 1, wherein the specific implementation method of the step 3.3 is as follows:

step 3.3.1 calculating forgetting door f at t moment_tThe forgetting control module is used for controlling forgetting information and calculating through a formula (2);

f_t＝σ(W_fx_t+U_fh_t-1+b_f) (2)

wherein, W_fRepresenting a parameter matrix, x_tRepresenting the word vector, U, input at time t_fRepresents a parameter matrix, h_t-1Representing the hidden layer vector at time t-1, b_fDenotes the offset vector parameter, when t is d-l, h_t-1＝h_d-l-1And h is_d-l-1Is a zero vector, sigma represents a Sigmoid function, and is an activation function of the LSTM model;

step 3.3.2 input Gate i at time t_tThe new information to be added at the current moment is controlled and calculated through a formula (3);

i_t＝σ(W_ix_t+U_ih_t-1+b_i) (3)

wherein, W_iRepresenting a parameter matrix, x_tRepresenting the word vector, U, input at time t_iRepresents a parameter matrix, h_t-1Representing the hidden layer vector at time t-1, b_iRepresenting a bias vector parameter, and sigma representing a Sigmoid function, which is an activation function of an LSTM model;

step 3.3.3 calculating updated information at time tBy the formula (4) Calculating;

wherein,representing a parameter matrix, x_tRepresenting the word vector entered at time t,represents a parameter matrix, h_t-1Representing the hidden layer vector at time t-1,representing a bias vector parameter, and tanh representing a hyperbolic tangent function, which is an activation function of an LSTM model;

step 3.3.4, calculating the information of the t moment, adding the information of the previous moment and the updated information of the current moment to obtain the information, and calculating the information through a formula (5);

wherein, c_tInformation indicating time t, f_tIndicating forgetting to leave door at time t, c_t-1Information indicating the time t-1, i_tThe input gate at time t is shown,information indicating the update at time t is provided,represents a cross product of the vectors;

step 3.3.5 output gate o at time t is calculated_tFor controlling the input information, calculated by equation (6):

o_t＝σ(W_ox_t+U₀h_t-1+b_o) (6)

wherein, W_oRepresenting a parameter matrix, x_tRepresenting the word vector, U, input at time t₀Represents a parameter matrix, h_t-1Representing the hidden layer vector at time t-1, b_oRepresenting a bias vector parameter, and sigma representing a Sigmoid function, which is an activation function of an LSTM model; wherein the parameter matrix W_f，U_f，W_i，U_i，W_o，U_oHas different matrix element sizes, and is biased with vector parameter b_f，b_i，b_oThe elements in (A) are different in size;

step 3.3.6 calculating the hidden layer vector h at time t_tCalculated by equation (7):

wherein o is_tOutput gate representing time t, c_tInformation indicating time t.

4. The method for extracting document theme vectors based on deep learning of claim 1, wherein the concrete implementation method of the fifth step is as follows:

step 5.1 learning the document theme mapping matrix, then according to the document theme mapping matrix and doc_idEach document is mapped into a-dimensional vector D of equal length and width of the word vector matrix in step 2.1_z；

Step 5.2 vector D output from step 5.1_zAnd the average value of the context phrase semantic vector output in the step fourSplicing to obtain a splicing vector V_d，

Step 5.3 uses the V output from step 5.2_dTo predict the target word w_d+1；

Step 5.4, calculating a loss function of the target function (11) by using a cross entropy method through a formula (12):

L＝-log(P(y＝w_d+1|V_d)) (12)

wherein, w_d+1Representing a target word, V_dIs the concatenation vector of step 4.2, log () represents a base-10 logarithmic function;

and (3) updating and solving the loss function (12) by a Sampled Softmax algorithm and a small batch random gradient descent parameter updating method to obtain a document theme vector.

5. The method for extracting document theme vector based on deep learning as claimed in claim 4, wherein in the step 5.3, classification is performed by a logistic regression method, and the objective function is as the formula (11)

Wherein, theta_d+1Is the target word w_d+1The parameter, theta, corresponding to the location_iWord w in the corresponding word list_iCorresponding parameters, | V | representing the size of the vocabulary, V_dThe splicing vector obtained in the step 5.2 is used, exp represents an exponential function with e as a base, and sigma represents summation; p denotes probability, y denotes dependent variable, and T denotes matrix transposition.