CN109726383A - A kind of article semantic vector representation method and system - Google Patents
A kind of article semantic vector representation method and system Download PDFInfo
- Publication number
- CN109726383A CN109726383A CN201711024027.4A CN201711024027A CN109726383A CN 109726383 A CN109726383 A CN 109726383A CN 201711024027 A CN201711024027 A CN 201711024027A CN 109726383 A CN109726383 A CN 109726383A
- Authority
- CN
- China
- Prior art keywords
- vector
- sentence
- article
- semantic
- attention rate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Machine Translation (AREA)
Abstract
The present invention provides a kind of article semantic vector representation method and system, and representation method includes: S1, the sentence vector for obtaining according to all term vectors of sentence any in article any sentence；S2, the corresponding sentence vector of the correspondence sentence vector sum inverted order arrangement according to the sentence permutation with positive order of article is inputted into two-way length time memory network model, obtains the second output quantity of the first output quantity and the corresponding attention rate sentence vector of any vector that correspond to any vector；S3, the attention rate according to the second output quantity and corresponding attention rate sentence vector to any vector, the semantic vector of the corresponding sentence of any vector of acquisition；S4, according to the semantic vector of the corresponding sentence of all vectors in article, obtain article semantic vector.Have the invention enables the calculation amount in paragraph extraction of semantics stage totality and be greatly reduced, and solves the function that semantic vector of the tradition based on term vector extracts the paragraph semantic expressiveness that cannot achieve.
Description
Technical field
The present invention relates to article semantic analysis fields, more particularly, to a kind of article semantic vector representation method and are
System.
Background technique
The vector expression of article semanteme plays an important role in numerous areas relevant to natural language processing, such as
The research of the extraction of text centric thought, the semantic analysis of text, text classification, conversational system and machine translation etc..
But the semantic expressiveness that existing technology is made an issue of calculates section on the basis of term vector using the method based on term vector
The semantic expressiveness fallen.
Fig. 1 is a kind of schematic diagram of the article semantic vector representation method based on term vector in the prior art, is please referred to
Fig. 1, this method are the semantic expressiveness based on term vector, and the process of this method is that term vector is passed through length time memory net
Network (Long Short-Term Memory, LSTM) directly output obtains the semantic vector of sentence or article.This method needs
First text to be segmented, vocabulary vectorization is carried out after participle, term vector is obtained, by term vector according to the time sequencing of sentence
Input LSTM model.The last output result of model is exactly the final semantic vector of this text.
Wherein, { x1,x2,...,xnIt is the word sequence inputted, pass through Word2vector and obtains term vector, theoretical upper mold
The last output result of type by comprising the information that should retain all in this word, therefore can using this output result as
The semantic vector of this words indicates, but extracts the mode taken to semantic vector and will greatly affect to text semantic expression
Effect.
For this sentence semantics representation method based on term vector, to translate (neural based on neural network machine
Machine translation, NMT) in sequence to sequence model for, in the Encode into model according to
Perhaps term vector is until we input the last character vector or term vector of this sentence to secondary input word vector, at this time
Encode exports the semantic vector of whole sentence.The characteristics of NMT is exactly the input information each step before consideration, so in theory
This upper semantic vector can include the information of entire sentence.But in actual mechanical process, with the continuous increasing of word sequence
It is long, especially reach the other amount of text of paragraph level, discovery has following problem: when sequence continually enters, semantic information can not be remembered
With the information for indicating entire sequence；Impact factor between vocabulary and vocabulary is similar, can not protrude the emphasis in text；It is difficult to mention
Taking the semantic vector of paragraph text indicates.
In existing technology, another kind be also most representational semantic vector representation method be it is word-based to
The attention model of amount obtains semantic vector and indicates.The transfer of attention is the brain of people when receiving or handling external information
By the ingenious focus reasonably changed to external information of sense organ, selectively ignore to itself less relevant content,
And the information itself needed is amplified.By changing focus, human brain is sensitive to the reception of the information at focused on position
Degree and information processing rate all greatly enhance, and can effectively filter incoherent information, prominent closely related information.For total
It, the basic thought of attention mechanism is not disposable each position fair play entire scene, but according to demand
Emphasis is focused on into specific position.Once the rule of specific extraction determines, machine learning or deep neural network are recycled
Learn to observe the position that image attention power should be concentrated to future.
The expression of text semantic vector most starts introducing attention mechanism and has been used on NMT, and neural network machine translation is
One typical sequence to sequence model, wherein including an encoder to decoder model.
Fig. 2 is a kind of signal of the article semantic vector representation method based on term vector attention model in the prior art
Figure, as shown in Fig. 2, existing indicate to be to use a circulation nerve net based on term vector attention model acquisition semantic vector
Network (Recurrent neural Network, RNN) encodes the word in source text according to time series, after coding
Each output with corresponding attention rate be multiplied, finally sum obtain the intermediate semantic vector an of fixed dimension.Specifically
Semantic vector indicates are as follows:
Wherein, ciFor semantic vector, TsIt is the sentence number of article, h for b, bjPass through LSTM's for j-th term vector
Output quantity, exp (eij) it is e using e the bottom of asijPower, TxFor i, vaIt is term vector by the output quantity of LSTM,For Si-1's
Weight matrix, Si-1Hidden state for decoder at the (i-1)-th moment, UaFor hjWeight matrix, tanh () be activation primitive 0
< j≤i, 0 < i≤b, i, j ∈ Z, Z is set of integers.
The semantic vector representation method of attention model based on term vector is made due to introducing attention mechanism
Each vocabulary is extracting synchronous attention rate after data learn, every kind of vocabulary has corresponding attention rate, and
The semantic expressiveness for being weighted the sentence of acquisition realizes emphasis between sentence and extracts.
But this method is only simply to indicate directly to be added by sentence semantics, and the translation of article, article are plucked
It extracts this kind of long text to be difficult to carry out semantic vector expression, cannot indicate based on term vector and well article or paragraph
It is whole semantic.
Summary of the invention
The present invention provides a kind of a kind of article semantic vector representation method for overcoming the above problem and system.
According to an aspect of the present invention, a kind of article semantic vector representation method is provided, comprising: S1, according to the text
All term vectors of any sentence in chapter obtain the sentence vector of any sentence；S2, by the sentence positive sequence according to the article
The corresponding sentence vector of the corresponding sentence vector sum inverted order arrangement of arrangement inputs two-way length time memory network model, obtains
The second output quantity of the first output quantity and the corresponding attention rate sentence vector of any vector that correspond to any vector is taken,
Wherein, the corresponding attention rate sentence vector of any vector are as follows: described any under the sentence vector permutation with positive order of the article
At least one vector that sentence vector before any vector described in sentence vector sum is constituted；S3, according to second output quantity
With the corresponding attention rate sentence vector to the attention rate of any vector, the corresponding sentence of any vector is obtained
Semantic vector；S4, according to the semantic vector of the corresponding sentence of all vectors in the article, it is semantic to obtain the article
Vector.
Preferably, step S1 further comprises: by the same dimension point of all term vectors of sentence any in the article
It sums up, obtains the sentence vector of any sentence.
Preferably, step S2 further comprises: will be defeated according to the corresponding sentence vector of the sentence permutation with positive order of the article
Enter two-way length time memory network model, obtains first sentence vector to the semantic information of any vector and described
Semantic information of first sentence vector to the attention rate sentence vector；By the corresponding of the arrangement of the sentence inverted order according to the article
Sentence vector input two-way length time memory network model, obtain the last one vector to any vector semanteme
The semantic information of information and the last one described vector to the attention rate sentence vector；By first sentence vector described in
Semantic information integration of the semantic information and the last one described vector of any vector to any vector, acquisition pair
It should believe in the first output quantity of any vector, also, by the semanteme of first sentence vector to the attention rate sentence vector
The semantic information of breath and the last one described vector to the attention rate sentence vector is integrated, and is obtained and is corresponded to the attention rate sentence
Second output quantity of vector.
Preferably, corresponding attention rate sentence vector described in step S3 passes through following formula to the attention rate of any vector
It obtains:
Wherein, aijAttention rate for j-th vector to i-th vector, eijFor bilinear function, TxFor i, exp
(eij) it is e using e the bottom of asijPower, j-th vector are any attention rate sentence vector of i-th vector, 0 < j≤i, 0 <
I≤b, i, j ∈ Z, b are the sentence vector number of article, and Z is set of integers.
Preferably, step S3 further comprises: according to second output quantity and the corresponding attention rate sentence vector pair
The attention rate of any vector obtains the semantic vector of the corresponding sentence of any vector by following formula:
Wherein, ciFor the semantic vector of the corresponding sentence of i-th vector, TsIt is the sentence vector number of article, a for b, bij
Attention rate for j-th vector to i-th vector, hjFor the first output quantity of j-th vector, 0 < j≤i, 0 < i≤
B, i, j ∈ Z, Z are set of integers.
Preferably, the bilinear function are as follows:
eij=ci-1Whj
Wherein, eijFor bilinear function, ci-1For the semantic vector of the corresponding sentence of (i-1)-th sentence vector, W hjPower
Weight matrix, W ∈ Rh*h, Rh*hMultiply the real number field of h for h, h ∈ R, R are set of real numbers, hjFor the first output quantity for j-th vector,
0 < j≤i, 0 < i≤b, i, j ∈ Z, b is the sentence vector number of article, and Z is set of integers.
Preferably, the weight matrix is obtained by back-propagation algorithm.
According to another aspect of the present invention, a kind of article semantic vector expression system is provided, comprising: obtain sentence vector mould
Block obtains the sentence vector of any sentence for all term vectors according to sentence any in the article；Obtain output quantity
Module, the corresponding sentence vector for arranging the corresponding sentence vector sum inverted order according to the sentence permutation with positive order of the article are equal
Input two-way length time memory network model, obtain the first output quantity for corresponding to any vector and any sentence to
Measure the second output quantity of corresponding attention rate sentence vector, wherein the corresponding attention rate sentence vector of any vector are as follows: in institute
Under the sentence vector permutation with positive order for stating article, the sentence vector before any vector described in any vector sum is constituted at least
One sentence vector；Obtain sentence semantics vector module, for according to second output quantity and the corresponding attention rate sentence to
The attention rate to any vector is measured, the semantic vector of the corresponding sentence of any vector is obtained；Obtain article language
Adopted vector module obtains the article language for the semantic vector according to the corresponding sentence of all vectors in the article
Adopted vector.
Preferably, the acquisition output quantity module is further used for: by pair according to the sentence permutation with positive order of the article
The sentence vector answered inputs two-way length time memory network model, obtain first sentence vector to any vector language
The semantic information of adopted information and first sentence vector to the attention rate sentence vector；By the sentence inverted order according to the article
The corresponding sentence vector of arrangement inputs two-way length time memory network model, obtains the last one vector to described any
Sentence vector semantic information and the last one described vector to the attention rate sentence vector semantic information；By described first
Sentence vector is believed to the semantic information of any vector and the semanteme of the last one described vector to any vector
Breath integration, obtains the first output quantity for corresponding to any vector, also, by first sentence vector to the attention rate sentence
The semantic information of vector and the semantic information of the last one described vector to the attention rate sentence vector are integrated, and acquisition corresponds to
Second output quantity of the attention rate sentence vector.
Preferably, the acquisition sentence semantics vector module is further used for: according to second output quantity and described right
The attention rate sentence vector answered obtains the corresponding sentence of any vector by following formula to the attention rate of any vector
Semantic vector:
Wherein, ciFor the semantic vector of the corresponding sentence of i-th vector, TsIt is the sentence vector number of article, a for b, bij
Attention rate for j-th vector to i-th vector, hjFor the first output quantity of j-th vector, 0 < j≤i, 0 < i≤
B, i, j ∈ Z, Z are set of integers.
A kind of article semantic vector representation method provided by the invention and system obtain owning in article by being arranged
Sentence vector carries out operating last acquisition article semantic vector by distich vector, so that in paragraph extraction of semantics stage totality
Calculation amount, which has, to be greatly reduced, and is solved semantic vector of the tradition based on term vector and extracted the paragraph language that cannot achieve
The function that justice indicates, the semantic vector extracted include all useful information of entire article.
Detailed description of the invention
Fig. 1 is a kind of schematic diagram of the article semantic vector representation method based on term vector in the prior art；
Fig. 2 is a kind of signal of the article semantic vector representation method based on term vector attention model in the prior art
Figure；
Fig. 3 is the flow chart of one of embodiment of the present invention article semantic vector representation method；
Fig. 4 is the schematic diagram that the first output quantity in the embodiment of the present invention obtains；
Fig. 5 is the acquisition schematic diagram of the article semantic vector in the embodiment of the present invention
Fig. 6 is the module map that one of embodiment of the present invention article semantic vector indicates system.
Specific embodiment
With reference to the accompanying drawings and examples, specific embodiments of the present invention will be described in further detail.Implement below
Example is not intended to limit the scope of the invention for illustrating the present invention.
Fig. 3 is the flow chart of one of embodiment of the present invention article semantic vector representation method, as shown in Figure 3, comprising:
S1, according to all term vectors of sentence any in the article, obtain the sentence vector of any sentence；S2, will be according to described
The corresponding sentence vector of the corresponding sentence vector sum inverted order arrangement of the sentence permutation with positive order of article inputs the two-way long short time
Memory network model, obtain the first output quantity for corresponding to any vector and the corresponding attention rate sentence of any vector to
Second output quantity of amount, wherein the corresponding attention rate sentence vector of any vector are as follows: in the sentence vector positive sequence of the article
Under arrangement, at least one vector of the sentence vector composition before any vector described in any vector sum；S3, basis
Second output quantity and the corresponding attention rate sentence vector obtain any sentence to the attention rate of any vector
The semantic vector of the corresponding sentence of vector；S4, according to the semantic vector of the corresponding sentence of all vectors in the article, obtain
Take the article semantic vector.
Specifically, all term vectors of any sentence are preferably obtained by Word2vector in article.
Further, sequence length has the problem of gap, those skilled in the art in the data to solve the problems, such as sequence
It devises Recognition with Recurrent Neural Network (recurrent neural network, RNN) and carrys out processing sequence problem.But common RNN
There are problems that two.First is that long-distance dependence, second is that gradient disappears and gradient explosion, this problem is when handling long sequence
It is particularly evident.
In order to solve problem above, those skilled in the art proposes length time memory network (Long Short-
Term Memory, LSTM) model.This RNN framework disappears and gradient explosion issues dedicated for solving the gradient of RNN model.
By the state of activation of three multiplication gate control block of memory: input gate (input gate), is forgotten out gate (output gate)
Door (forget gate).This structure can be allowed to the preceding information preservation inputted in a network, and the transmitting that goes ahead, input
Door, which opens stylish input, can just change the historic state saved in network, and the historic state that out gate saves when opening can be interviewed
Ask, and the output after influencing, forget for emptying previously stored historical information.
Due to the input of LSTM model be it is unidirectional, often have ignored following contextual information.Two-way length is in short-term
Between the basic thought of memory network be using a training sequence one LSTM model of each training backward forward.Again by two models
Output carry out linear combination, can completely rely on all contextual informations to reach each node in sequence.
Specifically, described in step S2 under the sentence vector permutation with positive order of the article, described in any vector sum
At least one vector that sentence vector before any vector is constituted refers to, if any vector be article first sentence to
Amount, then the attention rate sentence vector of any vector is first sentence vector, only includes a sentence vector.If any vector is not
First sentence vector of article, then the attention rate sentence vector of any vector is under permutation with positive order from first sentence vector to any
Several vectors for including in sentence vector.
It should be noted that first sentence vector in the embodiment of the present invention be first sentence under permutation with positive order to
It measures, the last one vector in the embodiment of the present invention is the last one vector under permutation with positive order.The embodiment of the present invention
In several refer to two and more than two.
A kind of article semantic vector representation method provided by the invention, by be arranged obtain all sentences in article to
Amount carries out operating last acquisition article semantic vector by distich vector, so that in the calculating of paragraph extraction of semantics stage totality
Amount, which has, to be greatly reduced, and is solved semantic vector of the tradition based on term vector and extracted the paragraph semanteme table that cannot achieve
The function of showing, the semantic vector extracted include all useful information of entire article.
Based on the above embodiment, step S1 further comprises: by the same of all term vectors of sentence any in the article
Dimension point sums up, and obtains the sentence vector of any sentence.
Specifically, sentence vector is therefore can be used in no too many label obtained in the unlabelled data
In data, and it is based on sentence vector, each paragraph even every document can be mapped on unique vector.
Further, the term vector in sentence is the basic component units of a vector.For a sentence, packet
The dimension of each term vector contained is consistent.The same dimension point of all term vectors of sentence any in article is summed it up,
The sentence vector of any sentence can be obtained.The dimension of all term vectors of the sentence vector sum of any sentence sentence is consistent.
After obtaining sentence vector, below to the acquisition of the first output quantity and the second output quantity in above-described embodiment make into
The explanation of one step, herein it should be noted that the first output quantity and the second output quantity are by double in the embodiment of the present invention
To length time memory network model export amount, property having the same.
Fig. 4 is the schematic diagram that the first output quantity in the embodiment of the present invention obtains, referring to Fig. 4, in above-described embodiment
Step S2 further comprises: the corresponding sentence vector according to the sentence permutation with positive order of the article is inputted the two-way long short time
Memory network model obtains first sentence vector to the semantic information of any vector and first sentence vector to institute
State the semantic information of attention rate sentence vector；The corresponding sentence vector that sentence inverted order according to the article arranges is inputted two-way
Length time memory network model, obtain the last one vector to any vector semantic information and it is described last
Semantic information of a vector to the attention rate sentence vector；By first sentence vector to the semanteme of any vector
Information and the semantic information of the last one described vector to any vector are integrated, and are obtained and are corresponded to any vector
First output quantity, also, by the semantic information of first sentence vector to the attention rate sentence vector and it is described the last one
The semantic information of sentence vector to the attention rate sentence vector is integrated, and the second output for corresponding to the attention rate sentence vector is obtained
Amount.
Specifically, corresponding first output quantity of any vector.
Further, two-way length time memory network model (Bi-LSTM) has positive coding and phase-reversal coding
Function, it is described above that the corresponding sentence vector according to the sentence permutation with positive order of the article is inputted into two-way length time memory
Network model obtains first sentence vector to the semantic information of any vector and first sentence vector to the pass
The semantic information of note degree sentence vector is the process of positive coding.Correspondingly, the sentence inverted order described above by according to the article
The corresponding sentence vector of arrangement inputs two-way length time memory network model, obtains the last one vector to described any
The semantic information of sentence vector and the semantic information of the last one described vector to the attention rate sentence vector are phase-reversal coding
Process.Sentence vector is the basic unit of the input of two-way length time memory network model (Bi-LSTM).
Further, first sentence vector refers to any vector, the sentence vector of article under permutation with positive order, from
First sentence vector sequentially sequence to any vector.Correspondingly, the last one vector refers to any vector,
The sentence of article is under permutation with positive order, from the last one vector against order to any vector.
Further, first sentence vector to the attention rate sentence vector refer to first sentence vector to any sentence to
Each attention rate sentence vector of amount.The last one vector refers to the last one vector described in the attention rate sentence vector
Each attention rate sentence vector of any vector.
Further, for any vector, two-way length time memory network model (Bi-LSTM) is positive
Coding and the semantic information of phase-reversal coding integrate, the first output quantity of any vector of acquisition contain this vector it
The semantic information of preceding sentence vector and the semantic information of the sentence vector after this vector.
It should be noted that VDThe semantic vector that can also indicate article to a certain extent, but work as paragraph sentence amount
Following problem is had when excessive: the information of entire article sequence can not be remembered and be indicated to semantic information；Due to the shadow of all vocabulary
It is similar to ring the factor, article emphasis and central idea study inaccuracy can not be protruded.Based on this, present invention further proposes steps
S3。
It is illustrated in above-described embodiment according to second output quantity and the corresponding attention rate sentence vector to described
The attention rate of one vector obtains the semantic vector of the corresponding sentence of any vector, and the present embodiment is to corresponding concern
Degree sentence vector makes explanations to the acquisition modes of the attention rate of any vector.
Attention rate of the corresponding attention rate sentence vector described in step S3 in above-described embodiment to any vector
It is obtained by following formula:
Wherein, aijAttention rate for j-th vector to i-th vector, eijFor bilinear function, TxFor i, exp
(eij) it is e using e the bottom of asijPower, j-th vector are any attention rate sentence vector of i-th vector, 0 < j≤i, 0 <
I≤b, i, j ∈ Z, b are the sentence vector number of article, and Z is set of integers.
Specifically, attention rate is the degree of concern, is also believed to impacted degree.
A kind of article semantic vector representation method provided by the invention calculates attention rate sentence vector to described by setting
The attention rate of one vector can know the degree that influences each other between the sentence in article, so that the article semantic vector obtained
It is more accurate.
Below based on above-described embodiment, step S3 is made and is further described.
Specifically, step S3 further comprises: according to second output quantity and the corresponding attention rate sentence vector pair
The attention rate of any vector obtains the semantic vector of the corresponding sentence of any vector by following formula:
Wherein, ciFor the semantic vector of the corresponding sentence of i-th vector, TsIt is the sentence vector number of article, a for b, bij
Attention rate for j-th vector to i-th vector, hjFor the first output quantity of j-th vector, 0 < j≤i, 0 < i≤
B, i, j ∈ Z, Z are set of integers.
Further, the h in the embodiment of the present inventionjAnd VDAll refer to the first output quantity.
A kind of article semantic vector representation method provided by the invention, by being arranged according to the second output quantity and corresponding pass
Note degree sentence vector obtains the semantic vector of the corresponding sentence of any vector to the attention rate of any vector, can be effective right
The contribution degree of each sentence makes differentiation.
Based on the above embodiment, the bilinear function are as follows:
eij=ci-1Whj
Wherein, eijFor bilinear function, ci-1For the semantic vector of the corresponding sentence of (i-1)-th sentence vector, W hjPower
Weight matrix, W ∈ Rh*h, Rh*hMultiply the real number field of h for h, h ∈ R, R are set of real numbers, hjFor the first output quantity for j-th vector,
0 < j≤i, 0 < i≤b, i, j ∈ Z, b is the sentence vector number of article, and Z is set of integers.
A kind of article semantic vector representation method provided by the invention is compared to it by the way that bilinear function is arranged
The performance of neural network can be preferably promoted based on forward direction neural activation function used in term vector attention model, and
It is applied to as better result can be obtained under paragraph center extraction network.
Based on the above embodiment, the weight matrix is obtained by back-propagation algorithm.
Specifically, back-propagation algorithm (Backpropagation algorithm, BP algorithm) is a kind of supervised learning calculation
Method is often used to train multi-layer perception (MLP).BP algorithm is the popularization of Delta rule, it is desirable that used in each artificial neuron
Function must can be micro-.BP algorithm is particularly suitable for being used to Training Multilayer Neural Network.
As a preferred embodiment, Fig. 5 is the acquisition schematic diagram of the article semantic vector in the embodiment of the present invention, please be joined
Fig. 5 is read, the acquisition of article semantic vector specifically may include following step:
Firstly, the same dimension point of all term vectors of sentence any in article is summed it up, any sentence is obtained
Sentence vector.
Secondly, the corresponding sentence vector of the sentence permutation with positive order according to article is inputted two-way length time memory network
Model, obtain first sentence vector to any vector semantic information and first sentence vector to attention rate sentence vector semanteme
Information；The corresponding sentence vector arranged according to the sentence inverted order of article is inputted into two-way length time memory network model, is obtained
The last one vector is taken to believe to the semantic information of any vector and the semanteme of the last one vector to attention rate sentence vector
Breath；By the semantic information of the semantic information of first sentence vector to any vector and the last one vector to any vector
Integration obtains the first output quantity for corresponding to any vector, also, by the semanteme of first sentence vector to attention rate sentence vector
Information and the semantic information of the last one vector to attention rate sentence vector are integrated, and are obtained and are corresponded to the second of attention rate sentence vector
Output quantity.
Then, following formula is passed through to the attention rate of any vector according to the second output quantity and corresponding attention rate sentence vector
Obtain the semantic vector of the corresponding sentence of any vector:
Wherein, ciFor the semantic vector of the corresponding sentence of i-th vector, TsIt is the sentence vector number of article, a for b, bij
Attention rate for j-th vector to i-th vector, hjFor the first output quantity of j-th vector, 0 < j≤i, 0 < i≤
B, i, j ∈ Z, Z are set of integers.
A in above formulaijIt is obtained by following formula:
Wherein, aijAttention rate for j-th vector to i-th vector, eijFor bilinear function, TxFor i, exp
(eij) it is e using e the bottom of asijPower, j-th vector are any attention rate sentence vector of i-th vector, 0 < j≤i, 0 <
I≤b, i, j ∈ Z, b are the sentence vector number of article, and Z is set of integers.
Finally, the semantic vector of the corresponding sentence of all vectors in article is obtained, according to all sentences in article
The semantic vector of the corresponding sentence of vector obtains article semantic vector.
Based on the above embodiment, the present invention also provides a kind of article semantic vectors to indicate system, to implement above-mentioned
The article semantic vector representation method of one embodiment, Fig. 6 are that one of embodiment of the present invention article semantic vector indicates system
Module map, as shown in Figure 6, comprising: obtain sentence vector module, for according to all words of sentence any in the article to
Amount obtains the sentence vector of any sentence；Output quantity module is obtained, for will be according to the sentence permutation with positive order of the article
The corresponding sentence vector of corresponding sentence vector sum inverted order arrangement inputs two-way length time memory network model, obtains and corresponds to
In the first output quantity of any vector and the second output quantity of the corresponding attention rate sentence vector of any vector, wherein
The corresponding attention rate sentence vector of any vector are as follows: under the sentence vector permutation with positive order of the article, any sentence to
At least one vector that sentence vector before amount and any vector is constituted；Sentence semantics vector module is obtained, is used for
According to second output quantity and the corresponding attention rate sentence vector to the attention rate of any vector, described appoint is obtained
The semantic vector of the corresponding sentence of one vector；Article semantic vector module is obtained, for according to all sentences in the article
The semantic vector of the corresponding sentence of vector obtains the article semantic vector.
Based on the above embodiment, the acquisition output quantity module is further used for: by the sentence positive sequence according to the article
The corresponding sentence vector of arrangement inputs two-way length time memory network model, obtains first sentence vector to any sentence
The semantic information of the semantic information of vector and first sentence vector to the attention rate sentence vector；It will be according to the article
The corresponding sentence vector of sentence inverted order arrangement inputs two-way length time memory network model, obtains the last one vector and arrives
The semantic information of the semantic information of any vector and the last one described vector to the attention rate sentence vector；By institute
First sentence vector is stated to the semantic information of any vector and the last one described vector to any vector
Semantic information integration, obtain correspond to any vector the first output quantity, also, will first sentence vector to described in
The semantic information of attention rate sentence vector and the semantic information of the last one described vector to the attention rate sentence vector are integrated, and are obtained
Take the second output quantity corresponding to the attention rate sentence vector.
Based on the above embodiment, the acquisition sentence semantics vector module is further used for: according to second output quantity
With the corresponding attention rate sentence vector to the attention rate of any vector, any vector pair is obtained by following formula
The semantic vector for the sentence answered:
Wherein, ciFor the semantic vector of the corresponding sentence of i-th vector, TsIt is the sentence vector number of article, a for b, bij
Attention rate for j-th vector to i-th vector, hjFor the first output quantity of j-th vector, 0 < j≤i, 0 < i≤
B, i, j ∈ Z, Z are set of integers.
A kind of article semantic vector representation method provided by the invention and system obtain owning in article by being arranged
Sentence vector carries out operating last acquisition article semantic vector by distich vector, so that in paragraph extraction of semantics stage totality
Calculation amount, which has, to be greatly reduced, and is solved semantic vector of the tradition based on term vector and extracted the paragraph language that cannot achieve
The function that justice indicates, the semantic vector extracted include all useful information of entire article.Attention rate is calculated by setting
Sentence vector can know the degree that influences each other between the sentence in article to the attention rate of any vector, so that obtaining
Article semantic vector it is more accurate.By being arranged according to the second output quantity and corresponding attention rate sentence vector to any vector
Attention rate, obtain the semantic vector of the corresponding sentence of any vector, can the contribution degree effectively to each sentence make area
Point.By the way that bilinear function is arranged, it is compared to it based on forward direction neural activation function used in term vector attention model
The performance of neural network can be preferably promoted, and can be obtained in the case where being applied to such as paragraph center extraction network better
As a result.
Finally, method of the invention is only preferable embodiment, it is not intended to limit the scope of the present invention.It is all
Within the spirit and principles in the present invention, any modification, equivalent replacement, improvement and so on should be included in protection of the invention
Within the scope of.
Claims (10)
1. a kind of article semantic vector representation method characterized by comprising
S1, according to all term vectors of sentence any in the article, obtain the sentence vector of any sentence；
S2, the corresponding sentence vector for arranging the corresponding sentence vector sum inverted order according to the sentence permutation with positive order of the article are defeated
Enter two-way length time memory network model, obtains the first output quantity for corresponding to any vector and any vector
Second output quantity of corresponding attention rate sentence vector, wherein the corresponding attention rate sentence vector of any vector are as follows:
Sentence vector structure under the sentence vector permutation with positive order of the article, before any vector described in any vector sum
At at least one vector；
S3, according to second output quantity and the corresponding attention rate sentence vector to the attention rate of any vector, obtain
Take the semantic vector of the corresponding sentence of any vector；
S4, according to the semantic vector of the corresponding sentence of all vectors in the article, obtain the article semantic vector.
2. representation method according to claim 1, which is characterized in that step S1 further comprises:
The same dimension point of all term vectors of sentence any in the article is summed it up, the sentence of any sentence is obtained
Vector.
3. representation method according to claim 1, which is characterized in that step S2 further comprises:
Corresponding sentence vector according to the sentence permutation with positive order of the article is inputted to two-way length time memory network model,
Obtain first sentence vector to any vector semantic information and first sentence vector to the attention rate sentence to
The semantic information of amount；
The corresponding sentence vector that sentence inverted order according to the article arranges is inputted into two-way length time memory network model,
The last one vector is obtained to the semantic information of any vector and the last one described vector to the attention rate
The semantic information of sentence vector；
The semantic information of first sentence vector to any vector and the last one described vector are appointed to described
The semantic information integration of one vector, obtains the first output quantity for corresponding to any vector, also, by first sentence to
The semanteme of the semantic information and the last one described vector to the attention rate sentence vector of measuring the attention rate sentence vector is believed
Breath integration, obtains the second output quantity for corresponding to the attention rate sentence vector.
4. representation method according to claim 1, which is characterized in that corresponding attention rate sentence vector pair described in step S3
The attention rate of any vector is obtained by following formula:
Wherein, aijAttention rate for j-th vector to i-th vector, eijFor bilinear function, TxFor i, exp (eij) be
Using e as the e at bottomijPower, any attention rate sentence vector of j-th vector for i-th vector, 0 < j≤i, 0 < i≤b, i,
J ∈ Z, b are the sentence vector number of article, and Z is set of integers.
5. representation method according to claim 4, which is characterized in that step S3 further comprises:
According to second output quantity and the corresponding attention rate sentence vector to the attention rate of any vector, under
Formula obtains the semantic vector of the corresponding sentence of any vector:
Wherein, ciFor the semantic vector of the corresponding sentence of i-th vector, TsIt is the sentence vector number of article, a for b, bijFor jth
Attention rate of a vector to i-th vector, hjFor the first output quantity of j-th vector, 0 < j≤i, 0 < i≤b, i, j
∈ Z, Z are set of integers.
6. representation method according to claim 4, which is characterized in that the bilinear function are as follows:
eij=ci-1Whj
Wherein, eijFor bilinear function, ci-1For the semantic vector of the corresponding sentence of (i-1)-th sentence vector, W hjWeight square
Battle array, W ∈ Rh*h, Rh*hMultiply the real number field of h for h, h ∈ R, R are set of real numbers, hjFor the first output quantity for j-th vector, 0 < j
≤ i, 0 < i≤b, i, j ∈ Z, b is the sentence vector number of article, and Z is set of integers.
7. representation method according to claim 6, which is characterized in that the weight matrix is obtained by back-propagation algorithm
It takes.
8. a kind of article semantic vector indicates system characterized by comprising
Sentence vector module is obtained, for all term vectors according to sentence any in the article, obtains any sentence
Sentence vector；
Output quantity module is obtained, for arrange the corresponding sentence vector sum inverted order according to the sentence permutation with positive order of the article
Corresponding sentence vector inputs two-way length time memory network model, obtains the first output quantity for corresponding to any vector
Second output quantity of attention rate sentence vector corresponding with any vector, wherein the corresponding concern of any vector
Spend sentence vector are as follows:
Sentence vector structure under the sentence vector permutation with positive order of the article, before any vector described in any vector sum
At at least one vector；
Sentence semantics vector module is obtained, is used for according to second output quantity and the corresponding attention rate sentence vector to described
The attention rate of any vector obtains the semantic vector of the corresponding sentence of any vector；
Article semantic vector module is obtained, for the semantic vector according to the corresponding sentence of all vectors in the article,
Obtain the article semantic vector.
9. expression system according to claim 8, which is characterized in that the acquisition output quantity module is further used for: will
The corresponding sentence vector of sentence permutation with positive order according to the article inputs two-way length time memory network model, obtains the
One sentence vector to any vector semantic information and first sentence vector to the attention rate sentence vector language
Adopted information；
The corresponding sentence vector that sentence inverted order according to the article arranges is inputted into two-way length time memory network model,
The last one vector is obtained to the semantic information of any vector and the last one described vector to the attention rate
The semantic information of sentence vector；
The semantic information of first sentence vector to any vector and the last one described vector are appointed to described
The semantic information integration of one vector, obtains the first output quantity for corresponding to any vector, also, by first sentence to
The semanteme of the semantic information and the last one described vector to the attention rate sentence vector of measuring the attention rate sentence vector is believed
Breath integration, obtains the second output quantity for corresponding to the attention rate sentence vector.
10. expression system according to claim 8, which is characterized in that the acquisition sentence semantics vector module is further
For:
According to second output quantity and the corresponding attention rate sentence vector to the attention rate of any vector, under
Formula obtains the semantic vector of the corresponding sentence of any vector:
Wherein, ciFor the semantic vector of the corresponding sentence of i-th vector, TsIt is the sentence vector number of article, a for b, bijFor jth
Attention rate of a vector to i-th vector, hjFor the first output quantity of j-th vector, 0 < j≤i, 0 < i≤b, i, j
∈ Z, Z are set of integers.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711024027.4A CN109726383B (en) | 2017-10-27 | 2017-10-27 | Article semantic vector representation method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711024027.4A CN109726383B (en) | 2017-10-27 | 2017-10-27 | Article semantic vector representation method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109726383A true CN109726383A (en) | 2019-05-07 |
| CN109726383B CN109726383B (en) | 2023-06-23 |
Family
ID=66290802
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711024027.4A Active CN109726383B (en) | 2017-10-27 | 2017-10-27 | Article semantic vector representation method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109726383B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112784597A (en) * | 2019-11-06 | 2021-05-11 | 阿里巴巴集团控股有限公司 | Method and device for evaluating quality of article |
| CN113761182A (en) * | 2020-06-17 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Method and device for determining service problem |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007140602A (en) * | 2005-11-14 | 2007-06-07 | Nippon Telegr & Teleph Corp <Ntt> | Topic extraction method, device and program, and computer readable recording medium |
| JP2009086858A (en) * | 2007-09-28 | 2009-04-23 | Nippon Telegr & Teleph Corp <Ntt> | Content-retrieving device, content-retrieving method, program, and recording medium |
| CN106919646A (en) * | 2017-01-18 | 2017-07-04 | 南京云思创智信息科技有限公司 | Chinese text summarization generation system and method |
-
2017
- 2017-10-27 CN CN201711024027.4A patent/CN109726383B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007140602A (en) * | 2005-11-14 | 2007-06-07 | Nippon Telegr & Teleph Corp <Ntt> | Topic extraction method, device and program, and computer readable recording medium |
| JP2009086858A (en) * | 2007-09-28 | 2009-04-23 | Nippon Telegr & Teleph Corp <Ntt> | Content-retrieving device, content-retrieving method, program, and recording medium |
| CN106919646A (en) * | 2017-01-18 | 2017-07-04 | 南京云思创智信息科技有限公司 | Chinese text summarization generation system and method |
Non-Patent Citations (1)
| Title |
|---|
| 赵宇晴等: "基于分层编码的深度增强学习对话生成", 《计算机应用》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112784597A (en) * | 2019-11-06 | 2021-05-11 | 阿里巴巴集团控股有限公司 | Method and device for evaluating quality of article |
| CN113761182A (en) * | 2020-06-17 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Method and device for determining service problem |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109726383B (en) | 2023-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106650813B (en) | A kind of image understanding method based on depth residual error network and LSTM | |
| CN112818861B (en) | Emotion classification method and system based on multi-mode context semantic features | |
| CN109284506A (en) | A kind of user comment sentiment analysis system and method based on attention convolutional neural networks | |
| CN112487182A (en) | Training method of text processing model, and text processing method and device | |
| CN111507378A (en) | Method and apparatus for training image processing model | |
| CN107247702A (en) | A kind of text emotion analysis and processing method and system | |
| CN109492227A (en) | It is a kind of that understanding method is read based on the machine of bull attention mechanism and Dynamic iterations | |
| CN110287320A (en) | A kind of deep learning of combination attention mechanism is classified sentiment analysis model more | |
| CN112990296B (en) | Image-text matching model compression and acceleration method and system based on orthogonal similarity distillation | |
| CN112084331A (en) | Text processing method, text processing device, model training method, model training device, computer equipment and storage medium | |
| CN110222163A (en) | A kind of intelligent answer method and system merging CNN and two-way LSTM | |
| CN112257449B (en) | Named entity recognition method and device, computer equipment and storage medium | |
| CN109284406A (en) | Intension recognizing method based on difference Recognition with Recurrent Neural Network | |
| CN111881677A (en) | Address matching algorithm based on deep learning model | |
| CN108959556A (en) | Entity answering method, device and terminal neural network based | |
| CN110750998B (en) | Text output method, device, computer equipment and storage medium | |
| CN112131883A (en) | Language model training method and device, computer equipment and storage medium | |
| CN109492223A (en) | A kind of Chinese missing pronoun complementing method based on ANN Reasoning | |
| CN111259668B (en) | Reading task processing method, model training device and computer equipment | |
| CN111858940A (en) | Multi-head attention-based legal case similarity calculation method and system | |
| CN117218498B (en) | Multi-modal large language model training method and system based on multi-modal encoder | |
| CN108197707A (en) | Compression method based on the convolutional neural networks that global error is rebuild | |
| CN113505193A (en) | Data processing method and related equipment | |
| CN111695591A (en) | AI-based interview corpus classification method, device, computer equipment and medium | |
| CN113656563A (en) | Neural network searching method and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |