CN104915448B - A kind of entity based on level convolutional network and paragraph link method - Google Patents
A kind of entity based on level convolutional network and paragraph link method Download PDFInfo
- Publication number
- CN104915448B CN104915448B CN201510372795.3A CN201510372795A CN104915448B CN 104915448 B CN104915448 B CN 104915448B CN 201510372795 A CN201510372795 A CN 201510372795A CN 104915448 B CN104915448 B CN 104915448B
- Authority
- CN
- China
- Prior art keywords
- paragraph
- vectorization
- entity
- feature
- sentence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 68
- 239000013598 vector Substances 0.000 claims abstract description 51
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 41
- 238000012549 training Methods 0.000 claims abstract description 16
- 238000000605 extraction Methods 0.000 claims abstract description 8
- 230000006870 function Effects 0.000 claims description 33
- 230000008859 change Effects 0.000 claims description 10
- 239000011159 matrix material Substances 0.000 claims description 8
- 238000005070 sampling Methods 0.000 claims description 8
- 230000009466 transformation Effects 0.000 claims description 6
- 238000005516 engineering process Methods 0.000 claims description 5
- 238000011478 gradient descent method Methods 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 2
- 238000012545 processing Methods 0.000 claims description 2
- 230000004927 fusion Effects 0.000 claims 2
- 230000015572 biosynthetic process Effects 0.000 claims 1
- 238000012360 testing method Methods 0.000 abstract description 14
- 238000003062 neural network model Methods 0.000 abstract description 4
- 239000002585 base Substances 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 4
- 238000013139 quantization Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 239000012458 free base Substances 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 description 1
- 235000007926 Craterellus fallax Nutrition 0.000 description 1
- 240000007175 Datura inoxia Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000013517 stratification Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Fuzzy Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Databases & Information Systems (AREA)
- Machine Translation (AREA)
Abstract
A kind of entity based on level convolutional network and paragraph link method, including：Represent that changing into sentence vectorization represents by term vectorization using convolutional neural networks；Represent to again pass by convolutional neural networks using sentence vectorization and consider that the sentence order information obtains paragraph vectorization and represented；Sentence vectorization expression and paragraph vectorization represent to export by Softmax, carry out the training of the convolutional neural networks model as supervision message by existing entity；Meanwhile consider that the pair wise similarity informations between paragraph semantic vector feature and Entity Semantics vector characteristics further improve the training of convolutional neural networks model；A test description paragraph is given, carrying out Deep Semantics feature extraction using the neural network model trained obtains the vectorization expression of test paragraph, and being then based on this semantic expressiveness can be directly linked on target entity by Softmax outputs.
Description
Technical field
The present invention relates to construction of knowledge base technical field, relate more specifically to a kind of entity based on level convolutional network with
Paragraph link method.
Background technology
Nowadays, the large-scale knowledge base being widely used has Freebase, WordNet and YAGO etc..They are devoted to
A global resources bank is built, and allows machine more easily to access and obtains structuring public information.Meanwhile these are known
Know storehouse and provide application structure (APIs) to be convenient for people to inquire about the information of related entities more horn of plenty.For example, work as us
A city name is retrieved in YAGO databases, and " during Washington D.C. ", returning result is as shown in table 1 below：
Table 1
It can be seen that the object information of return is all the organizational information of some highly structurals.But these structured messages
And actual context and semantic information it is appreciated that entity are not met.Different with YAGO databases, Freebase and WordNet are then
The descriptive paragraph related to retrieval entity can be additionally returned while return structure information, it is as shown in table 2 below：
Table 2
It can be seen that descriptive paragraph as shown in table 2 be relatively beneficial to user understand query entity word specific linguistic context and
Semantic information.However, Freebase and WordNet descriptive paragraph information is all by manually entering edlin, this can cause
The limitation of paragraph description is carried out to entity under big data and taken a substantial amount of time and manpower.Therefore, a height how is designed
The entity of effect and the task that descriptive paragraph AutoLink method is that big data epoch construction of knowledge base is urgently needed badly.
It can further be seen that descriptive content is not necessarily to include query entity word from the returning result of table 2, and only need
Entity is carried out comprising some related terms to describe in many aspects.Therefore, in order to solve this problem, entity and paragraph side of link
Method needs to set about in terms of two：1st, the subject information of text is caught from a given segment description paragraph；2nd, find and real
The important descriptive content that body phase is closed.It is the subject information that paragraph is extracted based on topic model method mostly to compare traditional method,
The latent semantic analysis (PLSA) of such as Di Li Crays distribution (LDA) and probability.The common problem of these methods is the theme letter extracted
Breath is the Term co-occurrence information acquisition based on document level, and the high of short text character representation openness influences to compare in by social media
Seriously, and it lost the word order information in text.
In recent years, with the rise of deep neural network, some researchers attempt to use depth model and term vector table
Show that the deep layer latent semantic feature of the descriptive paragraph of method study is represented to solve the link problems of entity and paragraph.It is however, existing
Some, when the semantic feature for solving descriptive paragraph extracts, is simply simply regarded whole paragraph as based on depth model method
One long sentence, which is handled or directly multiple sentences are weighted, averagely obtains semantic vector.And in fact, sentence in paragraph
Sub- order is also with Semantic logical relation.
On the other hand, it is also very important to catch descriptive clue closely related with entity in paragraph.Such as above-mentioned table 2
Although descriptive paragraph in returning result directly comprising query entity word " Washington D.C. ", but comprising
Many related vocabulary or phrase, such as：" George Washington ", " United States " and " capital " etc..Cause
This, carry out that vectorization character representation contributes to entity and descriptive paragraph to entity links work.
The content of the invention
For above-mentioned technical problem, it is a primary object of the present invention to provide a kind of entity based on level convolutional network with
Paragraph link method, so as to need not participation be manually AutoLink by the entity word in internet and descriptive paragraph, have
Help the structure of the semantic knowledge-base under big data.
To achieve these goals, the invention provides a kind of entity based on level convolutional network and paragraph side of link
Method, comprise the following steps：
Represent that changing into sentence vectorization represents by term vectorization using convolutional neural networks, the convolutional network is favourable
In important clue of the extraction query entity in paragraph is described；
The sentence vectorization represents to again pass by convolutional neural networks and consider that the sentence order information obtains paragraph
Vectorization represents；
The sentence vectorization represents and the paragraph vectorization represents to export by Softmax, makees by existing entity
The training of the convolutional neural networks model is carried out for supervision message；
The pair-wise similarities letter between the paragraph semantic vector feature and Entity Semantics vector characteristics is considered simultaneously
Breath further improves the training of the convolutional neural networks model；
A test description paragraph is given, Deep Semantics feature extraction is carried out using the neural network model trained
The vectorization for obtaining the test paragraph represents that mesh can be directly linked to by Softmax outputs by being then based on this semantic expressiveness
Mark is physically.
Feature learning problem in the linking of entity and paragraph is divided into four by entity and the paragraph link method of the present invention
Individual level, it is respectively：The eigenmatrix layer that urtext paragraph represents to obtain by term vector；Obtained by convolutional neural networks
The sentence vectorization arrived represents characteristic layer；The paragraph vectorization obtained by convolutional neural networks represents characteristic layer；Using word to
The vectorization that amount look-up table obtains entity word represents characteristic layer.Tabled look-up by convolution character network and term vector, side of the invention
Method accuracy value ACC of entity and paragraph link method on two text data sets is significantly superior to other control methods, and phase
For best control methods two, accuracy value of the inventive method in two datasets improves 12.4% He respectively
16.76%.
Brief description of the drawings
Fig. 1 is the flow of the entity and paragraph link method based on level convolutional network as one embodiment of the invention
Figure；
Fig. 2 is the framework of the entity and paragraph link method based on level convolutional network as one embodiment of the invention
Schematic diagram；
Fig. 3 is the performance of the entity and paragraph link method based on level convolutional network as one embodiment of the invention
Schematic diagram.
Embodiment
For the object, technical solutions and advantages of the present invention are more clearly understood, below in conjunction with specific embodiment, and reference
Accompanying drawing, the present invention is described in further detail.
, can be by internet the invention discloses a kind of entity based on level convolutional network and paragraph link method
Entity word proceeds without artificial participation ground AutoLink with descriptive paragraph, and its general plotting is to pass through stratification convolutional Neural
Network first carries out convolution to the term vector in paragraph and obtains the vectorization expression of sentence.Consider order letter of the sentence in paragraph
Breath, and the vectorization to sentence represents to carry out convolution again and obtains the vectorization expression of paragraph.Then using substance feature as
Supervision message instructs the parameter learning of convolutional neural networks model, at the same consider the deep semantic feature of paragraph and Entity Semantics to
Pair-wise affinity informations between quantization means improve the study of convolutional neural networks model.Given one new descriptive
Paragraph, then using convolutional neural networks model extraction its deep semantic feature trained, and export to obtain based on this feature
Corresponding entity link.
More specifically, this method represents to change into sentence vectorization table first with convolutional neural networks by term vectorization
Show.Then using sentence vectorization represent again pass by convolutional neural networks and consider the sentence order information obtain paragraph to
Quantization means.Sentence vectorization expression and paragraph vectorization represent to export by Softmax, believe by existing entity as supervision
Breath carries out the training of the convolutional neural networks model.Meanwhile consider paragraph semantic vector feature and Entity Semantics vector characteristics
Between pair-wise similarity informations further improve the training of convolutional neural networks model.Give a test description section
Fall, carrying out Deep Semantics feature extraction using the neural network model trained obtains the vectorization expression of test paragraph, then
It can be directly linked to based on this semantic expressiveness by Softmax outputs on target entity.
The entity based on level convolutional network as one embodiment of the invention is linked with paragraph below in conjunction with the accompanying drawings
Method is described in detail.
Fig. 1 is the flow of the entity and paragraph link method based on level convolutional network as one embodiment of the invention
Figure.
Reference picture 1, in step S101, represented by convolutional neural networks model and term vectorization, extract pending paragraph
In the vectorization of every sentence represent feature；
It is described to be represented by convolutional neural networks model and term vectorization according to one exemplary embodiment of the present invention,
Extracting in pending paragraph the step of the vectorization of every sentence represents feature includes：
In step S1011, a sentence in pending paragraph is given, lexical item quantization means is obtained using look-up table and incites somebody to action
Sentence is characterized into matrix form；
In step S1012, represent to carry out one-dimensional convolution in feature in the sentence matrixing, obtain the feature square after convolution
Battle array；
Average sampling is carried out on step S1013, the eigenmatrix after the convolution to be compressed feature, obtains sentence
The vectorization of son represents.
It is described to obtain lexical item quantization means and by sentence table using look-up table according to one exemplary embodiment of the present invention
The step of levying into matrix form includes：
Give the term vector set that a word2vec is trainedWherein, | V | be dictionary size, d be word to
The dimension of amount.Then length is that n sentence can be expressed as in any paragraph：
S=(x1；x2；...；xn) (1)
Wherein, xiIt is that vectorization corresponding to i-th of the word found using look-up table in term vector set is represented.Wherein,
If word xiDo not appear in the term vector set trained, then directly it is carried out in the exemplary embodiment of the present invention
Random initializtion represents.
It is described to represent to carry out one-dimensional convolution in feature in sentence matrixing in step S1012, obtain the feature square after convolution
The step of battle array, includes：
Here, useRepresent in sentence s from the h of i-th of word startingsIndividual continuous word is special
Sign.Give an one-dimensional convolution kernelThen hsEigenmatrix after individual continuous word feature convolution is：
Wherein, b(1)It is bias term, f is activation primitive,It is hsIndividual continuous word featureConvolution
Eigenmatrix afterwards.Then the eigenmatrix of the sentence is after convolution：
Average sampling is carried out on step S1013, the eigenmatrix after convolution to be compressed feature, obtains sentence
The step of vectorization of son represents includes：
The present invention the exemplary embodiment in, it is described use average sampling the step of for：
So far, each convolution kernelA d dimensional feature vector can be generatedIf having used k convolution kernel, pass through
A convolutional layer is crossed, the vectorization that will eventually get sentence is expressed asThe dimension that then sentence vectorization represents
Spend for dk.
In step S102, represented using convolutional neural networks structure and the sentence vectorization, learn the depth of the paragraph
Spend semantic feature；
According to one exemplary embodiment of the present invention, the deep semantic feature learning method of the paragraph includes：
In step S1021, word order of the sentence in the paragraph is pressed by section using the sentence vector characteristics in the paragraph
Fall to characterize into matrix form；
In step S1022, represent to carry out one-dimensional convolution in feature in the paragraph matrixing, obtain the feature square after convolution
Battle array；
Average sampling is carried out on step S1023, the eigenmatrix after the convolution to be compressed feature and carry out
Once linear converts, and the vectorization for obtaining paragraph represents.
According to one exemplary embodiment of the present invention, the sentence vector characteristics using in paragraph are by sentence described
Word order in paragraph, which characterizes paragraph into the step of matrix form, to be included：
The vectorization for having obtained the l bar sentences of the paragraph represents that then paragraph can be expressed as：
T=(s1；s2；...；sl) (5)
It is described to represent to carry out one-dimensional convolution in feature in paragraph matrixing in step S1022, obtain the feature square after convolution
The step of battle array, includes：
Here, useRepresent in paragraph t from the h of i-th of sentence startingtIndividual continuous sentence
Feature.Give an one-dimensional convolution kernelThen htConvolution after individual continuous sentence characteristics convolution is characterized as：
Wherein, b(2)It is bias term, f is activation primitive,It is htIndividual continuous sentence characteristicsVolume
Feature after product.Then the feature of the paragraph is after convolution：
Average sampling is carried out on step S1023, the eigenmatrix after convolution to be compressed feature and carry out
Once linear convert, obtain paragraph vectorization represent the step of include：
The present invention the exemplary embodiment in, it is described use average sampling the step of for：
So far, by convolution kernel W(2)Generate a dk dimensional feature vectorIn order to facilitate calculating paragraph feature and reality
The similitude of body characteristicses, the unification of vector dimension need to be ensured, then once linear conversion be carried out to paragraph vector：
Wherein,For the matrix of a linear transformation, and characteristic vector z is one exemplary embodiment of the present invention
In final paragraph characteristic vector.
In step S103, the vectorization of the sentence represents and the vectorization of the paragraph represents to pass through Softmax respectively
Output is fitted the affiliated entity of paragraph；
According to one exemplary embodiment of the present invention, the vectorization of the sentence and the paragraph represents to be fitted institute respectively
The method for stating the affiliated entity of paragraph comprises the following steps：
In step S1031, linear transformation is carried out respectively to the sentence vector sum paragraph vector and obtains output vector, and is made
Canonical is carried out with Dropout technologies；
In step S1032, the link probability of Softmax functions calculating candidate's entity is used；
According to one exemplary embodiment of the present invention, distich subvector and the paragraph vector carries out linear transformation and obtained
Output vector, and included using the step of Dropout technologies progress canonical：
Distich subvector feature s and paragraph vector characteristics t carry out linear change respectively, obtain two output vectors：
Ys=W(4)·(sοr)+b(4) (10)
Y=W(5)·(zοr)+b(5) (11)
Wherein,WithIt is weight matrix, m is in one exemplary embodiment of the present invention
Entity number, symbol.Representing matrix element multiplies operation, andIt is then a Bernoulli Jacob for obeying certain probability ρ point
Cloth.Over-fitting can be prevented using Dropout technologies, can be with the robustness of strength neural network model.
In step S1032, the step of calculating the link probability of candidate's entity using the Softmax functions, includes：
Swashed respectively using Softmax in two output layers of the sentence vector characteristics and the paragraph vector characteristics
Function living calculates the probable value for each corresponding to the entity word：
Then in formula (12) and formula (13), psiAnd piThe probable value of corresponding i-th of entity word is represented respectively.
In step S104, the vectorization for calculating the entity represents the pair-wise phases represented with the paragraph vectorization
Like information；
Give an entity set of words E={ e1, e2..., em, the entity set of words is carried out just using word2vec
Beginningization, then entity set of words E and the paragraph characteristic vector z similitude be：
Sim (z, E)={ ze1, ze2..., zem} (14)
Wherein, operator ze represents the paragraph characteristic vector z and the corresponding entity word e similitude.
In step S105, pass through Softmax fit objects entity word and paragraph characteristic vector and the pair- of target entity word
Wise similarity informations carry out error back propagation training convolutional neural networks model；
It is described by Softmax fit objects entity word and the paragraph according to one exemplary embodiment of the present invention
Characteristic vector and the pair-wise similarity informations of target entity word carry out training convolutional neural networks described in error back propagation
The step of model, includes：
In step S1051, exported according to the sentence characteristics and paragraph feature, using the Softmax to the training
The fitting result sets target function of target entity word in data set；
In step S1052, set according to the pair-wise similarity informations of the paragraph feature and the target entity word
Object function；
In step S1053, global object constraint function is set；
In step S1054, the parameter in model is updated using stochastic gradient descent method；
It is described according to utilizing sentence characteristics and the output of paragraph feature according to one exemplary embodiment of the present invention
Softmax concentrates the step of fitting result sets target function of target entity word to include to the training data：
Using formula (10), (11) and formula (12), (13), the sentence vectorization feature and paragraph vector are set
Change feature goal constraint function be respectively：
Wherein, LsFor the goal constraint function of the sentence vectorization feature, Lp1For the mesh of the paragraph vectorization feature
Mark constraint function,For paragraph set in all training corpusIn all sentence set,It is belonging to i-th of sentence
Correct entity word andIt is the correct entity word belonging to i-th of paragraph.
It is described to be set according to the pair-wise similarity informations of paragraph feature and the target entity word in step S1052
The step of object function, includes：
In order to strengthen the semantic meaning representation ability of the paragraph and entity, the present invention strengthens institute by sets target constraint function
The similitude of the corresponding affiliated entity word vectorization feature of paragraph vectorization feature is stated, and weakens the paragraph vector
Change the similitude of the corresponding non-belonging entity word vectorization feature of feature, its goal constraint function is as follows：
Wherein, erIt is to give the correct entity word belonging to the paragraph z.
It is as follows in step S1053, described the step of setting global object constraint function：
L=Ls+(1-α)·Lp1+α·Lp2 (18)
Wherein, α is weight harmonic coefficient, for balancing two of paragraph vectorization feature constraint Lp1And Lp2。
In step S1054, described the step of being updated using stochastic gradient descent method to the parameter in the model
Including：
All model training parameters are collectively expressed as θ in the goal constraint function of setting：
θ=(x, W(1), b(1), W(2), b(2), α, W(3), W(4), b(4), W(5), b(5), E) and (19)
In one exemplary embodiment of the present invention, error back propagation is carried out to described using stochastic gradient descent method
Object function optimizes.
In step S106, deep semantic feature is carried out to test description paragraph using convolutional neural networks model after renewal
Extract, the vectorization for being then based on paragraph represents to be linked with corresponding entity word.
According to one exemplary embodiment of the present invention, the convolutional neural networks model with after renewal is to described
Test description paragraph carries out deep semantic feature extraction, and the vectorization for being then based on the paragraph represents and the corresponding reality
The step of pronouns, general term for nouns, numerals and measure words is linked includes：
In step S1061, a test paragraph text is given, is first passed through in formula (2), (3), (4) calculating paragraph
The vectorization feature s of sentence；
In step S1062, the vectorization feature z of the paragraph is calculated by formula (6), (7), (8), (9)；
In step S1063, using the vectorization feature z of the paragraph of generation, using the linear transformation without Dropout and
The matching probability of the entity word corresponding to the output of Softmax functions：
Y=W(5)·z+b(5) (20)
Then matching probability highest entity word is the affiliated entity word of the test paragraph.
Fig. 2 is the framework of the entity and paragraph link method based on level convolutional network as one embodiment of the invention
Schematic diagram.
Reference picture 2, the entity based on level convolutional network share the characteristic vector of four levels with paragraph link method
Represent, be respectively：
Feature hierarchy one：The eigenmatrix that urtext paragraph represents to obtain by term vector；
Feature hierarchy two：The sentence vectorization obtained by convolutional neural networks represents feature；
Feature hierarchy three：The paragraph vectorization obtained by convolutional neural networks represents feature；
Feature hierarchy four：The vectorization that entity word is obtained using term vector look-up table represents feature；
The whole model training stage shares supervision message at three and instructed, and is respectively：
Supervision message one：The vectorization of sentence represents feature after linear change and Softmax outputs to affiliated entity
The fitting information of word；
Supervision message two：The vectorization of paragraph represents feature after linear change and Softmax outputs to affiliated entity
The fitting information of word；
Supervision message three：The vectorization of paragraph represents Pair-wise of the feature with affiliated entity word after linear change
Similarity information；
For the entity of accurate evaluation the inventive method and the link performance of paragraph, the present invention passes through comparison entity and paragraph
Link result and the uniformity of the true affiliated entity of paragraph obtain the precision (ACC) of the inventive method.It is given one descriptive section
Fall sample x(i), the entity word of the inventive method link is e(i), and truly the entity word is paragraphThe then definition of precision
It is as follows：
Wherein,The number of descriptive paragraph, δ (x, y) is indicator function, as x=y indicator function be 1, when x ≠
Indicator function is 0 during y.
Two kinds of disclosure data sets are used in the experiment of the present invention：
History：The data set includes 409 entities, 1704 paragraphs.
Literature：The data set includes 445 entities, 2247 paragraphs.
For these text data sets, the present invention is (including going the operation such as stop words and stem reduction) without any processing.
Average each paragraph includes 4-6 bar sentences, and each paragraph only includes 1 entity word.The specific statistical information of data set such as table 3
It is shown：
Table 3
Following control methods is used in the experiment of the present invention：
Control methods one：Based on bag of words and this special homing method of logic, this method is directly in the bag of words of urtext
Using this special homing method of logic on model；
Control methods two：Link method based on convolutional neural networks, this method use traditional convolutional neural networks mould
Type simply regards entity and paragraph link problems as a classification problem.
It is as shown in table 4 using parameter setting in present invention experiment：
Table 4
| Data set | ρ | hs | ht | d | k |
| History | 0.5 | 3 | 6 | 100 | 1 |
| Literature | 0.5 | 3 | 8 | 100 | 1 |
In table 4, using Dropout specific gravity factor, h when parameter ρ is model trainingsFor sentence vectorization character representation when
The frame mouth size of convolution kernel, htFor paragraph vectorization character representation when convolution kernel frame mouth size, d is term vector dimension, and k is sentence
The number of convolution kernel during subvector character representation.
In present invention experiment, all entities perform 50 times with paragraph link method and ask for its mean accuracy value (ACC), finally
Result of the test it is as shown in table 5：
Table 5
| Method | History/ accuracy values (%) | Literature/ accuracy values (%) |
| Control methods one | 65.10±0.01 | 61.17±0.05 |
| Control methods two | 77.01±3.92 | 74.50±10.3 |
| The inventive method | 89.41±1.05 | 91.26±0.50 |
Table 5 is that the inventive method, control methods one, the entity on two text data sets of control methods two link with paragraph
Accuracy value (ACC) evaluation result of method.Result of the test shows that the performance of the inventive method is significantly superior to other to analogy
Method.And 12.4% is improved respectively relative to best control methods two, accuracy value of the inventive method in two datasets
With 16.76%.
Meanwhile the slip word window size of verification experimental verification of the present invention convolution kernel when carrying out sentence characteristics expression is to the present invention
Method carries out the influence for the accuracy value performance that entity links with paragraph, and result of the test is as shown in Figure 3.It can be seen that when word window
When size is 3, the inventive method performance is all optimal in two datasets, and when word window size is more than 3, the present invention
The accuracy value hydraulic performance decline of method.Thus the slip word window size for the sentence characteristics convolution kernel that the present invention uses in testing is
3。
Particular embodiments described above, the purpose of the present invention, technical scheme and beneficial effect are carried out further in detail
Describe in detail bright, it should be understood that the foregoing is only the present invention specific embodiment, be not intended to limit the invention, it is all
Within the spirit and principles in the present invention, any modification, equivalent substitution and improvements done etc., the protection of the present invention should be included in
Within the scope of.
Claims (10)
1. a kind of entity based on level convolutional network and paragraph link method, comprise the following steps：
Represented by convolutional neural networks model and term vectorization, the vectorization for extracting every sentence in pending paragraph represents special
Sign；
Represented using convolutional neural networks structure and sentence vectorization, learn the deep semantic feature of the paragraph；
The vectorization expression of the sentence and the vectorization of paragraph are represented true by Softmax output fitting paragraphs institute respectively
Body；
Calculate the pair-wise analog informations that the vectorization expression of the entity represents with paragraph vectorization；
Pass through Softmax fit objects entity word and paragraph characteristic vector and the pair-wise similarity informations of target entity word
Carry out error back propagation and train the convolutional neural networks model；
Deep semantic feature extraction is carried out to the pending paragraph using the convolutional neural networks model after renewal, then
Vectorization based on the paragraph represents to be linked with corresponding entity word.
2. the entity according to claim 1 based on level convolutional network and paragraph link method, it is characterised in that described
Represented by convolutional neural networks model and term vectorization, the vectorization for extracting every sentence in pending paragraph represents feature
Step includes：
A sentence in pending paragraph is given, term vectorization expression is obtained using look-up table and characterizes the sentence into square
Formation formula；
Represent to carry out one-dimensional convolution in feature in the sentence matrixing, obtain the eigenmatrix after convolution；
Average sampling is carried out in convolution feature after the convolution to be compressed feature, obtains the vectorization table of the sentence
Show.
3. the entity according to claim 1 based on level convolutional network and paragraph link method, it is characterised in that described
Represented using convolutional neural networks structure and the sentence vectorization, wrapped the step of the deep semantic feature for learning the paragraph
Include：
Paragraph is characterized into matrix form by word order of the sentence in the paragraph using the sentence vector characteristics in the paragraph；
Represent to carry out one-dimensional convolution in feature in the paragraph matrixing, obtain the eigenmatrix after convolution；
Average sampling is carried out in convolution feature after the convolution to be compressed feature and carry out once linear conversion, is obtained
The vectorization of the paragraph represents.
4. the entity according to claim 1 based on level convolutional network and paragraph link method, it is characterised in that described
The vectorization of sentence represents and the vectorization of the paragraph represents to be fitted the affiliated entity of paragraph by Softmax outputs respectively
The step of include：
Linear transformation is carried out respectively to the sentence vector sum paragraph vector and obtains output vector, and is entered using Dropout technologies
Row canonical；
The link probability of candidate's entity is calculated using Softmax functions.
5. the entity according to claim 1 based on level convolutional network and paragraph link method, it is characterised in that described
The vectorization for calculating the entity represents that the method for the pair-wise analog informations represented with the paragraph vectorization is as follows：
Give an entity set of words E={ e1, e2..., em, the entity set of words is carried out using word2vec initial
Change, then entity set of words E and the paragraph characteristic vector z similitude are：
Sim (z, E)={ ze1, ze2..., zem}.
Wherein, operator ze represents the paragraph characteristic vector z and the corresponding entity word e similitude.
6. the entity according to claim 1 based on level convolutional network and paragraph link method, it is characterised in that described
Pass through the Softmax fit objects entity word and the paragraph characteristic vector and the pair-wise similarities of target entity word
The step of information progress error back propagation trains the convolutional neural networks model includes：
Exported according to the sentence characteristics and paragraph feature, target entity is concentrated to the training data using the Softmax
The fitting result sets target function of word；
According to the paragraph feature and the pair-wise similarity information sets target functions of the target entity word；
Set global object constraint function and the object function is subjected to unified fusion；
The parameter in the convolutional neural networks model is updated using stochastic gradient descent method.
7. the entity according to claim 6 based on level convolutional network and paragraph link method, it is characterised in that described
Included according to the step of paragraph feature and pair-wise similarity information sets target functions of the target entity word：
In order to strengthen the semantic meaning representation ability of the paragraph and entity, the paragraph vector is strengthened by sets target constraint function
Change the similitude of the corresponding affiliated entity word vectorization feature of feature, and it is corresponding to weaken the paragraph vectorization feature
Non-belonging entity word vectorization feature similitude, its described goal constraint function is as follows：
<mrow>
<msub>
<mi>L</mi>
<mrow>
<mi>p</mi>
<mn>2</mn>
</mrow>
</msub>
<mo>=</mo>
<munderover>
<mo>&Sigma;</mo>
<mrow>
<mi>i</mi>
<mo>=</mo>
<mn>1</mn>
</mrow>
<mrow>
<mo>|</mo>
<mi>c</mi>
<mo>|</mo>
</mrow>
</munderover>
<munder>
<mo>&Sigma;</mo>
<mrow>
<msub>
<mi>e</mi>
<mi>j</mi>
</msub>
<mo>&Element;</mo>
<mi>E</mi>
<mrow>
<mo>(</mo>
<msub>
<mi>e</mi>
<mi>j</mi>
</msub>
<mo>&NotEqual;</mo>
<msub>
<mi>e</mi>
<mi>r</mi>
</msub>
<mo>)</mo>
</mrow>
</mrow>
</munder>
<mi>m</mi>
<mi>a</mi>
<mi>x</mi>
<mrow>
<mo>(</mo>
<mn>0</mn>
<mo>,</mo>
<mn>1</mn>
<mo>-</mo>
<mi>s</mi>
<mi>i</mi>
<mi>m</mi>
<mo>(</mo>
<mrow>
<msup>
<mi>z</mi>
<mrow>
<mo>(</mo>
<mi>i</mi>
<mo>)</mo>
</mrow>
</msup>
<mo>,</mo>
<msubsup>
<mi>e</mi>
<mi>r</mi>
<mrow>
<mo>(</mo>
<mi>i</mi>
<mo>)</mo>
</mrow>
</msubsup>
</mrow>
<mo>)</mo>
<mo>+</mo>
<mi>s</mi>
<mi>i</mi>
<mi>m</mi>
<mo>(</mo>
<mrow>
<msup>
<mi>z</mi>
<mrow>
<mo>(</mo>
<mi>i</mi>
<mo>)</mo>
</mrow>
</msup>
<mo>,</mo>
<msubsup>
<mi>e</mi>
<mi>j</mi>
<mrow>
<mo>(</mo>
<mi>i</mi>
<mo>)</mo>
</mrow>
</msubsup>
</mrow>
<mo>)</mo>
<mo>)</mo>
</mrow>
<mo>;</mo>
</mrow>
Wherein, erIt is to give the correct entity word belonging to the paragraph z, er (i)For the true entity word of paragraph.
8. the entity according to claim 6 based on level convolutional network and paragraph link method, it is characterised in that described
The step of object function is carried out unified fusion by setting global object constraint function includes：
It is as follows to set the global object constraint function：
L=Ls+(1-α)·Lp1+α·Lp2；
Wherein, LsFor the goal constraint function of sentence vectorization feature, α is weight harmonic coefficient, for balancing the paragraph vector
Change two constraints of feature, i.e. the fitting of target entity word is concentrated in paragraph feature output using Softmax to the training data
Bound term Lp1With paragraph feature and the pair-wise similarity bound terms L of the target entity wordp2。
9. the entity according to claim 1 based on level convolutional network and paragraph link method, it is characterised in that described
Processing section is treated with the convolutional neural networks model after renewal and drops into row deep semantic feature extraction, is then based on the paragraph
Vectorization represents to include the step of link with the corresponding entity word：
A pending paragraph text is given, the convolutional neural networks model trained is first passed through and calculates sentence in the paragraph
Vectorization feature；
The vectorization feature of the paragraph is calculated by the convolutional neural networks model trained；
Using the vectorization feature of the paragraph of generation, the linear transformation without Dropout and the output pair of Softmax functions are used
The matching probability for the entity word answered.
10. the entity according to claim 1 based on level convolutional network and paragraph link method, it is characterised in that institute
State in convolutional neural networks model, the slip word window size of the sentence characteristics convolution kernel used is 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510372795.3A CN104915448B (en) | 2015-06-30 | 2015-06-30 | A kind of entity based on level convolutional network and paragraph link method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510372795.3A CN104915448B (en) | 2015-06-30 | 2015-06-30 | A kind of entity based on level convolutional network and paragraph link method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104915448A CN104915448A (en) | 2015-09-16 |
| CN104915448B true CN104915448B (en) | 2018-03-27 |
Family
ID=54084511
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510372795.3A Active CN104915448B (en) | 2015-06-30 | 2015-06-30 | A kind of entity based on level convolutional network and paragraph link method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104915448B (en) |
Families Citing this family (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107220220A (en) * | 2016-03-22 | 2017-09-29 | 索尼公司 | Electronic equipment and method for text-processing |
| CN106326985A (en) * | 2016-08-18 | 2017-01-11 | 北京旷视科技有限公司 | Neural network training method, neural network training device, data processing method and data processing device |
| CN106339718A (en) * | 2016-08-18 | 2017-01-18 | 苏州大学 | Classification method based on neural network and classification device thereof |
| CN106446526B (en) * | 2016-08-31 | 2019-11-15 | 北京千安哲信息技术有限公司 | Electronic health record entity relation extraction method and device |
| CN106844765B (en) * | 2017-02-22 | 2019-12-20 | 中国科学院自动化研究所 | Significant information detection method and device based on convolutional neural network |
| CN107144569A (en) * | 2017-04-27 | 2017-09-08 | 西安交通大学 | The fan blade surface defect diagnostic method split based on selective search |
| CN107168956B (en) * | 2017-05-26 | 2020-06-02 | 北京理工大学 | Chinese chapter structure analysis method and system based on pipeline |
| CN109426664A (en) * | 2017-08-30 | 2019-03-05 | 上海诺悦智能科技有限公司 | A kind of sentence similarity calculation method based on convolutional neural networks |
| CN107704563B (en) * | 2017-09-29 | 2021-05-18 | 广州多益网络股份有限公司 | Question recommendation method and system |
| CN108304552B (en) * | 2018-02-01 | 2021-01-08 | 浙江大学 | Named entity linking method based on knowledge base feature extraction |
| CN108764233B (en) * | 2018-05-08 | 2021-10-15 | 天津师范大学 | Scene character recognition method based on continuous convolution activation |
| CN109344244B (en) * | 2018-10-29 | 2019-11-08 | 山东大学 | A kind of the neural network relationship classification method and its realization system of fusion discrimination information |
| CN109697288B (en) * | 2018-12-25 | 2020-09-15 | 北京理工大学 | Instance alignment method based on deep learning |
| CN110147533B (en) | 2019-01-24 | 2023-08-29 | 腾讯科技（深圳）有限公司 | Encoding method, apparatus, device and storage medium |
| CN109992629B (en) * | 2019-02-28 | 2021-08-06 | 中国科学院计算技术研究所 | Neural network relation extraction method and system fusing entity type constraints |
| CN112328800A (en) * | 2019-08-05 | 2021-02-05 | 上海交通大学 | System and method for automatically generating programming specification question answers |
| CN110674317B (en) * | 2019-09-30 | 2022-04-12 | 北京邮电大学 | Entity linking method and device based on graph neural network |
| CN110717339B (en) * | 2019-12-12 | 2020-06-30 | 北京百度网讯科技有限公司 | Semantic representation model processing method and device, electronic equipment and storage medium |
| CN111222314B (en) * | 2020-01-03 | 2021-12-21 | 北大方正集团有限公司 | Layout document comparison method, device, equipment and storage medium |
| CN113361261B (en) * | 2021-05-19 | 2022-09-09 | 重庆邮电大学 | Method and device for selecting legal case candidate paragraphs based on enhance matrix |
| CN115130435B (en) * | 2022-06-27 | 2023-08-11 | 北京百度网讯科技有限公司 | Document processing method, device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104317834A (en) * | 2014-10-10 | 2015-01-28 | 浙江大学 | Cross-media sorting method based on deep neural network |
| CN104462357A (en) * | 2014-12-08 | 2015-03-25 | 百度在线网络技术（北京）有限公司 | Method and device for realizing personalized search |
| CN104615767A (en) * | 2015-02-15 | 2015-05-13 | 百度在线网络技术（北京）有限公司 | Searching-ranking model training method and device and search processing method |
| CN104679863A (en) * | 2015-02-28 | 2015-06-03 | 武汉烽火众智数字技术有限责任公司 | Method and system for searching images by images based on deep learning |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130212049A1 (en) * | 2012-02-15 | 2013-08-15 | American Gnc Corporation | Machine Evolutionary Behavior by Embedded Collaborative Learning Engine (eCLE) |
-
2015
- 2015-06-30 CN CN201510372795.3A patent/CN104915448B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104317834A (en) * | 2014-10-10 | 2015-01-28 | 浙江大学 | Cross-media sorting method based on deep neural network |
| CN104462357A (en) * | 2014-12-08 | 2015-03-25 | 百度在线网络技术（北京）有限公司 | Method and device for realizing personalized search |
| CN104615767A (en) * | 2015-02-15 | 2015-05-13 | 百度在线网络技术（北京）有限公司 | Searching-ranking model training method and device and search processing method |
| CN104679863A (en) * | 2015-02-28 | 2015-06-03 | 武汉烽火众智数字技术有限责任公司 | Method and system for searching images by images based on deep learning |
Non-Patent Citations (3)
| Title |
|---|
| A Convolutional Neural Network for Modelling Sentences;N. Kalchbrenner;《Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics》;20140630;第655-665页 * |
| A Neural Network for Factoid Question Answering over Paragraphs;M Iyyer etal;《Conference on Empirical Methods in Natural Language Processing》;20141231;第633-644页 * |
| Convolutional neural networks for sentence classification;Y. Kim etal;《Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing》;20141231;第1746-1751页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104915448A (en) | 2015-09-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104915448B (en) | A kind of entity based on level convolutional network and paragraph link method | |
| CN104834747B (en) | Short text classification method based on convolutional neural networks | |
| CN108595632B (en) | Hybrid neural network text classification method fusing abstract and main body characteristics | |
| CN107766324B (en) | Text consistency analysis method based on deep neural network | |
| CN109933670B (en) | Text classification method for calculating semantic distance based on combined matrix | |
| CN108399163A (en) | Bluebeard compound polymerize the text similarity measure with word combination semantic feature | |
| CN104951548B (en) | A kind of computational methods and system of negative public sentiment index | |
| CN109697285A (en) | Enhance the hierarchical B iLSTM Chinese electronic health record disease code mask method of semantic expressiveness | |
| WO2019080863A1 (en) | Text sentiment classification method, storage medium and computer | |
| CN109740154A (en) | A kind of online comment fine granularity sentiment analysis method based on multi-task learning | |
| CN109753660B (en) | LSTM-based winning bid web page named entity extraction method | |
| CN104216876B (en) | Information text filter method and system | |
| CN106484675A (en) | Fusion distributed semantic and the character relation abstracting method of sentence justice feature | |
| CN110245229A (en) | A kind of deep learning theme sensibility classification method based on data enhancing | |
| CN106980609A (en) | A kind of name entity recognition method of the condition random field of word-based vector representation | |
| CN106776562A (en) | A kind of keyword extracting method and extraction system | |
| Lei et al. | Patent analytics based on feature vector space model: A case of IoT | |
| Chang et al. | Research on detection methods based on Doc2vec abnormal comments | |
| CN104657350A (en) | Hash learning method for short text integrated with implicit semantic features | |
| CN109299462A (en) | Short text similarity calculating method based on multidimensional convolution feature | |
| CN106599054A (en) | Method and system for title classification and push | |
| CN102289522A (en) | Method of intelligently classifying texts | |
| CN107943824A (en) | A kind of big data news category method, system and device based on LDA | |
| CN103020167B (en) | A kind of computer Chinese file classification method | |
| CN107895000A (en) | A kind of cross-cutting semantic information retrieval method based on convolutional neural networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |