CN104915448A - Substance and paragraph linking method based on hierarchical convolutional network - Google Patents
Substance and paragraph linking method based on hierarchical convolutional network Download PDFInfo
- Publication number
- CN104915448A CN104915448A CN201510372795.3A CN201510372795A CN104915448A CN 104915448 A CN104915448 A CN 104915448A CN 201510372795 A CN201510372795 A CN 201510372795A CN 104915448 A CN104915448 A CN 104915448A
- Authority
- CN
- China
- Prior art keywords
- paragraph
- vectorization
- feature
- entity
- sentence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 71
- 239000000126 substance Substances 0.000 title abstract description 7
- 239000013598 vector Substances 0.000 claims abstract description 56
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 39
- 238000012549 training Methods 0.000 claims abstract description 16
- 230000006870 function Effects 0.000 claims description 34
- 239000011159 matrix material Substances 0.000 claims description 9
- 238000005070 sampling Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 6
- 238000005516 engineering process Methods 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 239000012141 concentrate Substances 0.000 claims description 4
- 238000011478 gradient descent method Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 2
- 230000004927 fusion Effects 0.000 claims 2
- 238000012360 testing method Methods 0.000 abstract description 16
- 239000013076 target substance Substances 0.000 abstract 1
- 239000002585 base Substances 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 239000012458 free base Substances 0.000 description 3
- 238000003062 neural network model Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000013139 quantization Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 description 1
- 235000007926 Craterellus fallax Nutrition 0.000 description 1
- 240000007175 Datura inoxia Species 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000013100 final test Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000013517 stratification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Fuzzy Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Databases & Information Systems (AREA)
- Machine Translation (AREA)
Abstract
The invention discloses a substance and paragraph linking method based on a hierarchical convolutional network. The substance and paragraph linking method based on the hierarchical convolutional network comprises the steps that: a word vectorization representation is transformed into a sentence vectorization representation by using a convolutional neural network; the sentence vectorization representation is convolved via the convolutional neural network to obtain a paragraph vectorization representation while considering the sentence sequence information; with the existed substance as supervision information, the sentence vectorization representation and the paragraph vectorization representation are subjected to a training of the hierarchical convolutional network through a Softmax output; meanwhile, the training of the hierarchical convolutional network is further improved according to pair-wise similarity between a paragraph semantic vector feature and a substance semantic vector feature; a test description paragraph is set and a deep semantic feature is extracted by using the trained hierarchical convolutional network to obtain a vectorization representation of a test paragraph;, so that the paragraph can be directly linked to the target substance based on the deep semantic feature through the Softmax output.
Description
Technical field
The present invention relates to construction of knowledge base technical field, relate more specifically to a kind of entity based on level convolutional network and paragraph link method.
Background technology
Nowadays, widely used large-scale knowledge base has Freebase, WordNet and YAGO etc.They are all devoted to the overall resources bank of structure one, and allow machine to access more easily and obtain structuring public information.Meanwhile, these knowledge bases provide application structure (APIs) to be convenient for people to inquire about the information of related entities more horn of plenty.Such as, when we retrieve a city name " Washington D.C. " in YAGO database, return results as shown in table 1 below:
Table 1
Can see, the object information returned is all the organizational information of some highly structurals.But these structured messages also do not meet actual context and the semantic information that people understand entity.Different with YAGO database, Freebase with WordNet then additionally can return the descriptive paragraph relevant to retrieving entity while return structure information, as shown in table 2 below:
Table 2
Can see, descriptive paragraph as shown in table 2 is more of value to concrete linguistic context and the semantic information that user understands query entity word.But the descriptive paragraph information of Freebase and WordNet is all by manually editing, and this carries out the limitation of paragraph description and the time of at substantial and manpower to entity under causing large data.Therefore, an efficient entity how is designed and descriptive paragraph AutoLink method is the task that large data age construction of knowledge base is urgently needed badly.
Can also see from the returning results of table 2, descriptive content might not comprise query entity word, and only need comprise some related terms and describe in many aspects entity.Therefore, in order to head it off, entity and paragraph link method need to set about from two aspects: the subject information 1, catching text from given one section of descriptive paragraph; 2, the important descriptive content relevant with entity is found.Much more traditional method is to extract the subject information of paragraph based on topic model method, as Dirichlet distribute (LDA) and probability are dived semantic analysis (PLSA) etc.To be the subject informations extracted be for the common problem of these methods based on the Term co-occurrence information acquisition of document level, and the height by the short-and-medium Text Representation of social media is openness to be affected relatively more serious, and lost the word order information in text.
In recent years, along with the rise of deep neural network, the deep layer latent semantic feature that some researchers attempt adopting depth model and term vector method for expressing to learn descriptive paragraph represents to solve the link problems of entity and paragraph.But, existing based on depth model method solve descriptive paragraph semantic feature extract time, just simply whole paragraph is regarded as a long sentence and carries out processing or directly multiple statement being weighted on average obtaining semantic vector.And in fact, the sentence order in paragraph also has Semantic logical relation.
On the other hand, it is also very important for catching in paragraph with the closely-related descriptive clue of entity.As although the descriptive paragraph during above-mentioned table 2 returns results directly does not comprise query entity word " Washington D.C. ", but but contain a lot of relevant vocabulary or phrase, as: " GeorgeWashington ", " United States " and " capital " etc.Therefore, that carries out that vectorization character representation contributes to entity and descriptive paragraph to entity links work.
Summary of the invention
For above-mentioned technical matters, fundamental purpose of the present invention is to provide a kind of entity based on level convolutional network and paragraph link method, thus by the entity word in internet and descriptive paragraph without the need to manually participation and AutoLink, the structure of the semantic knowledge-base under large data can be contributed to.
To achieve these goals, the invention provides a kind of entity based on level convolutional network and paragraph link method, comprise the following steps:
Utilize convolutional neural networks to represent that changing into sentence vectorization represents by term vector, described convolutional network is conducive to extracting query entity and is describing the important clue in paragraph;
Through convolutional neural networks, described sentence vectorization represents considers that described sentence order information obtains paragraph vectorization and represents again;
Described sentence vectorization is represented and to represent with described paragraph vectorization and to be exported by Softmax, carries out the training of described convolutional neural networks model by existing entity as supervision message;
Consider that the pair-wise similarity information between described paragraph semantic vector feature and Entity Semantics vector characteristics improves the training of described convolutional neural networks model further simultaneously;
A given test description paragraph, the neural network model trained described in utilization carries out the vectorization that Deep Semantics feature extraction obtains described test paragraph and represents, then exports based on this semantic expressiveness can directly be linked on target entity through Softmax.
Feature learning problem in the linking of entity and paragraph is divided into four levels by entity of the present invention and paragraph link method, is respectively: urtext paragraph represents the eigenmatrix layer obtained by term vector; The sentence vectorization representation feature layer obtained by convolutional neural networks; The paragraph vectorization representation feature layer obtained by convolutional neural networks; Term vector look-up table is utilized to obtain the vectorization representation feature layer of entity word.Tabled look-up by convolution character network and term vector, the accuracy value ACC of method of the present invention entity and paragraph link method on two text data sets is significantly superior to other control methodss, and relative to best control methods two, the accuracy value of the inventive method on two data sets improves 12.4% and 16.76% respectively.
Accompanying drawing explanation
Fig. 1 is the process flow diagram of the entity based on level convolutional network as one embodiment of the invention and paragraph link method;
Fig. 2 is the block schematic illustration of the entity based on level convolutional network as one embodiment of the invention and paragraph link method;
Fig. 3 is the performance schematic diagram of the entity based on level convolutional network as one embodiment of the invention and paragraph link method.
Embodiment
For making the object, technical solutions and advantages of the present invention clearly understand, below in conjunction with specific embodiment, and with reference to accompanying drawing, the present invention is described in further detail.
The invention discloses a kind of entity based on level convolutional network and paragraph link method, entity word in internet and descriptive paragraph can be carried out without the need to manually participating in ground AutoLink, its general plotting is, first carries out to the term vector in paragraph the vectorization that convolution obtains sentence represent by stratification convolutional neural networks.Consider the order information of sentence in paragraph, and the vectorization of sentence is represented that carrying out again the vectorization that convolution obtains paragraph represents.Then utilize substance feature to instruct the parameter learning of convolutional neural networks model as supervision message, consider simultaneously the deep semantic feature of paragraph and Entity Semantics vectorization represent between pair-wise affinity information improve the study of convolutional neural networks model.A given new descriptive paragraph, then can utilize its deep semantic feature of convolutional neural networks model extraction trained, and export the entity link obtaining correspondence based on this feature.
More specifically, first the method utilizes convolutional neural networks to represent that changing into sentence vectorization represents by term vector.Then utilize sentence vectorization to represent and again consider that described sentence order information obtains paragraph vectorization and represents through convolutional neural networks.Sentence vectorization is represented and to represent with paragraph vectorization and to be exported by Softmax, carries out the training of described convolutional neural networks model by existing entity as supervision message.Meanwhile, consider that the pair-wise similarity information between paragraph semantic vector feature and Entity Semantics vector characteristics improves the training of convolutional neural networks model further.A given test description paragraph, vectorization that Deep Semantics feature extraction obtains testing paragraph represents to utilize the neural network model that trains to carry out, and then exports based on this semantic expressiveness can directly be linked on target entity through Softmax.
Be described in detail as the entity based on level convolutional network of one embodiment of the invention and paragraph link method below in conjunction with accompanying drawing.
Fig. 1 is the process flow diagram of the entity based on level convolutional network as one embodiment of the invention and paragraph link method.
With reference to Fig. 1, in step S101, represented by convolutional neural networks model and term vector, extract the vectorization representation feature of every bar sentence in pending paragraph;
According to one exemplary embodiment of the present invention, describedly to be represented by convolutional neural networks model and term vector, the step extracting the vectorization representation feature of every bar sentence in pending paragraph comprises:
In step S1011, a sentence in given pending paragraph, utilizes look-up table obtain lexical item quantization means and sentence is characterized into matrix form;
In step S1012, described sentence matrixing representation feature carries out one dimension convolution, obtain the eigenmatrix after convolution;
In step S1013, the eigenmatrix after described convolution carries out average sampling and compress feature, the vectorization obtaining sentence represents.
According to one exemplary embodiment of the present invention, describedly look-up table is utilized to obtain lexical item quantization means and step sentence being characterized into matrix form comprises:
The term vector set that a given word2vec trains
wherein, | V| is dictionary size, and d is the dimension of term vector.Then arbitrarily length is that the sentence of n can be expressed as in paragraph:
s＝(x
1；x
2；...；x
n) (1)
Wherein, x
ithat the vectorization that i-th word utilizing look-up table to find in term vector set is corresponding represents.Wherein, if word x
ido not appear in the term vector set trained, then in this exemplary embodiment of the present invention, directly random initializtion expression is carried out to it.
In step S1012, describedly on sentence matrixing representation feature, carry out one dimension convolution, the step obtaining the eigenmatrix after convolution comprises:
Here, use
represent h initial from i-th word in sentence s
sindividual continuous word feature.A given one dimension convolution kernel
then h
seigenmatrix after individual continuous word feature convolution is:
Wherein, b
(1)be bias term, f is activation function,
h
sindividual continuous word feature
eigenmatrix after convolution.Then the eigenmatrix of described sentence is after convolution:
In step S1013, described eigenmatrix after convolution carries out average sampling and compress feature, the step that the vectorization obtaining sentence represents comprises:
In this exemplary embodiment of the present invention, the step of described employing average sampling is:
So far, each convolution kernel
a d dimensional feature vector can be generated
if employ k convolution kernel, then through a convolutional layer, the vectorization that finally can obtain sentence is expressed as
the dimension that then sentence vectorization represents is dk.
In step S102, utilize convolutional neural networks structure and described sentence vectorization to represent, learn the deep semantic feature of described paragraph;
According to one exemplary embodiment of the present invention, the deep semantic feature learning method of described paragraph comprises:
In step S1021, utilize the sentence vector characteristics in described paragraph, by the word order of sentence in described paragraph, paragraph is characterized into matrix form;
In step S1022, described paragraph matrixing representation feature carries out one dimension convolution, obtain the eigenmatrix after convolution;
In step S1023, the eigenmatrix after described convolution carries out average sampling and compress feature and carry out once linear conversion, the vectorization obtaining paragraph represents.
According to one exemplary embodiment of the present invention, the described step utilizing the sentence vector characteristics in paragraph paragraph to be characterized into matrix form by the word order of sentence in described paragraph comprises:
The vectorization having obtained the l bar sentence of described paragraph represents, then paragraph can be expressed as:
t＝(s
1；s
2；...；s
l) (5)
In step S1022, describedly on paragraph matrixing representation feature, carry out one dimension convolution, the step obtaining the eigenmatrix after convolution comprises:
Here, use
represent h initial from i-th sentence in paragraph t
tindividual sequence sentence subcharacter.A given one dimension convolution kernel
then h
tconvolution after individual sequence sentence subcharacter convolution is characterized as:
Wherein, b
(2)be bias term, f is activation function,
h
tindividual sequence sentence subcharacter
feature after convolution.Then the feature of described paragraph is after convolution:
In step S1023, described eigenmatrix after convolution carries out average sampling and compress feature and carry out once linear conversion, the step that the vectorization obtaining paragraph represents comprises:
In this exemplary embodiment of the present invention, the step of described employing average sampling is:
So far, through convolution kernel W
(2)generate a dk dimensional feature vector
conveniently calculate the similarity of paragraph feature and substance feature, the unification of vector dimension need be ensured, then once linear conversion is carried out to described paragraph vector:
Wherein,
for the matrix of a linear transformation, and proper vector z is final paragraph proper vector in one exemplary embodiment of the present invention.
In step S103, the vectorization of described sentence represents and to represent with the vectorization of described paragraph and export entity belonging to paragraph described in matching respectively through Softmax;
According to one exemplary embodiment of the present invention, the vectorization of described sentence and described paragraph represents that described in matching, the method for entity belonging to paragraph comprises the following steps respectively:
In step S1031, linear transformation is carried out respectively to described sentence vector sum paragraph vector and obtains output vector, and use Dropout technology to carry out canonical;
In step S1032, use the link probability of Softmax function calculated candidate entity;
According to one exemplary embodiment of the present invention, described distich subvector and paragraph vector carry out linear transformation and obtain output vector, and the step using Dropout technology to carry out canonical comprises:
Distich subvector feature s and paragraph vector characteristics t carries out linear change respectively, obtains two output vectors:
ys＝W
(4)·(sοr)+b
(4)(10)
y＝W
(5)·(zοr)+b
(5)(11)
Wherein,
with
be weight matrix, m is the entity number in one exemplary embodiment of the present invention, symbol.Representing matrix element take advantage of operation, and
it is then a Bernoulli Jacob's distribution obeying certain probability ρ.Dropout technology is used to prevent over-fitting, can the robustness of strength neural network model.
In step S1032, the step using described Softmax function to calculate the link probability of described candidate's entity comprises:
Softmax activation function is used to calculate the probable value of the described entity word of each correspondence respectively at two described output layers of described sentence vector characteristics and described paragraph vector characteristics:
Then in formula (12) and formula (13), ps
iand p
irepresent the probable value of corresponding i-th described entity word respectively.
In step S104, the vectorization calculating described entity represents and the pair-wise analog information that described paragraph vectorization represents;
A given entity word set E={e
1, e
2..., e
m, utilize word2vec to carry out initialization to described entity word set, then the similarity of entity word set E and described paragraph proper vector z is:
sim(z，E)＝{z·e
1，z·e
2，...，z·e
m} (14)
Wherein, operational character ze represents the similarity of described paragraph proper vector z and the described entity word e of correspondence.
In step S105, carry out error back propagation training convolutional neural networks model by the pair-wise similarity information of Softmax fit object entity word and paragraph proper vector and target entity word;
According to one exemplary embodiment of the present invention, the step that the described pair-wise similarity information by Softmax fit object entity word and described paragraph proper vector and target entity word carries out training convolutional neural networks model described in error back propagation comprises:
In step S1051, export according to described sentence characteristics and paragraph feature, utilize described Softmax to concentrate the fitting result target setting function of target entity word to described training data;
In step S1052, according to the pair-wise similarity information setting objective function of described paragraph feature and described target entity word;
In step S1053, setting global object constraint function;
In step S1054, stochastic gradient descent method is utilized to upgrade the parameter in model;
According to one exemplary embodiment of the present invention, described output according to sentence characteristics and paragraph feature utilizes described Softmax to concentrate the step of the fitting result target setting function of target entity word to comprise to described training data:
Utilize formula (10), (11) and formula (12), (13), the goal constraint function setting described sentence vectorization feature and described paragraph vectorization feature is respectively:
Wherein, L
sfor the goal constraint function of described sentence vectorization feature, L
p1for the goal constraint function of described paragraph vectorization feature,
for paragraph set in all corpus
in all sentence set,
be correct entity word belonging to i-th sentence and
it is the correct entity word belonging to i-th paragraph.
In step S1052, the step of the described pair-wise similarity information setting objective function according to paragraph feature and described target entity word comprises:
In order to strengthen the semantic meaning representation ability of described paragraph and entity, the present invention strengthens the similarity of the described paragraph vectorization feature described affiliated entity word vectorization feature corresponding with it by target setting constraint function, and weaken the similarity of the described paragraph vectorization feature described non-belonging entity word vectorization feature corresponding with it, its goal constraint function is as follows:
Wherein, e
rit is the correct entity word belonging to given described paragraph z.
In step S1053, the step of described setting global object constraint function is as follows:
L＝L
s+(1-α)·L
p1+α·L
p2(18)
Wherein, α is weight harmonic coefficient, and two that are used for balancing described paragraph vectorization feature retrain L
p1and L
p2.
In step S1054, the described stochastic gradient descent method that utilizes comprises the step that the parameter in described model upgrades:
In the described goal constraint function of setting, all model training improve parameter unifications are expressed as θ:
θ＝(x，W
(1)，b
(1)，W
(2)，b
(2)，α，W
(3)，W
(4)，b
(4)，W
(5)，b
(5)，E) (19)
In one exemplary embodiment of the present invention, adopt stochastic gradient descent method to carry out error back propagation and described objective function is optimized.
In step S106, utilize convolutional neural networks model after upgrading to carry out deep semantic feature extraction to test description paragraph, then represent based on the vectorization of paragraph and to link with corresponding entity word.
According to one exemplary embodiment of the present invention, described convolutional neural networks model after described renewal carries out deep semantic feature extraction to described test description paragraph, and the vectorization then based on described paragraph represents that the step of carrying out linking with corresponding described entity word comprises:
In step S1061, a given test paragraph text, first calculates the vectorization feature s of sentence in described paragraph by formula (2), (3), (4);
In step S1062, calculated the vectorization feature z of described paragraph by formula (6), (7), (8), (9);
In step S1063, utilize the vectorization feature z of the described paragraph generated, use the matching probability without the linear transformation of Dropout and the described entity word of Softmax function output correspondence:
y＝W
(5)·z+b
(5)(20)
The entity word that then matching probability is the highest is the affiliated entity word of described test paragraph.
Fig. 2 is the block schematic illustration of the entity based on level convolutional network as one embodiment of the invention and paragraph link method.
With reference to Fig. 2, the proper vector having four levels based on the entity of level convolutional network and paragraph link method represents, is respectively:
Feature hierarchy one: urtext paragraph represents the eigenmatrix obtained by term vector;
Feature hierarchy two: the sentence vectorization representation feature obtained by convolutional neural networks;
Feature hierarchy three: the paragraph vectorization representation feature obtained by convolutional neural networks;
Feature hierarchy four: utilize term vector look-up table to obtain the vectorization representation feature of entity word;
The whole model training stage has three place's supervision messages and instructs, and is respectively:
Supervision message one: the matching information of vectorization representation feature to affiliated entity word after linear change and Softmax export of sentence;
Supervision message two: the matching information of vectorization representation feature to affiliated entity word after linear change and Softmax export of paragraph;
Supervision message three: the vectorization representation feature of paragraph is after linear change and the Pair-wise similarity information of affiliated entity word;
In order to the entity of accurate evaluation the inventive method and the link performance of paragraph, the present invention by comparison entity and paragraph link result and paragraph true belonging to the consistance of entity obtain the precision (ACC) of the inventive method.A given descriptive paragraph sample x
(i), the entity word of the inventive method link is e
(i), and the true described entity word of paragraph is
then precision is defined as follows:
Wherein,
be the number of descriptive paragraph, δ (x, y) is indicator function, and as x=y, indicator function is 1, and as x ≠ y, indicator function is 0.
Two kinds of open text data sets are adopted in test of the present invention:
History: this data set comprises 409 entities, 1704 paragraphs.
Literature: this data set comprises 445 entities, 2247 paragraphs.
For these text data sets, the present invention is left intact (comprising the operation such as stop words and stem reduction).Average each paragraph comprises 4-6 bar sentence, and each paragraph only comprises 1 entity word.The concrete statistical information of data set is as shown in table 3:
Table 3
Following control methods is adopted in test of the present invention:
Control methods one: based on word bag model and this special homing method of logic, the method directly adopts this special homing method of logic on the word bag model of urtext;
Control methods two: based on the link method of convolutional neural networks, the method adopts traditional convolutional neural networks model to regard entity and paragraph link problems as a classification problem simply.
Adopt optimum configurations as shown in table 4 in the present invention's test:
Table 4
| Data set | ρ | h s | h t | d | k |
| History | 0.5 | 3 | 6 | 100 | 1 |
| Literature | 0.5 | 3 | 8 | 100 | 1 |
In table 4, adopt the specific gravity factor of Dropout when parameter ρ is model training, h
sfor the frame mouth size of convolution kernel during sentence vectorization character representation, h
tfor the frame mouth size of convolution kernel during paragraph vectorization character representation, d is term vector dimension, the number of convolution kernel when k is sentence vectorization character representation.
In the present invention's test, all entities and paragraph link method perform asks for its mean accuracy value (ACC) for 50 times, and final test findings is as shown in table 5:
Table 5
| Method | History/ accuracy value (%) | Literature/ accuracy value (%) |
| Control methods one | 65.10±0.01 | 61.17±0.05 |
| Control methods two | 77.01±3.92 | 74.50±10.3 |
| The inventive method | 89.41±1.05 | 91.26±0.50 |
Table 5 is accuracy value (ACC) evaluation result of the inventive method, control methods one, control methods two entity and paragraph link method on two text data sets.Test findings shows, the performance of the inventive method is significantly superior to other control methodss.And relative to best control methods two, the accuracy value of the inventive method on two data sets improves 12.4% and 16.76% respectively.
Meanwhile, the slip word window size of verification experimental verification of the present invention convolution kernel when carrying out sentence characteristics and representing carries out the impact of the accuracy value performance that entity links with paragraph on the inventive method, test findings as shown in Figure 3.Can see, when word window size is 3, the inventive method performance all reaches optimum on two data sets, and when word window size is greater than 3, the accuracy value hydraulic performance decline of the inventive method.The slip word window size of the sentence characteristics convolution kernel thus adopted in the present invention's experiment is 3.
Above-described specific embodiment; object of the present invention, technical scheme and beneficial effect are further described; be understood that; the foregoing is only specific embodiments of the invention; be not limited to the present invention; within the spirit and principles in the present invention all, any amendment made, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1., based on entity and the paragraph link method of level convolutional network, comprise the following steps:
Represented by convolutional neural networks model and term vector, extract the vectorization representation feature of every bar sentence in pending paragraph;
Utilize convolutional neural networks structure and sentence vectorization to represent, learn the deep semantic feature of described paragraph;
The vectorization of described sentence is represented and to represent with the vectorization of paragraph and export entity belonging to matching paragraph respectively through Softmax;
The vectorization calculating described entity represents the pair-wise analog information represented with paragraph vectorization;
Carry out error back propagation by the pair-wise similarity information of Softmax fit object entity word and paragraph proper vector and target entity word and train described convolutional neural networks model;
Utilize the described convolutional neural networks model after upgrading to carry out deep semantic feature extraction to described pending paragraph, then represent based on the vectorization of described paragraph and to link with corresponding entity word.
2. the entity based on level convolutional network according to claim 1 and paragraph link method, it is characterized in that, describedly represented by convolutional neural networks model and term vector, the step extracting the vectorization representation feature of every bar sentence in pending paragraph comprises:
A sentence in given pending paragraph, utilizes look-up table to obtain term vector and represents and described sentence is characterized into matrix form;
Described sentence matrixing representation feature carries out one dimension convolution, obtains the eigenmatrix after convolution;
Convolution feature after described convolution is carried out average sampling to compress feature, the vectorization obtaining described sentence represents.
3. the entity based on level convolutional network according to claim 1 and paragraph link method, is characterized in that, describedly utilizes convolutional neural networks structure and described sentence vectorization to represent, the step learning the deep semantic feature of described paragraph comprises:
Utilize the sentence vector characteristics in described paragraph, by the word order of sentence in described paragraph, paragraph is characterized into matrix form;
Described paragraph matrixing representation feature carries out one dimension convolution, obtains the eigenmatrix after convolution;
Convolution feature after described convolution is carried out average sampling compress feature and carry out once linear conversion, the vectorization obtaining described paragraph represents.
4. the entity based on level convolutional network according to claim 1 and paragraph link method, it is characterized in that, the vectorization of described sentence represents and to represent that with the vectorization of described paragraph the step exporting entity belonging to paragraph described in matching respectively through Softmax comprises:
Linear transformation is carried out respectively to described sentence vector sum paragraph vector and obtains output vector, and use Dropout technology to carry out canonical;
Use the link probability of Softmax function calculated candidate entity.
5. the entity based on level convolutional network according to claim 1 and paragraph link method, is characterized in that, the vectorization of the described entity of described calculating represents that the method for the pair-wise analog information represented with described paragraph vectorization is as follows:
A given entity word set E={e
1, e
2..., e
m, utilize word2vec to carry out initialization to described entity word set, then the similarity of entity word set E and described paragraph proper vector z is:
sim(z，E)＝{z·e
1，z·e
2，...，z·e
m}；
Wherein, operational character ze represents the similarity of described paragraph proper vector z and the described entity word e of correspondence.
6. the entity based on level convolutional network according to claim 1 and paragraph link method, it is characterized in that, the described pair-wise similarity information by described Softmax fit object entity word and described paragraph proper vector and target entity word is carried out error back propagation and is trained the step of described convolutional neural networks model to comprise:
Export according to described sentence characteristics and paragraph feature, utilize described Softmax to concentrate the fitting result target setting function of target entity word to described training data;
According to the pair-wise similarity information setting objective function of described paragraph feature and described target entity word;
Described objective function is carried out unified fusion by setting global object constraint function;
Stochastic gradient descent method is utilized to upgrade the parameter in described convolutional neural networks model.
7. the entity based on level convolutional network according to claim 6 and paragraph link method, is characterized in that, the step of the described pair-wise similarity information setting objective function according to described paragraph feature and described target entity word comprises:
In order to strengthen the semantic meaning representation ability of described paragraph and entity, the similarity of the described paragraph vectorization feature described affiliated entity word vectorization feature corresponding with it is strengthened by target setting constraint function, and weaken the similarity of the described paragraph vectorization feature described non-belonging entity word vectorization feature corresponding with it, described in it, goal constraint function is as follows:
Wherein, e
rit is the correct entity word belonging to given described paragraph z.
8. the entity based on level convolutional network according to claim 6 and paragraph link method, is characterized in that, the step that described objective function carries out unified fusion comprises by described setting global object constraint function:
Set described global object constraint function as follows:
L＝L
s+(1-α)·L
p1+α·L
p2；
Wherein, α is weight harmonic coefficient, is used for balancing two constraints of described paragraph vectorization feature, and namely paragraph feature exports and utilizes Softmax to concentrate the matching bound term L of target entity word to described training data
p1with the pair-wise similarity bound term L of paragraph feature and described target entity word
p2.
9. the entity based on level convolutional network according to claim 1 and paragraph link method, it is characterized in that, convolutional neural networks model after described renewal is treated processing section and is dropped into the feature extraction of row deep semantic, and the vectorization then based on described paragraph represents that the step of carrying out linking with corresponding described entity word comprises:
A given pending paragraph text, the convolutional neural networks model first by training calculates the vectorization feature of sentence in described paragraph;
The vectorization feature of described paragraph is calculated by the convolutional neural networks model trained;
Utilize the vectorization feature of the described paragraph generated, use the matching probability without the linear transformation of Dropout and the entity word of Softmax function output correspondence.
10. the entity based on level convolutional network according to claim 1 and paragraph link method, is characterized in that, in described convolutional neural networks model, the slip word window size of the sentence characteristics convolution kernel of employing is 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510372795.3A CN104915448B (en) | 2015-06-30 | 2015-06-30 | A kind of entity based on level convolutional network and paragraph link method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510372795.3A CN104915448B (en) | 2015-06-30 | 2015-06-30 | A kind of entity based on level convolutional network and paragraph link method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104915448A true CN104915448A (en) | 2015-09-16 |
| CN104915448B CN104915448B (en) | 2018-03-27 |
Family
ID=54084511
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510372795.3A Active CN104915448B (en) | 2015-06-30 | 2015-06-30 | A kind of entity based on level convolutional network and paragraph link method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104915448B (en) |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106326985A (en) * | 2016-08-18 | 2017-01-11 | 北京旷视科技有限公司 | Neural network training method, neural network training device, data processing method and data processing device |
| CN106339718A (en) * | 2016-08-18 | 2017-01-18 | 苏州大学 | Classification method based on neural network and classification device thereof |
| CN106446526A (en) * | 2016-08-31 | 2017-02-22 | 北京千安哲信息技术有限公司 | Electronic medical record entity relation extraction method and apparatus |
| CN106844765A (en) * | 2017-02-22 | 2017-06-13 | 中国科学院自动化研究所 | Notable information detecting method and device based on convolutional neural networks |
| CN107144569A (en) * | 2017-04-27 | 2017-09-08 | 西安交通大学 | The fan blade surface defect diagnostic method split based on selective search |
| CN107168956A (en) * | 2017-05-26 | 2017-09-15 | 北京理工大学 | A kind of Chinese structure of an article analysis method and system based on pipeline |
| WO2017162134A1 (en) * | 2016-03-22 | 2017-09-28 | 索尼公司 | Electronic device and method for text processing |
| CN107704563A (en) * | 2017-09-29 | 2018-02-16 | 广州多益网络股份有限公司 | A kind of question sentence recommends method and system |
| CN108304552A (en) * | 2018-02-01 | 2018-07-20 | 浙江大学 | A kind of name entity link method that knowledge based planting modes on sink characteristic extracts |
| CN108764233A (en) * | 2018-05-08 | 2018-11-06 | 天津师范大学 | A kind of scene character recognition method based on continuous convolution activation |
| CN109344244A (en) * | 2018-10-29 | 2019-02-15 | 山东大学 | A kind of the neural network relationship classification method and its realization system of fusion discrimination information |
| CN109426664A (en) * | 2017-08-30 | 2019-03-05 | 上海诺悦智能科技有限公司 | A kind of sentence similarity calculation method based on convolutional neural networks |
| CN109697288A (en) * | 2018-12-25 | 2019-04-30 | 北京理工大学 | A kind of example alignment schemes based on deep learning |
| CN109992629A (en) * | 2019-02-28 | 2019-07-09 | 中国科学院计算技术研究所 | A kind of neural network Relation extraction method and system of fusion entity type constraint |
| CN110674317A (en) * | 2019-09-30 | 2020-01-10 | 北京邮电大学 | Entity linking method and device based on graph neural network |
| CN110717339A (en) * | 2019-12-12 | 2020-01-21 | 北京百度网讯科技有限公司 | Semantic representation model processing method and device, electronic equipment and storage medium |
| CN111222314A (en) * | 2020-01-03 | 2020-06-02 | 北大方正集团有限公司 | Layout document comparison method, device, equipment and storage medium |
| WO2020151688A1 (en) * | 2019-01-24 | 2020-07-30 | 腾讯科技（深圳）有限公司 | Coding method and device, equipment and storage medium |
| CN112328800A (en) * | 2019-08-05 | 2021-02-05 | 上海交通大学 | System and method for automatically generating programming specification question answers |
| CN113361261A (en) * | 2021-05-19 | 2021-09-07 | 重庆邮电大学 | Method and device for selecting legal case candidate paragraphs based on enhance matrix |
| CN115130435A (en) * | 2022-06-27 | 2022-09-30 | 北京百度网讯科技有限公司 | Document processing method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130212049A1 (en) * | 2012-02-15 | 2013-08-15 | American Gnc Corporation | Machine Evolutionary Behavior by Embedded Collaborative Learning Engine (eCLE) |
| CN104317834A (en) * | 2014-10-10 | 2015-01-28 | 浙江大学 | Cross-media sorting method based on deep neural network |
| CN104462357A (en) * | 2014-12-08 | 2015-03-25 | 百度在线网络技术（北京）有限公司 | Method and device for realizing personalized search |
| CN104615767A (en) * | 2015-02-15 | 2015-05-13 | 百度在线网络技术（北京）有限公司 | Searching-ranking model training method and device and search processing method |
| CN104679863A (en) * | 2015-02-28 | 2015-06-03 | 武汉烽火众智数字技术有限责任公司 | Method and system for searching images by images based on deep learning |
-
2015
- 2015-06-30 CN CN201510372795.3A patent/CN104915448B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130212049A1 (en) * | 2012-02-15 | 2013-08-15 | American Gnc Corporation | Machine Evolutionary Behavior by Embedded Collaborative Learning Engine (eCLE) |
| CN104317834A (en) * | 2014-10-10 | 2015-01-28 | 浙江大学 | Cross-media sorting method based on deep neural network |
| CN104462357A (en) * | 2014-12-08 | 2015-03-25 | 百度在线网络技术（北京）有限公司 | Method and device for realizing personalized search |
| CN104615767A (en) * | 2015-02-15 | 2015-05-13 | 百度在线网络技术（北京）有限公司 | Searching-ranking model training method and device and search processing method |
| CN104679863A (en) * | 2015-02-28 | 2015-06-03 | 武汉烽火众智数字技术有限责任公司 | Method and system for searching images by images based on deep learning |
Non-Patent Citations (3)
| Title |
|---|
| M IYYER ETAL: "A Neural Network for Factoid Question Answering over Paragraphs", 《CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING》 * |
| N. KALCHBRENNER: "A Convolutional Neural Network for Modelling Sentences", 《PROCEEDINGS OF THE 52ND ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS》 * |
| Y. KIM ETAL: "Convolutional neural networks for sentence classification", 《PROCEEDINGS OF THE 2014 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING》 * |
Cited By (32)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10860798B2 (en) | 2016-03-22 | 2020-12-08 | Sony Corporation | Electronic device and method for text processing |
| WO2017162134A1 (en) * | 2016-03-22 | 2017-09-28 | 索尼公司 | Electronic device and method for text processing |
| CN106339718A (en) * | 2016-08-18 | 2017-01-18 | 苏州大学 | Classification method based on neural network and classification device thereof |
| CN106326985A (en) * | 2016-08-18 | 2017-01-11 | 北京旷视科技有限公司 | Neural network training method, neural network training device, data processing method and data processing device |
| CN106446526B (en) * | 2016-08-31 | 2019-11-15 | 北京千安哲信息技术有限公司 | Electronic health record entity relation extraction method and device |
| CN106446526A (en) * | 2016-08-31 | 2017-02-22 | 北京千安哲信息技术有限公司 | Electronic medical record entity relation extraction method and apparatus |
| CN106844765A (en) * | 2017-02-22 | 2017-06-13 | 中国科学院自动化研究所 | Notable information detecting method and device based on convolutional neural networks |
| CN106844765B (en) * | 2017-02-22 | 2019-12-20 | 中国科学院自动化研究所 | Significant information detection method and device based on convolutional neural network |
| CN107144569A (en) * | 2017-04-27 | 2017-09-08 | 西安交通大学 | The fan blade surface defect diagnostic method split based on selective search |
| CN107168956A (en) * | 2017-05-26 | 2017-09-15 | 北京理工大学 | A kind of Chinese structure of an article analysis method and system based on pipeline |
| CN107168956B (en) * | 2017-05-26 | 2020-06-02 | 北京理工大学 | Chinese chapter structure analysis method and system based on pipeline |
| CN109426664A (en) * | 2017-08-30 | 2019-03-05 | 上海诺悦智能科技有限公司 | A kind of sentence similarity calculation method based on convolutional neural networks |
| CN107704563A (en) * | 2017-09-29 | 2018-02-16 | 广州多益网络股份有限公司 | A kind of question sentence recommends method and system |
| CN107704563B (en) * | 2017-09-29 | 2021-05-18 | 广州多益网络股份有限公司 | Question recommendation method and system |
| CN108304552A (en) * | 2018-02-01 | 2018-07-20 | 浙江大学 | A kind of name entity link method that knowledge based planting modes on sink characteristic extracts |
| CN108764233A (en) * | 2018-05-08 | 2018-11-06 | 天津师范大学 | A kind of scene character recognition method based on continuous convolution activation |
| CN108764233B (en) * | 2018-05-08 | 2021-10-15 | 天津师范大学 | Scene character recognition method based on continuous convolution activation |
| CN109344244A (en) * | 2018-10-29 | 2019-02-15 | 山东大学 | A kind of the neural network relationship classification method and its realization system of fusion discrimination information |
| CN109697288A (en) * | 2018-12-25 | 2019-04-30 | 北京理工大学 | A kind of example alignment schemes based on deep learning |
| US11934788B2 (en) | 2019-01-24 | 2024-03-19 | Tencent Technology (Shenzhen) Company Limited | Encoding method, apparatus, and storage medium |
| WO2020151688A1 (en) * | 2019-01-24 | 2020-07-30 | 腾讯科技（深圳）有限公司 | Coding method and device, equipment and storage medium |
| CN109992629B (en) * | 2019-02-28 | 2021-08-06 | 中国科学院计算技术研究所 | Neural network relation extraction method and system fusing entity type constraints |
| CN109992629A (en) * | 2019-02-28 | 2019-07-09 | 中国科学院计算技术研究所 | A kind of neural network Relation extraction method and system of fusion entity type constraint |
| CN112328800A (en) * | 2019-08-05 | 2021-02-05 | 上海交通大学 | System and method for automatically generating programming specification question answers |
| CN110674317A (en) * | 2019-09-30 | 2020-01-10 | 北京邮电大学 | Entity linking method and device based on graph neural network |
| CN110674317B (en) * | 2019-09-30 | 2022-04-12 | 北京邮电大学 | Entity linking method and device based on graph neural network |
| CN110717339A (en) * | 2019-12-12 | 2020-01-21 | 北京百度网讯科技有限公司 | Semantic representation model processing method and device, electronic equipment and storage medium |
| US11520991B2 (en) | 2019-12-12 | 2022-12-06 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Method, apparatus, electronic device and storage medium for processing a semantic representation model |
| CN111222314A (en) * | 2020-01-03 | 2020-06-02 | 北大方正集团有限公司 | Layout document comparison method, device, equipment and storage medium |
| CN113361261A (en) * | 2021-05-19 | 2021-09-07 | 重庆邮电大学 | Method and device for selecting legal case candidate paragraphs based on enhance matrix |
| CN115130435A (en) * | 2022-06-27 | 2022-09-30 | 北京百度网讯科技有限公司 | Document processing method and device, electronic equipment and storage medium |
| CN115130435B (en) * | 2022-06-27 | 2023-08-11 | 北京百度网讯科技有限公司 | Document processing method, device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104915448B (en) | 2018-03-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104915448A (en) | Substance and paragraph linking method based on hierarchical convolutional network | |
| CN104951548B (en) | A kind of computational methods and system of negative public sentiment index | |
| CN110321925B (en) | Text multi-granularity similarity comparison method based on semantic aggregated fingerprints | |
| CN103235772B (en) | A kind of text set character relation extraction method | |
| CN103154936B (en) | For the method and system of robotization text correction | |
| CN109508459B (en) | Method for extracting theme and key information from news | |
| CN111191002B (en) | Neural code searching method and device based on hierarchical embedding | |
| CN106156272A (en) | A kind of information retrieval method based on multi-source semantic analysis | |
| Maharjan et al. | A multi-task approach to predict likability of books | |
| CN106202010A (en) | The method and apparatus building Law Text syntax tree based on deep neural network | |
| CN106980609A (en) | A kind of name entity recognition method of the condition random field of word-based vector representation | |
| Chang et al. | Research on detection methods based on Doc2vec abnormal comments | |
| CN106055673A (en) | Chinese short-text sentiment classification method based on text characteristic insertion | |
| CN103336852B (en) | Across language ontology construction method and device | |
| CN106326212A (en) | Method for analyzing implicit type discourse relation based on hierarchical depth semantics | |
| CN110598219A (en) | Emotion analysis method for broad-bean-net movie comment | |
| CN111597328B (en) | New event theme extraction method | |
| CN109325114A (en) | A kind of text classification algorithm merging statistical nature and Attention mechanism | |
| CN103631858A (en) | Science and technology project similarity calculation method | |
| CN107688870A (en) | A kind of the classification factor visual analysis method and device of the deep neural network based on text flow input | |
| CN110851593A (en) | Complex value word vector construction method based on position and semantics | |
| CN114997288A (en) | Design resource association method | |
| CN106055633A (en) | Chinese microblog subjective and objective sentence classification method | |
| CN108009187A (en) | A kind of short text Topics Crawling method for strengthening Text Representation | |
| CN116629258B (en) | Structured analysis method and system for judicial document based on complex information item data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |