CN113591483B - A document-level event argument extraction method based on sequence labeling - Google Patents
A document-level event argument extraction method based on sequence labeling Download PDFInfo
- Publication number
- CN113591483B CN113591483B CN202110460585.5A CN202110460585A CN113591483B CN 113591483 B CN113591483 B CN 113591483B CN 202110460585 A CN202110460585 A CN 202110460585A CN 113591483 B CN113591483 B CN 113591483B
- Authority
- CN
- China
- Prior art keywords
- word
- span
- context
- representation
- entity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000605 extraction Methods 0.000 title claims abstract description 57
- 238000002372 labelling Methods 0.000 title claims description 22
- 238000000034 method Methods 0.000 claims abstract description 35
- 230000007246 mechanism Effects 0.000 claims abstract description 26
- 230000004927 fusion Effects 0.000 claims abstract description 19
- 238000012549 training Methods 0.000 claims abstract description 18
- 239000013598 vector Substances 0.000 claims abstract description 16
- 238000013528 artificial neural network Methods 0.000 claims abstract description 5
- 230000000306 recurrent effect Effects 0.000 claims abstract 2
- 239000011159 matrix material Substances 0.000 claims description 18
- 238000012360 testing method Methods 0.000 claims description 17
- 230000006870 function Effects 0.000 claims description 10
- 230000009193 crawling Effects 0.000 claims description 7
- 102100022693 Mucin-4 Human genes 0.000 claims description 6
- 108010008699 Mucin-4 Proteins 0.000 claims description 6
- 239000000284 extract Substances 0.000 claims description 6
- 238000012546 transfer Methods 0.000 claims description 6
- 230000002776 aggregation Effects 0.000 claims description 3
- 238000004220 aggregation Methods 0.000 claims description 3
- 238000011161 development Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 claims description 2
- 238000013507 mapping Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 10
- 230000002457 bidirectional effect Effects 0.000 abstract 1
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 4
- 230000001419 dependent effect Effects 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 102100021908 3-mercaptopyruvate sulfurtransferase Human genes 0.000 description 1
- 102100031048 Coiled-coil domain-containing protein 6 Human genes 0.000 description 1
- 101000753843 Homo sapiens 3-mercaptopyruvate sulfurtransferase Proteins 0.000 description 1
- 101000777370 Homo sapiens Coiled-coil domain-containing protein 6 Proteins 0.000 description 1
- 238000007476 Maximum Likelihood Methods 0.000 description 1
- 101000640206 Tityus serrulatus Alpha-mammal toxin Ts2 Proteins 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/211—Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- General Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Machine Translation (AREA)
Abstract
本发明请求保护一种基于序列标注的文档级事件论元抽取方法，包括以下几个步骤：获取语料库实体相关的Wikipedia先验知识，生成词跨度实体语义增强嵌入表示；将词跨度实体语义增强嵌入表示与预训练语言模型得到的上下文表示进行拼接，得到嵌入层的词向量输入；将词表示输入到多跨度双向循环神经网络中，获取词的多跨度上下文特征表示；将多跨度上下文特征表示输入到上下文注意力机制和门控注意力机制模块中，获取词的上下文语义融合特征表示；最后将输出特征表示采用序列标注进行事件论元抽取，利用训练得到的最优模型对未知文档进行事件论元抽取；本发明通过融入先验知识和多跨度上下语义特征表示，有效的提高了文档级事件论元抽取的效果。
The present invention claims a method for extracting document-level event arguments based on sequence annotation, comprising the following steps: obtaining Wikipedia prior knowledge related to corpus entities, generating word span entity semantics enhanced embedding representation; concatenating the word span entity semantics enhanced embedding representation with the context representation obtained by a pre-trained language model to obtain a word vector input for the embedding layer; inputting the word representation into a multi-span bidirectional recurrent neural network to obtain a multi-span context feature representation of the word; inputting the multi-span context feature representation into a context attention mechanism and a gated attention mechanism module to obtain a context semantics fusion feature representation of the word; finally, using sequence annotation to extract event arguments from the output feature representation, and using the optimal model obtained by training to extract event arguments from unknown documents; the present invention effectively improves the effect of document-level event argument extraction by integrating prior knowledge and multi-span contextual semantic feature representations.
Description
Technical Field
The invention belongs to the field of information extraction of natural language processing, and provides a document-level event argument extraction model based on sequence annotation. The model can provide basic services for tasks such as knowledge graph construction, relation extraction, information retrieval, automatic question and answer and the like.
Background
The constituent elements of an event include trigger words, event types, argument and argument roles. The goal of event extraction is to automatically acquire the information from unstructured information and display the information in a structured way. However, most of the conventional event extraction tasks are based on sentence level, and have obvious defects that one event involves a trigger word and a plurality of arguments, and in general, there are few ideal cases that the event trigger word and its related arguments all appear in the same sentence. Therefore, the event extraction at the document level is researched to acquire the complete event information of a long sequence, which is more beneficial to the practical application of the event extraction.
The document level event argument extraction task is to identify an event argument and a corresponding event role type in a document according to a predefined event type and a corresponding role type thereof. In a document, event meta-information is often presented in the form of entities. Therefore, accurate understanding of entity semantics is beneficial to better completing event argument extraction tasks for the model. In the past, most of event extraction adopts entity type characteristics and subtype characteristics of the entity, and ignores semantic deviation between the entity serving as a semantic whole and words forming the entity.
The complete meta information of the document level event is scattered in a plurality of sentences of the document, and it is difficult to extract the event information by a sentence level event extraction method. The document level event argument extraction task requires a good understanding of long sequence semantics to make decisions. More recently, pipeline methods have been employed to train classifiers for event argument role and related context detection, respectively, in document level event argument extraction task related work. However, the pipeline method is easy to generate cascading errors, and the downstream task cannot feed information back to the upstream task to assist the decision of the upstream task. The neural network sequence labeling method has been proven to be effective in extracting event information. The sequence labeling method aims at multi-classification tasks at word level, event arguments can be directly extracted and event argument roles can be matched at the same time, and therefore error transmission and one-way flow of task information of the pipeline method are avoided. In addition, the existing research shows that the semantic meaning of a long sequence text is captured and the semantic meaning of sentences and the semantic meaning among sentences are required to be understood, but the previous research does not consider the influence of semantic information among words on a document-level event argument extraction task.
Through retrieval, patent application number 202011506990.8 is a document level event argument extraction method based on sequence labeling, the method disclosed by the invention is to construct a document level event argument extraction model based on sequence labeling by taking unstructured document text as input data, and comprises two stages, wherein the first stage is to optimize word span entity semantic embedding as word representation of an embedded layer through priori knowledge, and the second stage is to acquire sentence spans and paragraph span context semantic features in the document through a feature extractor and a double-layer attention mechanism and use the context semantic features to label event arguments in the document. Context semantic information of multiple spans is fused simultaneously in two stages to generate more accurate event argument extraction results. For a commonly used document-level event argument extraction data set, the method has the effect remarkably superior to the existing document-level event argument extraction method, and proves that multi-span context semantic information integrated into a document is beneficial to extraction of event arguments from the document.
Disclosure of Invention
The present invention is directed to solving the above problems of the prior art. A document level event argument extraction method based on sequence annotation is provided. The technical scheme of the invention is as follows:
a document level event argument extraction method based on sequence annotation, comprising the following steps:
step 1, carrying out entity identification processing on document corpus in a corpus, and acquiring Wikipedia priori knowledge aiming at the entity identified by the document;
extracting semantic attribute words of each entity through the priori knowledge obtained in the step 1, and obtaining word span semantic enhancement embedded representation by utilizing the entity semantic attribute words;
Step 3, splicing the word span entity semantic enhancement embedded representation obtained in the step 2 with the context semantic representation obtained by the pre-training language model bert _base to obtain an embedded word representation of each word;
step 4, inputting the embedded layer word representation obtained in the step 3 into a cyclic neural network, and obtaining the sentence span and the paragraph span context word representation of each word through two 3-layer BiLSTM feature extractors of the sentence span and the paragraph span;
Step 5, inputting the context word representations of the two spans obtained in the step 4 into a context attention mechanism module of sentence spans and paragraph spans respectively to obtain unequal characteristic representations of each word in different context spans;
step 6, inputting the sentence span and paragraph span characteristic representation obtained in the step 5 into a gating attention mechanism module for characteristic fusion to obtain a final document multispan context semantic fusion characteristic representation;
and 7, taking the multi-span context semantic fusion feature representation obtained in the step 6 as input of a CRF sequence labeling layer, labeling event arguments and role types thereof by adopting a BIO labeling format, training to obtain an optimal model, and finally extracting the event arguments from the test set document of the corpus by utilizing the trained extraction model.
Further, the step 1 specifically includes the steps of first performing entity identification on the document by using spaCy tools, and then crawling the prior knowledge related to the entity by the identified entity.
Further, the crawling of the prior knowledge related to the entity by the identified entity specifically includes crawling an explanatory article corresponding to the entity in the document on the Wikipedia website by using a crawler technology, and mapping a first segment of the explanatory article with the entity as a prior knowledge corpus.
Further, in the step 2, semantic attribute words of each entity are extracted through priori knowledge, and word span semantic enhancement embedded representation is obtained by using the entity semantic attribute words, which specifically includes:
Extracting and screening entity semantic word sets of each entity through the prior knowledge corpus obtained in the step 1, assuming that each entity has N semantic type words, the semantic type word set of the entity e is expressed as D e,si∈De as the ith semantic attribute word of the entity e, GloVe, which is entity semantic attribute words s i, is embedded, and then entity semantic embedded e d of entity e is generated:
The parameter a is used to control the weight of the semantic embedding of an entity, for each word w j that constitutes an entity, Is GloVe embedding of the word w j that constitutes entity e, whose word span semantic enhancement embedding representation e s is as follows:
Further, the step 3 is to splice the word span entity semantic enhancement embedded representation obtained in the step 2 with the context semantic representation obtained in the pre-training language model bert _base to obtain an embedded word representation of each word, and specifically includes:
For each word x i, the word span entity semantic enhancement embedding is obtained Context representation of sentence spans and paragraph spans generated using BERT-base, with for each word x i, its context representationThe final embedded word representation is formed by splicing entity semantic enhancement word embedding and context word representation,
Further, in step 4, the sentence span and paragraph span context of each word represent specific methods as follows:
firstly, two Bi-LSTM encoders with 3 layers, namely BiLSTM sent. and BiLSTM para. are established, then all documents in the MUC-4 dataset are divided into single sentences s 1,s2,...,sn, in order to construct a paragraph span training dataset, the average paragraph number of all documents in the dataset is calculated and recorded as m, from each sentence i, m continuous sentences s i to s i +m-1 are connected to form an overlapped candidate sequence with the length of m, the sequence 1 consists of { s 1,s2,s3 }, the sequence 2 consists of { s 2,s3,s4 }, and the like;
sentence span context semantic representation extraction to characterize the sentence span context of each word in a paragraph
The expression is as follows:
paragraph span context semantic representation extraction to extract paragraph context features for each word in a paragraph
The expression is as follows:
further, in the step 5, the inequality feature of each word in different context spans represents a specific method that:
word context hidden layer representation of sentence span obtained by step 4
Recording single word representation vectors of sentence spans asThe word representation vector for each word introduces a bilinear sentence context attention mechanism to obtain context-dependent semantic information that is more important to the sentence, specifically as follows:
wherein u si represents a single word representation in a sentence Context word representation in sentenceW s represents a learnable weight matrix, b s represents a bias, a si represents the importance of each word in the sentence context, and R sent. represents the sum of weights of each word in the sentence based on semantic contribution to the sentence span context;
similar to the sentence context attention mechanism, the word context hidden layer representation for the paragraph span resulting from step 4 Marking single word representation vectors within paragraph spans asWord representation vectors for each wordIntroducing a bilinear paragraph context attention mechanism to obtain context-dependent semantic information of greater importance to the paragraph is as follows:
Wherein u pi represents a single word in a paragraph Contextual word representation in paragraphsW p represents a learnable weight matrix, b p represents a bias, a pi represents the importance of each word in a paragraph in the paragraph context, and R para. represents the sum of weights of each word in a paragraph based on semantic contribution to the paragraph span context.
Further, the specific steps of feature fusion in the step 6 are as follows:
For each word x i, adopting gating attention information fusion, wherein gating aggregation calculates a gating vector g i through sentence span context word representation and paragraph span context word representation, so as to control information contribution degree from the two representations;
gi＝σ(Wc1Rsent.+Wc2Rpara.+bc)
Wherein W c1,Wc2 represents a learnable weight matrix, b c represents a bias, and σ represents a sigmoid function;
adopting g i and 1-g i as the assigned weights of the sentence span representation R sent. and the paragraph span representation R para., and finally, the gating information fusion representation R i is the sum of the weights of R sent. and R para.:
Wherein the method comprises the steps of Representing the corresponding multiplication of the parity elements.
Further, in the step 7, the specific method for sequence labeling is as follows:
In the document level event argument extraction task, a CRF conditional random field is adopted to label the event argument in the document in a BIO labeling format, the CRF considers the dependence between adjacent labels, the CRF takes the output of a linear layer as a CRF emission probability matrix P, P i,j represents the probability that the word at the ith position is a label j, A is a CRF transition matrix, For a tag sequence y= { y 1,y2,...,yk }, the score for the current sequence may be defined as:
From the above formula, it can be seen that the score of each word mapped to the tag is determined via two parts, including upper layer output and CRF self-contained transfer matrix, and normalized probability is calculated by softmax, where the formula is
Optimizing an objective function using a maximized log-likelihood function, training the log-likelihood of the sample (x|y) to be
In the decoding stage, a Viterbi algorithm of dynamic programming is adopted to calculate the rightmost path to obtain the probability of the corresponding label in the sequence, the label corresponding to the maximum probability is considered to be the correct label, and the probability formula is that
Further, in the step 7, the specific method for extracting the event argument of the test set document is as follows:
Firstly, converting a test set document into a symbol sequence through a word segmentation device, then obtaining a corresponding tag sequence pair in a BIO format by using a trained event argument extraction model, and restoring a corresponding word in the test set document through the obtained tag sequence pair to serve as an event argument extraction result of the test set document.
The invention has the advantages and beneficial effects as follows:
The invention provides a document-level event argument extraction model based on multi-span context semantic fusion, which is used for fusing entity enhancement semantics and context Wen Yuyi of word spans in the longitudinal direction and extracting and dynamically fusing sentence spans and paragraph span context semantics in the transverse direction. Specifically, the method and the system firstly explore the influence of word span entity semantic enhancement on a long sequence event argument extraction model so as to optimize the context semantics of word spans, dynamically fuse the context semantics of sentence spans and paragraph spans by using a context attention mechanism and a gating attention mechanism, and simultaneously solve the problems of long sequence semantic capture and document-level event argument dispersion by adopting a sequence labeling strategy. By improving and fusing the multi-span context semantics, the effect of extracting event arguments from the document is improved.
The innovation of the invention mainly comprises the steps of providing entity semantic enhancement of word spans so as to optimize the context Wen Yuyi of the word spans, and providing important information in the context of sentence spans and paragraph spans to be extracted and fused by utilizing a context attention mechanism and a gating attention mechanism so as to optimize the context semantics of the sentence spans and the paragraph spans.
Drawings
FIG. 1 is a flow chart of a method for extracting document level event arguments based on sequence tags.
FIG. 2 is a graph of the macro average result of document level event argument extraction based on MUC-4 data set.
FIG. 3 is a graph of extraction results of various event argument roles based on head noun phrase matching evaluation criteria.
Fig. 4 is a trend chart of influence of word span entity semantic enhancement blending weight alpha on extraction effect of event meta-model.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and specifically described below with reference to the drawings in the embodiments of the present invention. The described embodiments are only a few embodiments of the present invention.
The technical scheme for solving the technical problems is as follows:
the document level event argument extraction task is to identify an event argument and a corresponding event role in a document according to a predefined event type and a corresponding role type thereof.
The document-level event argument extraction method based on sequence annotation provided by the invention adopts a commonly used document-level event argument extraction data set MUC-4 in the implementation stage. The invention adopts the same data partitioning standard, namely 1300 documents are used as training sets, 200 documents (TST1+TST2) are used for development sets and 200 documents (TST3+TST4) are used as test sets.
The invention adopts two evaluation standards of head noun phrase matching and accurate matching. The judging standard of the head noun phrase matching is that if the first noun is matched, the marking is successfully matched, and if the first noun is matched, the accurate matching is that the predicted word is completely matched with the template marking word, so that the capability of evaluating the boundary of the event element matched by the model is facilitated. The experimental results of the invention give the precision (P), recall (R) and F-1 scores, and also give the macro average of all event argument experimental results.
FIG. 1 is a technical roadmap of the invention, as shown in FIG. 1, the method comprises:
step 1, acquiring Wikipedia priori knowledge of a document corpus entity;
The goal of this stage is to obtain a priori knowledge of the document corpus for subsequent optimal representation of the contextual semantics of the word span. First, spaCy is adopted to perform entity identification on the whole MUC-4 dataset, and an entity set is obtained through manual screening. And then crawling corresponding Wikipedia entity explanatory articles through the obtained entity set.
Step 2, generating word span entity semantic enhancement embedded representation;
The Wikipedia related explanatory article of the entity is obtained in step 1, and the word span entity semantic enhancement embedded representation is generated by utilizing the Wikipedia priori knowledge.
By observation, it is found that there are semantic type words semantically associated with an entity in the first segment in the entity-related interpretation article of Wikipedia. First, crawling Wikipedia explanation articles related to all entities aiming at the extracted entities, and calculating the occurrence frequency of nouns in the first section of the Wikipedia explanation articles. The invention manually screens few seed words, uses spacy to calculate word similarity to expand the seed words, and finally obtains a dictionary containing 1762 entity semantic attribute words.
Generating word span entity semantic enhancement embedding aggregate representation first generates entity semantic embedments using the resulting semantic attribute words of each entity, and then combines them with existing word embedments to generate entity semantic enhancement embedments. Each semantic attribute word of an entity can be considered as a semantic component of the entity. An entity semantic embedding may be generated based on a set of semantic type words for each entity. The semantic type word for each entity is represented using a 100-dimensional GloVe pre-training word insert trained from the web crawler number of 6B. Assuming that each entity has N semantic type words, the semantic type word set for entity e is denoted as D e. Entity semantic embedding e d of entity e is then generated as follows:
Where s i∈De is the ith semantic attribute term for entity e, Is GloVe embedding of entity semantic attribute words s i.
An entity semantic insert is created for each entity by linearly aggregating entity semantic attribute word inserts. For the entity semantic enhancement embedding of word spans, each word constituting an entity is reasonably considered to contain the semantics of the entity as a whole, so the word span entity semantic enhancement embedding is expressed as a weighted sum of the entity semantic embedding and the GloVe word embedding of each word constituting the entity. The parameter α is used to control the weight of entity semantic embedding. For each word w j that constitutes an entity, its word span semantic enhancement embedded representation e s is as follows:
Wherein the method comprises the steps of Is GloVe embedment of the word w j that constitutes entity e.
Step3, generating an embedded word representation;
The context embedding generated by the pre-trained language model has proven to model contexts outside of sentence boundaries and improve performance of various tasks.
In the embedding layer of the present invention, for each word x i in a sentence, a word span entity semantic enhancement embeds and pre-trains a concatenation matrix of contextual word representations of the language model. The pre-trained language model may slice the words during the preprocessing stage, which may increase OOV word generation. It is assumed that the slices of words that make up an entity also have the complete semantics of the whole word. Based on this, an embedded dictionary is generated that contains 9326 word span entity semantic enhancement embedments, where word span entity semantic enhancement embedded representations contain word slices.
From this, for each word x i, the word span entity semantic enhancement embedding is obtained from step 2Here, the present invention uses BERT-base generated sentence spans and paragraph span context representations, with for each word x i, its context representationThe final embedded word representation is formed by splicing entity semantic enhancement word embedding and context word representation,
Step 4, generating context word representations of the document sentence spans and the paragraph spans;
Although the word embedding is the same for words from different spans, the sentence spans of words and the context encoding of paragraph spans are different. To help the model better capture contextual features between words of different span sequences in documents, the present invention builds two 3-layer Bi-LSTM encoders, biLSTM sent. and BiLSTM para. the present invention divides all documents in the MUC-4 dataset into a single sentence s 1,s2,...,sn to construct a paragraph span training dataset, the average paragraph number of all documents in the dataset is calculated and denoted as m. Starting with each sentence i, m consecutive sentences s i to s i +m-1 are concatenated as one paragraph, forming overlapping candidate sequences of length m, sequence 1 consisting of { s 1,s2,s3 }, sequence 2 consisting of { s 2,s3,s4 }, and so on. To balance the training set, the present invention samples the same number of positive and negative sequences from the candidate sequences, where positive sequence means that at least one event argument is contained and negative sequence means that no event argument is contained. To construct the paragraph span development dataset and the test dataset, only consecutive m sentences need to be grouped in order, generating n/p sequences.
Sentence span context semantic feature representation extraction, the present invention extracts sentence span context features for each word in a paragraphThe expression is as follows:
paragraph span context semantic feature representation extraction, the invention extracts paragraph context features for each word in a paragraph The expression is as follows:
step 5, generating unequal characteristic representations of sentence spans and paragraph spans;
obtaining context semantic feature hidden layer representation of sentence span through step 4 Marking single word representation vectors within sentence spans asBecause the contextual word representation of each word is not equally important to the sentence, a bilinear sentence contextual attention mechanism is introduced for the word representation vector of each word to obtain contextually relevant semantic information that is more important to the sentence in order to make an accurate decision on the type of word. The method comprises the following steps:
wherein u si represents a single word representation in a sentence Context word representation in sentenceW s represents a learnable weight matrix and b s represents a bias. a si represents the importance of each word in the sentence context, and R sent. represents the sum of weights of each word in the sentence based on the semantic contribution to the sentence span context.
Similar to the sentence context attention mechanism, hidden layer representation is semantically characterized for paragraph spans derived from step 4Marking single word representation vectors within paragraph spans asWord representation vectors for each wordA bilinear paragraph context attention mechanism is introduced to obtain context-dependent semantic information that is more important to the paragraph in order to make an accurate decision on the type of word. The method comprises the following steps:
Wherein u pi represents a single word in a paragraph Contextual word representation in paragraphsW p represents a learnable weight matrix and b p represents a bias. a pi represents the importance of each word in the paragraph context, and R para. represents the sum of weights of each word in the paragraph based on the semantic contribution to the paragraph span context.
Step 6, generating multi-span context semantic fusion feature representation;
For each word x i, in order to aggregate the sentence span obtained in step 5 and the context semantic inequality representation learned by paragraph span, adopting gating attention information fusion, wherein gating aggregation calculates a gating vector g i through the sentence span context word representation and the paragraph span context word representation, so as to control the information contribution degree of the two representations.
gi＝σ(Wc1Rsent.+Wc2Rpara.+bc)
Where W c1,Wc2 represents a learnable weight matrix and b c represents a bias. Sigma denotes a sigmoid function.
The invention adopts g i and 1-g i as the assigned weights of the sentence span R sent. and the paragraph span R para.. The final gating information fusion represents that R i is the sum of the weights of R sent. and R para.,
Wherein the method comprises the steps ofRepresenting the corresponding multiplication of the parity elements.
Step 7, outputting a prediction label based on the fusion characteristic representation, and training to obtain an optimal model so as to extract event arguments of the test set document;
In the invention, a CRF conditional random field is adopted to carry out predictive label on event arguments in a document in a BIO labeling format. The CRF considers the dependency between adjacent tags, and uses the output of the CRF passing through the linear layer as the transmission probability matrix P of the CRF, where P i,j represents the probability that the i-th word is the tag j. A is the transfer matrix of the CRF, For the tag sequence y= { y 1,y2,...,yk }, the score for the current sequence may be defined as:
from the above formula, it can be seen that the score of each word mapped to the tag is determined via two parts, including the multi-span context semantic fusion feature representation obtained in step 6 and the transfer matrix of the CRF itself. Calculating normalized probability by softmax, the formula is
Optimizing an objective function using a maximized log-likelihood function, training the log-likelihood of the sample (x|y) to be
In the decoding stage, a Viterbi algorithm of dynamic programming is adopted to calculate the rightmost path to obtain the probability of the corresponding label in the sequence, the label corresponding to the maximum probability is considered to be the correct label, and the probability formula is that
And training and learning parameters of the overall extraction model by using known labeling data of the training set, and extracting relevant event arguments from the unknown documents to be extracted serving as test set data.
And converting the document into a symbol sequence by using the same processing mode as the training set data for the document of the test set through a word segmentation device, then inputting the symbol sequence into an extraction model obtained by previous training, and outputting a BIO label sequence pair with the maximum likelihood probability. And after the BIO tag sequence is obtained, restoring words or phrases in the text according to the BIO tag, and obtaining an event argument extraction result in the test set document.
The macro average experimental results evaluated using the head noun matching criteria and the exact matching criteria are shown in fig. 2. While for head noun matching evaluation criteria, more refined experimental results for each event argument role are shown in fig. 3.
In order to explore the importance of word span entity semantic enhancement and context attention semantic enhancement in a document-level event argument extraction task, an entity semantic enhancement model and a context attention mechanism semantic enhancement model are respectively set. In order to explore the influence of the entity semantic embedding duty ratio on the performance of the document-level event argument extraction model, the invention provides a variable experiment SADEAE-Word-SPAN ENTITY SEMANTIC reinforced Extract aiming at the entity semantic duty ratio weight a, and experiments are carried out by respectively taking alpha as 0.1,0.3,0.5,0.7,0.9. The variable experiment result shows that the word-level entity semantic enhancement embedding effectively improves the effect of the neural network event argument extraction model. Through the experimental data of fig. 2 and 3, a macro average F-1 value trend graph based on head noun matching and a respective F-1 value trend graph for each event role type are plotted. As shown in fig. 4, as α increases, the overall effect of the model appears as an experimental effect of rising before falling, and the F-1 macro average achieves the best effect when α=0.7.
To explore the impact of context awareness mechanism semantic enhancements on sentence spans and Paragraph spans on document-level event argument extraction models, the present invention sets models SADEAE-Sentence-Span Contextual Semantic that incorporate sentence span-only context awareness mechanisms and models SADEAE-Paragraph-Span Contextual Semantic that incorporate Paragraph-level context awareness mechanisms, and the results of all experiments are presented in FIGS. 3 and 4. The experimental result shows that the contextual semantic extraction effect of the contextual attention mechanism on the sentence span and paragraph exaggeration of the document is obvious.
The experimental results of the document-level event extraction model SADEAE based on sequence labeling, which is presented in the present invention, are shown in the last line of fig. 2 and 3, which combine entity semantic enhancement and contextual attention semantic enhancement, and simultaneously acquire multiple levels of semantics. The discovery obtained from the experimental result can achieve better effect than the prior multi-granularity reader with optimal performance. By fusing word-level entity semantic enhancement and upper and lower Wen Yuyi under a double-layer attention mechanism, the F-1 value of a document-level event argument extraction model (SADEAE) based on sequence labeling is increased by more than 3 minutes on the basis of a predecessor optimal model, and the score of >63 is achieved, wherein the accuracy is improved by 3.85, and the recall rate is improved by 4.71. This effectively demonstrates that the model of the present invention has a greatly improved ability to match document-level event arguments and their boundaries.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.
The above examples should be understood as illustrative only and not limiting the scope of the invention. Various changes and modifications to the present invention may be made by one skilled in the art after reading the teachings herein, and such equivalent changes and modifications are intended to fall within the scope of the invention as defined in the appended claims.
Claims (7)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110460585.5A CN113591483B (en) | 2021-04-27 | 2021-04-27 | A document-level event argument extraction method based on sequence labeling |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110460585.5A CN113591483B (en) | 2021-04-27 | 2021-04-27 | A document-level event argument extraction method based on sequence labeling |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113591483A CN113591483A (en) | 2021-11-02 |
| CN113591483B true CN113591483B (en) | 2024-12-13 |
Family
ID=78243063
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110460585.5A Active CN113591483B (en) | 2021-04-27 | 2021-04-27 | A document-level event argument extraction method based on sequence labeling |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113591483B (en) |
Families Citing this family (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114185868B (en) * | 2021-10-30 | 2023-05-30 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | Intelligent construction method for Chinese hot event library |
| CN113779227B (en) * | 2021-11-12 | 2022-01-25 | 成都数之联科技有限公司 | Case fact extraction method, system, device and medium |
| CN114168727B (en) * | 2021-12-06 | 2024-07-12 | 哈尔滨工业大学 | Method, storage medium and equipment for extracting document-level event main body pairs facing financial field |
| CN114741516B (en) * | 2021-12-08 | 2024-12-03 | 商汤国际私人有限公司 | Event extraction method and device, electronic device and storage medium |
| CN114492377B (en) * | 2021-12-30 | 2024-04-16 | 永中软件股份有限公司 | Event role labeling method, computer equipment and computer readable storage medium |
| CN114298053B (en) * | 2022-03-10 | 2022-05-24 | 中国科学院自动化研究所 | A joint event extraction system based on the fusion of feature and attention mechanism |
| CN114625842B (en) * | 2022-03-25 | 2024-11-29 | 电子科技大学长三角研究院(衢州) | False comment recognition device based on structural attention enhancement mechanism |
| CN114648016B (en) * | 2022-03-29 | 2025-02-11 | 河海大学 | An event argument extraction method based on event element interaction and label semantic enhancement |
| CN114742016B (en) * | 2022-04-01 | 2024-07-09 | 山西大学 | Chapter-level event extraction method and device based on multi-granularity entity different composition |
| CN114936563B (en) * | 2022-04-27 | 2023-07-25 | 苏州大学 | Event extraction method, device and storage medium |
| CN114880431B (en) * | 2022-05-10 | 2024-11-19 | 中国人民解放军国防科技大学 | A prompt-based event argument extraction method and system |
| CN115017879A (en) * | 2022-05-27 | 2022-09-06 | 深圳证券信息有限公司 | Text comparison method, computer equipment and computer storage medium |
| CN114818721B (en) * | 2022-06-30 | 2022-11-01 | 湖南工商大学 | Event joint extraction model and method combined with sequence labeling |
| CN114881038B (en) * | 2022-07-12 | 2022-11-11 | 之江实验室 | Chinese entity and relation extraction method and device based on span and attention mechanism |
| CN115238685B (en) * | 2022-09-23 | 2023-03-21 | 华南理工大学 | Combined extraction method for building engineering change events based on position perception |
| CN116151241B (en) * | 2023-04-19 | 2023-07-07 | 湖南马栏山视频先进技术研究院有限公司 | Entity identification method and device |
| CN116739000A (en) * | 2023-06-08 | 2023-09-12 | 北京智源人工智能研究院 | Method and device for training speaker extraction model for complex context and electronic equipment |
| CN117909505B (en) * | 2024-03-13 | 2024-06-07 | 北京邮电大学 | Event argument extraction method and related equipment |
| CN118133823B (en) * | 2024-03-14 | 2024-07-30 | 北京语言大学 | Candidate argument filtering method and device for extracting document-level event |
| CN118780269B (en) * | 2024-09-12 | 2025-02-18 | 山东浪潮科学研究院有限公司 | High-efficiency cross-document information extraction system and method based on large language model |
| CN119003794B (en) * | 2024-10-22 | 2025-02-21 | 北京市科学技术研究院 | Urban site knowledge graph construction method integrating event extraction technology, cultural relic data management system and readable storage medium |
| CN119067121B (en) * | 2024-11-05 | 2025-01-28 | 浙江大学 | A general information extraction method |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107797993A (en) * | 2017-11-13 | 2018-03-13 | 成都蓝景信息技术有限公司 | A kind of event extraction method based on sequence labelling |
| CN108829801B (en) * | 2018-06-06 | 2020-11-20 | 大连理工大学 | An event-triggered word extraction method based on document-level attention mechanism |
| CN109582949B (en) * | 2018-09-14 | 2022-11-22 | 创新先进技术有限公司 | Event element extraction method and device, computing equipment and storage medium |
| CN110032641B (en) * | 2019-02-14 | 2024-02-13 | 创新先进技术有限公司 | Method and device for extracting event by using neural network and executed by computer |
| CN110135457B (en) * | 2019-04-11 | 2021-04-06 | 中国科学院计算技术研究所 | Method and system for event-triggered word extraction based on autoencoder fusion of document information |
| CN112183095A (en) * | 2019-07-02 | 2021-01-05 | 普天信息技术有限公司 | Event extraction method and device |
| CN110705289B (en) * | 2019-09-29 | 2023-03-28 | 重庆邮电大学 | Chinese word segmentation method, system and medium based on neural network and fuzzy inference |
| CN111191031A (en) * | 2019-12-24 | 2020-05-22 | 上海大学 | An Entity Relationship Classification Method for Unstructured Text Based on WordNet and IDF |
| CN111581345A (en) * | 2020-04-26 | 2020-08-25 | 上海明略人工智能(集团)有限公司 | Document level event extraction method and device |
| CN111783394B (en) * | 2020-08-11 | 2024-03-08 | 深圳市北科瑞声科技股份有限公司 | Training method of event extraction model, event extraction method, system and equipment |
| CN112231447B (en) * | 2020-11-21 | 2023-04-07 | 杭州投知信息技术有限公司 | Method and system for extracting Chinese document events |
| CN112528676B (en) * | 2020-12-18 | 2022-07-08 | 南开大学 | Document-level event argument extraction method |
-
2021
- 2021-04-27 CN CN202110460585.5A patent/CN113591483B/en active Active
Non-Patent Citations (1)
| Title |
|---|
| Document-level Event Argument Extraction via Multispan Semantic Reinforcement;Dong Qiu 等;《ISKE2016》;20220418;223-230 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113591483A (en) | 2021-11-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113591483B (en) | A document-level event argument extraction method based on sequence labeling | |
| Wang et al. | Application of convolutional neural network in natural language processing | |
| CN109062893B (en) | Commodity name identification method based on full-text attention mechanism | |
| CN111401077B (en) | Language model processing method and device and computer equipment | |
| CN113743119B (en) | Chinese named entity recognition module, method and device and electronic equipment | |
| CN114757182B (en) | A BERT short text sentiment analysis method with improved training method | |
| CN113255320A (en) | Entity relation extraction method and device based on syntax tree and graph attention machine mechanism | |
| CN111126069A (en) | Social media short text named entity identification method based on visual object guidance | |
| JP5710581B2 (en) | Question answering apparatus, method, and program | |
| CN111738003A (en) | Named entity recognition model training method, named entity recognition method and medium | |
| CN114298055B (en) | Retrieval method and device based on multilevel semantic matching, computer equipment and storage medium | |
| CN113569050A (en) | Method and device for automatic construction of knowledge graph in government affairs field based on deep learning | |
| CN111159405B (en) | Irony detection method based on background knowledge | |
| CN114153971B (en) | Error correction recognition and classification equipment for Chinese text containing errors | |
| CN116151256A (en) | A Few-Shot Named Entity Recognition Method Based on Multi-task and Hint Learning | |
| CN114417851B (en) | Emotion analysis method based on keyword weighted information | |
| CN114818717A (en) | Chinese named entity recognition method and system fusing vocabulary and syntax information | |
| CN115238697A (en) | Judicial named entity recognition method based on natural language processing | |
| GB2572320A (en) | Hate speech detection system for online media content | |
| Ajees et al. | A named entity recognition system for Malayalam using neural networks | |
| CN112685594B (en) | Attention-based weak supervision voice retrieval method and system | |
| CN114330350A (en) | Named entity identification method and device, electronic equipment and storage medium | |
| CN115526176A (en) | Text recognition method and device, electronic equipment and storage medium | |
| CN118585641A (en) | A text summary generation method based on pre-training model | |
| Padia et al. | UMBC at SemEval-2018 Task 8: Understanding text about malware |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240529 Address after: 518000 1104, Building A, Zhiyun Industrial Park, No. 13, Huaxing Road, Henglang Community, Longhua District, Shenzhen, Guangdong Province Applicant after: Shenzhen Hongyue Information Technology Co.,Ltd. Country or region after: China Address before: 400065 Chongwen Road, Nanshan Street, Nanan District, Chongqing Applicant before: CHONGQING University OF POSTS AND TELECOMMUNICATIONS Country or region before: China |
|
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20241117 Address after: 318000 No. 809, Central Avenue, Jiaojiang District, Taizhou City, Zhejiang Province Applicant after: STATE GRID ZHEJIANG ELECTRIC POWER CO., LTD. TAIZHOU POWER SUPPLY Co. Country or region after: China Address before: 518000 1104, Building A, Zhiyun Industrial Park, No. 13, Huaxing Road, Henglang Community, Longhua District, Shenzhen, Guangdong Province Applicant before: Shenzhen Hongyue Information Technology Co.,Ltd. Country or region before: China |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |