CN118036577A - Sequence labeling method in natural language processing - Google Patents
Sequence labeling method in natural language processing Download PDFInfo
- Publication number
- CN118036577A CN118036577A CN202410431577.1A CN202410431577A CN118036577A CN 118036577 A CN118036577 A CN 118036577A CN 202410431577 A CN202410431577 A CN 202410431577A CN 118036577 A CN118036577 A CN 118036577A
- Authority
- CN
- China
- Prior art keywords
- labeling
- sequence
- text
- word
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000002372 labelling Methods 0.000 title claims abstract description 139
- 238000003058 natural language processing Methods 0.000 title claims abstract description 19
- 238000000034 method Methods 0.000 claims abstract description 50
- 238000012545 processing Methods 0.000 claims abstract description 32
- 230000008569 process Effects 0.000 claims abstract description 31
- 230000007246 mechanism Effects 0.000 claims abstract description 27
- 238000004458 analytical method Methods 0.000 claims abstract description 14
- 238000013136 deep learning model Methods 0.000 claims abstract description 7
- 238000007781 pre-processing Methods 0.000 claims abstract description 6
- 230000011218 segmentation Effects 0.000 claims abstract description 6
- 238000010606 normalization Methods 0.000 claims abstract description 5
- 230000006870 function Effects 0.000 claims description 33
- 239000013598 vector Substances 0.000 claims description 28
- 238000012549 training Methods 0.000 claims description 23
- 238000012546 transfer Methods 0.000 claims description 11
- 238000012795 verification Methods 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 238000002790 cross-validation Methods 0.000 claims description 6
- 230000004927 fusion Effects 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 6
- 230000001419 dependent effect Effects 0.000 claims description 5
- 239000011159 matrix material Substances 0.000 claims description 5
- 238000013528 artificial neural network Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 claims description 4
- 238000007499 fusion processing Methods 0.000 claims description 3
- 230000003993 interaction Effects 0.000 claims description 3
- 238000003062 neural network model Methods 0.000 claims description 3
- 230000000306 recurrent effect Effects 0.000 claims description 3
- 230000004931 aggregating effect Effects 0.000 claims description 2
- 238000012512 characterization method Methods 0.000 claims description 2
- 230000002194 synthesizing effect Effects 0.000 claims description 2
- 238000013519 translation Methods 0.000 abstract description 12
- 239000010410 layer Substances 0.000 description 50
- 230000000875 corresponding effect Effects 0.000 description 8
- 230000008901 benefit Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000004660 morphological change Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 239000002344 surface layer Substances 0.000 description 1
Landscapes
- Machine Translation (AREA)
Abstract
The invention relates to the technical field of machine translation, in particular to a sequence labeling method in natural language processing, which comprises the following steps: receiving text data input, and preprocessing, including word segmentation, stop word removal and normalization processing, to create a foundation for subsequent sequence annotation; performing part-of-speech tagging on the preprocessed text, and automatically identifying the part of speech of each word by using an deep learning model; identifying and classifying the entities in the text by applying a sequence labeling model, and labeling the attributes of the entities at the same time; combining the self-attention mechanism and the position code to process the word sequence in the sequence label; performing deep syntactic analysis, and marking the syntactic structure of each sentence in the text, wherein the syntactic structure comprises a main-predicate relation, a clause and phrase boundary; introducing a cross sequence labeling mechanism; and generating a depth annotation output of the text. The invention not only can identify the local mode in the text, such as the relation between the word and the phrase, but also can grasp the global structure and the semantic flow of the whole text.
Description
Technical Field
The invention relates to the technical field of machine translation, in particular to a sequence labeling method in natural language processing.
Background
In the field of machine translation, it is a very challenging task to accurately understand and convert source language text into target language, which requires not only direct translation of words and phrases, but also deep understanding of the syntactic structure and semantic information of the language, and conventional machine translation systems, such as rule-based translation and statistical machine translation, tend to focus on local text segments, while ignoring global context and deep semantic relationships of the text, resulting in an inability to efficiently handle complex language structures and semantic expressions, such as long-distance dependencies and subtle context changes.
With the development of deep learning technology, although Neural Machine Translation (NMT) has made significant progress in dealing with these problems, there are still problems of insufficient understanding of long-distance context dependencies and insufficient processing of syntactic and semantic information. These problems directly affect translation quality, especially when dealing with complex sentence structures and text that contains rich semantics.
In addition, the conventional sequence labeling method generally processes different language features (such as parts of speech, entities, syntax structures and the like) independently in the preprocessing stage of machine translation, so that the information islanding problem is caused, namely, the lack of effective information exchange and utilization between different features, thereby limiting deep understanding and accurate translation of source text.
Therefore, there is a need for a machine translation method that can effectively incorporate sequence labeling methods in natural language processing techniques to achieve comprehensive understanding of deep semantics and structure of source text.
Disclosure of Invention
Based on the above object, the present invention provides a sequence labeling method in natural language processing.
A sequence labeling method in natural language processing comprises the following steps:
S1: receiving text data input, and preprocessing, including word segmentation, stop word removal and normalization processing, to create a foundation for subsequent sequence annotation;
S2: performing part-of-speech tagging on the preprocessed text, automatically identifying the part of speech of each word by using a deep learning model, and providing grammatical clues for entity identification in the text;
S3: identifying and classifying the entities in the text by applying a sequence labeling model, wherein the entity comprises names, places and mechanisms, and labeling the attributes of the entities, such as time, quantity and position;
S4: combining a self-attention mechanism and position coding to process word sequences in sequence labeling, not only recognizing local modes, but also understanding global context, optimizing entity recognition and attribute labeling by considering global context relation and global text structure, and solving the problem of neglecting long-distance dependence in the traditional sequence labeling method;
s5: performing deep syntactic analysis based on the self-attention mechanism in S4, marking the syntactic structure of each sentence in the text, including a main-predicate-guest relationship, clauses and phrase boundaries, and providing structural information for semantic role marking;
S6: introducing a cross sequence labeling mechanism, and performing cross verification and fusion on labels generated by different labeling tasks to solve the problem of information island caused by independent processing of each task in the traditional sequence labeling method, and transmitting and sharing information among different labeling tasks through cross verification;
s7: and generating a depth marking output of the text by combining the results, wherein the output contains comprehensive information of the part of speech, the entity category and the syntactic structure.
Further, the deep learning model in S2 adopts a recurrent neural network model RNN, and S2 specifically includes:
s21: inputting the preprocessed text into an RNN model designed to process sequence data, and processing the vocabulary sequence in the input text by its internal state (memory);
s22: for each vocabulary, the RNN model predicts its part of speech by considering the preceding vocabulary;
S23: in the RNN model training stage, training an RNN model by using a training data set with correct part of speech tagging, and learning a sequence mode of vocabulary and how to correctly tag the part of speech based on context by using the training data set;
S24: after training is completed, feeding the preprocessed text data into a trained RNN model for part-of-speech tagging, and outputting a part-of-speech sequence, wherein each word corresponds to a part-of-speech tag;
S25: the sequence processing power of the RNN model is utilized to optimize the model to process complex text structures and to improve the accuracy of part-of-speech tagging, including considering contextual information both before and after by adding levels or introducing bi-directional RNN structures.
Further, the sequence labeling model in S3 adopts a bi-directional encoder characterization model BERT, and S3 specifically includes:
S31: inputting the preprocessed and part-of-speech tagged text into a BERT model, wherein the BERT model captures deep semantics and context relation of each word in the text by using a pre-trained contextualized word representation thereof;
s32: for each word in the text, the BERT model generates a high-dimensional vector representation that captures the contextual meaning of the word, for each word in the text sequence The BERT model outputs the corresponding code vector；
S33: based on the BERT model, a sequence labeling layer is added for processing the output vector of BERT and distributing entity labels for each vocabulary, wherein the sequence labeling layer is a full connection layer, specifically for each code vectorThe probability distribution of the entity tags is calculated through the full connection layer: Wherein, the method comprises the steps of, wherein, Is a wordIs a probability distribution of the entity tag of (c),Is a weight matrix of the full connection layer,Is a bias term, the softmax function is used to convert the output into a probability distribution;
S34: in the training process, parameters of the BERT model and the sequence labeling layer are adjusted by minimizing a loss function of entity labeling, and the recognition and classification capacity of the model to the entity is optimized;
S35: and after the entity identification and classification are completed, the attribute marking is carried out on the identified entity by utilizing the deep semantic understanding capability of the BERT model, and the specific attribute of the entity is identified.
Further, the step S4 specifically includes:
s41: applying a position code to each word in the sequence to generate a position-dependent vector representation, ensuring that the position of the word in the text sequence can be identified, the position code being a fixed code based on sine and cosine functions;
S42: adding the position codes and word vectors of the words to obtain a comprehensive representation containing both vocabulary content and position information;
S43: in the self-attention mechanism, the attention score of each word in the sequence to all other words is calculated to capture the dependency between the different words, for each word in the sequence Attention score thereofFor wordsThe calculation is as follows: Wherein, the method comprises the steps of, wherein, By comparing wordsAndIs obtained by dot product of the encoded vectors of (2) and represents the similarity between them;
S44: using the representation of each word in the attention score weighted sequence to obtain a weighted representation of each word in context;
S45: the weighted representation is used to identify local patterns in the text as well as global context, thereby taking into account both local and global information in the sequence annotation process.
Further, the step S5 specifically includes:
S51: based on the attention score in S43 Constructing a global syntax dependency graph, wherein each node in the graph represents a word, the edges connecting each node represent the syntax dependency relationship between words, and the weight of the edges is determined by the attention score;
s52: based on the syntax dependency graph, a graph neural network processing algorithm is adopted to identify main-predicate-guest relationships, clause and phrase boundaries, the global dependency graph is converted into series syntax structure labels, each label corresponds to one component or relationship in a sentence, each sentence component and relationship are marked, and a detailed syntax structure containing the main-predicate-guest relationships, clause and phrase boundaries is obtained.
Further, the introducing cross sequence labeling mechanism in S6 specifically includes:
S61: after part-of-speech tagging, entity identification and syntactic analysis are completed, collecting tag data generated by each tagging task, wherein the tag data comprises text information obtained through analysis from different angles;
S62: designing a multi-task learning framework, wherein part of the network structure is shared to learn the general features in each labeling task, and a corresponding task network layer is reserved for each task to capture the special features of the task;
S63: in the multi-task learning process, information is transmitted through a sharing layer, so that information flow and interaction between different labeling tasks are allowed;
S64: by means of the cross-validation technology, information is cross-validated among different labeling tasks, labeling errors or contradictions are identified and corrected by comparing labeling results of the different tasks, and overall accuracy and consistency of labeling are improved;
S65: in the training process, a joint optimization strategy is adopted, the loss functions of all labeling tasks are optimized at the same time, and the mutual influence and constraint among different tasks are considered.
Further, the multi-task learning framework is provided withMarking tasks by different sequences, each taskCorresponding to a specific labeling target, inputting text for a shared network structureFeature representation converted to sharing：WhereinIs a transfer function of the shared layer;
Conversion function of the shared layer The definition is as follows:
Input embedding: for a given input text sequence First, each word is divided intoEmbedded vector conversion into high-dimensional space；
Position coding: to preserve the order information of the words in the sequence, for each embedded vectorAdding position codingGenerating location-aware embeddings；
Transformer layer: embedding location awarenessInput to the transducer layer, computation of shared context sensitive feature representations through self-attention and feed forward networks：
。
Further, the corresponding task network layer performs, for each labeling taskAre all provided with a network layerTo handle shared featuresAnd outputs task-specific labeling results：WhereinIs the firstA plurality of task-specific transfer functions;
The transfer function The definition is as follows:
task-specific feed forward network: context sensitive features obtained from shared layers Processing features using one or more feed forward network layersTo capture task-specific patterns and relationships: Wherein, the method comprises the steps of, wherein, Represent the firstA task specific feed forward network;
And (3) output processing: depending on the nature of the task, The probability distribution is converted by the softmax layer for classification tasks.
Further, the joint optimization strategy comprises the step of enabling loss functions of all labeling tasksOptimizing in combination, total lossIs a weighted sum of the individual task losses: Wherein, the method comprises the steps of, wherein, Is the firstThe weight of the individual tasks is determined,Is a true annotation of the object,Is the calculation prediction labelAnd true annotationA loss function of the difference between them.
Further, the step S7 specifically includes:
Integrating the cross-validation results: utilizing results in a cross sequence labeling mechanism, wherein the output of each labeling task is optimized through a cross verification and fusion process, and consistent labeling information is provided for each word or phrase;
Constructing a comprehensive annotation frame: for each word or phrase in the text, aggregating the labeling results after optimizing each task into a comprehensive labeling set, wherein the comprehensive labeling set comprises information of part of speech, entity category and syntactic relation;
generating depth annotation output: and synthesizing the aggregated information to generate a depth annotation output for the whole text.
The invention has the beneficial effects that:
According to the invention, long-distance dependency in the sequence can be captured through a self-attention mechanism, the problem of long-distance dependency possibly neglected in the traditional sequence labeling method is solved, the global view angle enables entity identification and attribute labeling to be more accurate, the context relation crossing a long text paragraph can be understood, the self-attention mechanism is applied in deep syntactic analysis, the main-predicate-guest relation, clause and phrase boundary in the text can be effectively marked, fine syntactic understanding provides a solid structural information foundation for semantic role labeling, so that semantic analysis is more accurate and deep, the local mode in the text such as the relation of words and phrases can be identified, the global structure and semantic flow of the whole text can be mastered, the comprehensive understanding is a high-level natural language processing task, and more abundant and accurate language information is provided, so that the performance and reliability of the application are improved.
According to the method, through a cross sequence labeling mechanism, different labeling tasks such as part-of-speech labeling, entity identification, syntactic analysis and semantic role labeling can be mutually verified and optimized, the problem of information island possibly caused by independent processing of each task in a traditional sequence labeling method is effectively solved by the mechanism, the labeling accuracy and consistency are remarkably improved, and the text information obtained from different angles can be mutually supplemented and corrected by the aid of a comprehensive multi-task learning framework and the application of cross verification, so that errors are reduced, and the reliability of results is enhanced.
The invention generates the deep annotation output by integrating the annotation information of each level, so that the understanding of the text is not limited to the vocabulary or the syntactic structure of the surface layer, but extends to the deeper semantic and relation, the deeper understanding is the subsequent advanced natural language processing task, and a rich and accurate information basis is provided, thereby improving the performance and accuracy of the systems.
According to the invention, the multi-task learning process is optimized through the structures of the sharing layer and the task specific layer, different sequence labeling tasks are allowed to share the language characteristics of the bottom layer, and meanwhile, the independence and the specificity among the tasks are maintained, so that the learning efficiency is improved, the repeated calculation and the resource consumption are reduced, and the overall effect of multi-task learning is remarkably improved through the fine-granularity information fusion and mutual enhancement.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only of the invention and that other drawings can be obtained from them without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a sequence labeling method according to an embodiment of the invention;
fig. 2 is a schematic diagram of a cross sequence labeling mechanism according to an embodiment of the present invention.
Detailed Description
The present invention will be further described in detail with reference to specific embodiments in order to make the objects, technical solutions and advantages of the present invention more apparent.
It is to be noted that unless otherwise defined, technical or scientific terms used herein should be taken in a general sense as understood by one of ordinary skill in the art to which the present invention belongs. The terms "first," "second," and the like, as used herein, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that elements or items preceding the word are included in the element or item listed after the word and equivalents thereof, but does not exclude other elements or items. The terms "connected" or "connected," and the like, are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", etc. are used merely to indicate relative positional relationships, which may also be changed when the absolute position of the object to be described is changed.
As shown in fig. 1-2, a sequence labeling method in natural language processing includes the following steps:
S1: receiving text data input, and preprocessing, including word segmentation, stop word removal and normalization processing, to create a foundation for subsequent sequence annotation;
S11: and carrying out word segmentation processing on the received text data, and segmenting the continuous text into independent vocabulary units by adopting a language specific word segmentation algorithm so as to facilitate subsequent part-of-speech tagging and entity recognition tasks.
S12: stop words are removed from the segmented text, and a predefined stop word list (such as prepositions, pronouns, auxiliary verbs and the like) is used for removing words which have little influence on the marking task of the subsequent sequence from the text, so that noise is reduced, and processing efficiency is improved.
S13: the normalization processing is carried out on the rest vocabulary, including unifying the case of the vocabulary, converting the morphological change (such as unifying different time states of verbs into basic forms), eliminating synonym difference and the like, so as to reduce the diversity and complexity of texts and ensure the consistency and accuracy of the sequence labeling process.
S2: performing part-of-speech tagging on the preprocessed text, automatically identifying the part of speech of each word by using a deep learning model, and providing grammatical clues for entity identification in the text;
S3: identifying and classifying the entities in the text by applying a sequence labeling model, wherein the entity comprises names, places and mechanisms, and labeling the attributes of the entities, such as time, quantity and position;
S4: combining a self-attention mechanism and position coding to process word sequences in sequence labeling, not only recognizing local modes, but also understanding global context, optimizing entity recognition and attribute labeling by considering global context relation and global text structure, and solving the problem of neglecting long-distance dependence in the traditional sequence labeling method;
s5: performing deep syntactic analysis based on the self-attention mechanism in S4, marking the syntactic structure of each sentence in the text, including a main-predicate-guest relationship, clauses and phrase boundaries, and providing structural information for semantic role marking;
S6: introducing a cross sequence labeling mechanism, and performing cross verification and fusion on labels generated by different labeling tasks to solve the problem of information island caused by independent processing of each task in the traditional sequence labeling method, and transmitting and sharing information among different labeling tasks through cross verification, for example, assisting part-of-speech labeling and semantic role labeling by using a result of syntactic analysis, thereby improving the overall labeling precision and consistency;
s7: and generating a depth marking output of the text by combining the results, wherein the output contains comprehensive information of the part of speech, the entity category and the syntactic structure.
The deep learning model in S2 adopts a cyclic neural network model RNN, and S2 specifically comprises the following steps:
s21: inputting the preprocessed text into an RNN model designed to process sequence data, and processing the vocabulary sequence in the input text by its internal state (memory);
S22: for each word, the RNN model predicts its part of speech by considering the preceding word, this sequence-dependent nature making RNNs particularly suited for part of speech tagging tasks, as part of speech is typically dependent on the context of neighboring words;
S23: in the RNN model training stage, training an RNN model by using a training data set with correct part of speech tagging, and learning a sequence mode of vocabulary and how to correctly tag the part of speech based on context by using the training data set;
S24: after training is completed, feeding the preprocessed text data into a trained RNN model for part-of-speech tagging, and outputting a part-of-speech sequence, wherein each word corresponds to a part-of-speech tag;
S25: the sequence processing power of the RNN model is utilized to optimize the model to process complex text structures and to improve the accuracy of part-of-speech tagging, including considering contextual information both before and after by adding levels or introducing bi-directional RNN structures.
In the part-of-speech tagging using a Recurrent Neural Network (RNN), the core computation involves state update and output generation of the RNN, and the following computation description is used to describe how the RNN processes sequence data for part-of-speech tagging:
And (5) updating the state: for each element in the sequence (here, each word in the text), the RNN will calculate the current state from the current input and the previous state, the state update formula being: Wherein, the method comprises the steps of, wherein, Is the current timeIs a hidden state of (c).Is the input at the current time and corresponds to a word vector.Is the hidden state of the previous moment.AndRespectively, a weight matrix input to the hidden layer and hidden layer to hidden layer.Is a bias term.A nonlinear activation function such as tanh or ReLU.
And (3) output generation: hidden state per time stepThe part-of-speech tag used to calculate the output, i.e., the current word, is expressed as: Wherein, the method comprises the steps of, wherein, Is the moment of timeAnd the output of (2) represents the probability distribution of the part-of-speech tags.Is a hidden layer to output layer weight matrix.Is the hidden state at the current time.Is a bias term for the output layer.Typically a softmax function, is used to convert the output into a probability distribution, giving a probability to each possible part-of-speech tag.
In the part-of-speech tagging process of the RNN, the model traverses each word in the text, gradually updates the state by using the calculation process, and generates a part-of-speech probability distribution of each word. During training, these probabilities are compared with the actual parts of speech and model parameters are adjusted by back propagation algorithm、、、、To minimize the difference between the predicted and actual parts of speech and thereby improve the part of speech tagging capability of the model.
The sequence labeling model in S3 adopts a bi-directional encoder to characterize the model BERT, the BERT model is applied to carry out entity recognition and classification on the text after preprocessing and part-of-speech labeling, the models can learn and capture the contextual relation of words from the text, so that different types of entities such as names, places and organizations can be accurately identified, and in the entity recognition process, the model can allocate an entity label for each word or phrase in the text, for example, the names are labeled as 'names', the geographic positions are labeled as 'places', and the like. This process takes advantage of the model's ability to understand and classify different entities in the text, and for identified entities, attributes such as date, time, quantity attributes are noted. This requires that the model not only recognizes entities, but also understands specific properties or features of the entities and assigns them with corresponding property tags, and when labeling the model with training sequences, a large-scale corpus with detailed entity and property labels is used to ensure that the model can accurately learn representations of different entities and their properties, S3 specifically includes:
S31: inputting the preprocessed and part-of-speech tagged text into a BERT model, wherein the BERT model captures deep semantics and context relation of each word in the text by using a pre-trained contextualized word representation thereof;
s32: for each word in the text, the BERT model generates a high-dimensional vector representation that captures the contextual meaning of the word, for each word in the text sequence The BERT model outputs the corresponding code vector；
S33: based on the BERT model, a sequence labeling layer is added for processing the output vector of BERT and distributing entity labels for each vocabulary, wherein the sequence labeling layer is a full connection layer, specifically for each code vectorThe probability distribution of the entity tags is calculated through the full connection layer: Wherein, the method comprises the steps of, wherein, Is a wordIs a probability distribution of the entity tag of (c),Is a weight matrix of the full connection layer,Is a bias term, the softmax function is used to convert the output into a probability distribution;
S34: in the training process, parameters of the BERT model and the sequence labeling layer are adjusted by minimizing a loss function (cross entropy loss) of entity labeling, and the recognition and classification capacity of the model to the entity is optimized;
S35: and after the entity identification and classification are completed, the attribute marking is carried out on the identified entity by utilizing the deep semantic understanding capability of the BERT model, and the specific attribute of the entity is identified.
S4 specifically comprises the following steps:
s41: applying a position code to each word in the sequence to generate a position-dependent vector representation, ensuring that the position of the word in the text sequence can be identified, the position code being a fixed code based on sine and cosine functions;
S42: adding the position codes and word vectors of the words to obtain a comprehensive representation containing both vocabulary content and position information;
S43: in the self-attention mechanism, the attention score of each word in the sequence to all other words is calculated to capture the dependency between the different words, for each word in the sequence Attention score thereofFor wordsThe calculation is as follows: Wherein, the method comprises the steps of, wherein, By comparing wordsAndIs obtained by dot product of the encoded vectors of (2) and represents the similarity between them;
s44: using the representation of each word in the attention score weighted sequence to derive a weighted representation of each word in context such that the context information of the entire sequence is taken into account when processing each word;
S45: the weighted representation is used to identify local patterns in the text (e.g., usage of phrases or specific phrases) and global context (e.g., semantic streams in whole sentences or paragraphs) so that both local and global information is considered in the sequence labeling process.
S5 specifically comprises the following steps:
S51: based on the attention score in S43 Constructing a global syntax dependency graph, wherein each node in the graph represents a word, the edges connecting each node represent the syntax dependency relationship between words, and the weight of the edges is determined by the attention score;
s52: based on the syntax dependency graph, a graph neural network processing algorithm is adopted to identify main-predicate-guest relationships, clause and phrase boundaries, the global dependency graph is converted into series syntax structure labels, each label corresponds to one component or relationship in a sentence, each sentence component and relationship are marked, and a detailed syntax structure containing the main-predicate-guest relationships, clause and phrase boundaries is obtained.
The marking mechanism of the introduced cross sequence in the S6 specifically comprises the following steps:
S61: after part-of-speech tagging, entity identification and syntactic analysis are completed, collecting tag data generated by each tagging task, wherein the tag data comprises text information obtained through analysis from different angles;
S62: designing a multi-task learning framework, wherein part of the network structure is shared to learn the general features in each labeling task, and a corresponding task network layer is reserved for each task to capture the special features of the task;
S63: in the multi-task learning process, information is transmitted through a sharing layer, so that information flow and interaction between different labeling tasks are allowed, and knowledge learned from one task can be utilized by other tasks;
S64: by means of the cross-validation technology, information is cross-validated among different labeling tasks, labeling errors or contradictions are identified and corrected by comparing labeling results of different tasks, overall accuracy and consistency of labeling are improved, and information can be exchanged and validated in a sharing layer through different sequence labeling tasks through design of the sharing layer and a corresponding task network layer, so that knowledge transfer and integration among different tasks are promoted, and more comprehensive understanding and processing of text data are facilitated;
S65: in the training process, a joint optimization strategy is adopted, the loss functions of all labeling tasks are optimized at the same time, and the mutual influence and constraint among different tasks are considered.
The multi-task learning frame is provided withMarking tasks by different sequences, each taskCorresponding to a specific labeling target, inputting text for a shared network structureFeature representation converted to sharing：WhereinIs a transfer function of the shared layer;
Conversion function of shared layer The definition is as follows:
Input embedding: for a given input text sequence First, each word is divided intoEmbedded vector conversion into high-dimensional spaceThrough an embedding layer of a pre-trained Word embedding model such as Word2Vec, gloVe or BERT;
Position coding: to preserve the order information of the words in the sequence, for each embedded vector Adding position codingGenerating location-aware embeddings；
Transformer layer: embedding location awarenessInput to the transducer layer, computation of shared context sensitive feature representations through self-attention and feed forward networks：
。
The corresponding task network layer marks each taskAre all provided with a network layerTo handle shared featuresAnd outputs task-specific labeling results：WhereinIs the firstA plurality of task-specific transfer functions;
Conversion function The definition is as follows:
task-specific feed forward network: context sensitive features obtained from shared layers Processing features using one or more feed forward network layersTo capture task-specific patterns and relationships: Wherein, the method comprises the steps of, wherein, Represent the firstA task specific feed forward network;
And (3) output processing: depending on the nature of the task, Converting into probability distribution through a softmax layer for classifying tasks;
In this way, the transfer function of the layer is shared A generic context sensitive feature representation is provided, and the transfer function of each task specific layerSpecific labeling tasks are performed according to the features, and information communication and fusion between different tasks are allowed to be performed through a sharing layer, so that labeling performance and consistency of the whole system are enhanced.
The joint optimization strategy includes the loss function of all labeling tasksOptimizing in combination, total lossIs a weighted sum of the individual task losses: Wherein, the method comprises the steps of, wherein, Is the firstThe weight of the individual tasks is determined,Is a true annotation of the object,Is the calculation prediction labelAnd true annotationA loss function of the difference between them.
S7 specifically comprises the following steps:
Integrating the cross-validation results: utilizing results in a cross sequence labeling mechanism, wherein the output of each labeling task (such as part of speech, entity class, syntax structure) is optimized through a cross verification and fusion process, and consistent labeling information is provided for each word or phrase;
Constructing a comprehensive annotation frame: for each word or phrase in the text, the labeling results after optimizing each task are aggregated into a comprehensive labeling set, the comprehensive labeling set contains information of part of speech, entity category and syntactic relation, the aggregation considers mutual verification and information fusion in the process of labeling the cross sequences, and consistency and complementarity between labeling results of each dimension are ensured;
Generating depth annotation output: the aggregated information is synthesized to generate a deep annotation output for the whole text, the language attribute and structure of the text are described in detail by the output, and the deep annotation output comprises the part of speech, entity category, syntax role, semantic role and the like of each word or phrase.
Those of ordinary skill in the art will appreciate that: the discussion of any of the embodiments above is merely exemplary and is not intended to suggest that the scope of the invention is limited to these examples; the technical features of the above embodiments or in the different embodiments may also be combined within the idea of the invention, the steps may be implemented in any order and there are many other variations of the different aspects of the invention as described above, which are not provided in detail for the sake of brevity.
The present invention is intended to embrace all such alternatives, modifications and variances which fall within the broad scope of the appended claims. Therefore, any omission, modification, equivalent replacement, improvement, etc. of the present invention should be included in the scope of the present invention.
Claims (10)
1. A sequence labeling method in natural language processing, comprising the steps of:
S1: receiving text data input, and preprocessing, including word segmentation, stop word removal and normalization processing, to create a foundation for subsequent sequence annotation;
S2: performing part-of-speech tagging on the preprocessed text, and automatically identifying the part of speech of each word by using an deep learning model;
S3: identifying and classifying the entities in the text by applying a sequence labeling model, and labeling the attributes of the entities at the same time;
S4: combining a self-attention mechanism and position coding to process word sequences in sequence labeling, not only recognizing local modes, but also understanding global context, optimizing entity recognition and attribute labeling by considering global context relation and global text structure, and solving the problem of neglecting long-distance dependence in the traditional sequence labeling method;
s5: performing deep syntactic analysis based on the self-attention mechanism in S4, marking the syntactic structure of each sentence in the text, including a main-predicate-guest relationship, clauses and phrase boundaries, and providing structural information for semantic role marking;
S6: introducing a cross sequence labeling mechanism, and performing cross verification and fusion on labels generated by different labeling tasks to solve the problem of information island caused by independent processing of each task in the traditional sequence labeling method, and transmitting and sharing information among different labeling tasks through cross verification;
s7: and generating a depth marking output of the text by combining the results, wherein the output contains comprehensive information of the part of speech, the entity category and the syntactic structure.
2. The sequence labeling method in natural language processing according to claim 1, wherein the deep learning model in S2 adopts a recurrent neural network model RNN, and the S2 specifically includes:
S21: inputting the preprocessed text into an RNN model designed to process sequence data, and processing vocabulary sequences in the input text through the internal states thereof;
s22: for each vocabulary, the RNN model predicts its part of speech by considering the preceding vocabulary;
S23: in the RNN model training stage, training an RNN model by using a training data set with correct part of speech tagging, and learning a sequence mode of vocabulary and how to correctly tag the part of speech based on context by using the training data set;
S24: after training is completed, feeding the preprocessed text data into a trained RNN model for part-of-speech tagging, and outputting a part-of-speech sequence, wherein each word corresponds to a part-of-speech tag;
S25: the sequence processing power of the RNN model is utilized to optimize the model to process complex text structures and to improve the accuracy of part-of-speech tagging, including considering contextual information both before and after by adding levels or introducing bi-directional RNN structures.
3. The sequence labeling method in natural language processing according to claim 2, wherein the sequence labeling model in S3 adopts a bi-directional encoder characterization model BERT, and the S3 specifically includes:
S31: inputting the preprocessed and part-of-speech tagged text into a BERT model, wherein the BERT model captures deep semantics and context relation of each word in the text by using a pre-trained contextualized word representation thereof;
s32: for each word in the text, the BERT model generates a high-dimensional vector representation that captures the contextual meaning of the word, for each word in the text sequence The BERT model outputs the corresponding encoding vector/>；
S33: based on the BERT model, a sequence labeling layer is added for processing the output vector of BERT and distributing entity labels for each vocabulary, wherein the sequence labeling layer is a full connection layer, specifically for each code vectorThe probability distribution of the entity tags is calculated through the full connection layer:/>Wherein/>Is a word/>Probability distribution of entity tags of/>Is the weight matrix of the full connection layer,/>Is a bias term, the softmax function is used to convert the output into a probability distribution;
S34: in the training process, parameters of the BERT model and the sequence labeling layer are adjusted by minimizing a loss function of entity labeling, and the recognition and classification capacity of the model to the entity is optimized;
S35: and after the entity identification and classification are completed, the attribute marking is carried out on the identified entity by utilizing the deep semantic understanding capability of the BERT model, and the specific attribute of the entity is identified.
4. The method for sequence labeling in natural language processing according to claim 1, wherein S4 specifically comprises:
s41: applying a position code to each word in the sequence to generate a position-dependent vector representation, ensuring that the position of the word in the text sequence can be identified, the position code being a fixed code based on sine and cosine functions;
S42: adding the position codes and word vectors of the words to obtain a comprehensive representation containing both vocabulary content and position information;
S43: in the self-attention mechanism, the attention score of each word in the sequence to all other words is calculated to capture the dependency between the different words, for each word in the sequence Its attention score/>For words/>The calculation is as follows: wherein/> By comparing words/>And/>Is obtained by dot product of the encoded vectors of (2) and represents the similarity between them;
S44: using the representation of each word in the attention score weighted sequence to obtain a weighted representation of each word in context;
S45: the weighted representation is used to identify local patterns in the text as well as global context, thereby taking into account both local and global information in the sequence annotation process.
5. The method for sequence labeling in natural language processing according to claim 4, wherein S5 specifically comprises:
S51: based on the attention score in S43 Constructing a global syntax dependency graph, wherein each node in the graph represents a word, the edges connecting each node represent the syntax dependency relationship between words, and the weight of the edges is determined by the attention score;
s52: based on the syntax dependency graph, a graph neural network processing algorithm is adopted to identify main-predicate-guest relationships, clause and phrase boundaries, the global dependency graph is converted into series syntax structure labels, each label corresponds to one component or relationship in a sentence, each sentence component and relationship are marked, and a detailed syntax structure containing the main-predicate-guest relationships, clause and phrase boundaries is obtained.
6. The sequence labeling method in natural language processing according to claim 1, wherein the introducing cross sequence labeling mechanism in S6 specifically comprises:
S61: after part-of-speech tagging, entity identification and syntactic analysis are completed, collecting tag data generated by each tagging task, wherein the tag data comprises text information obtained through analysis from different angles;
S62: designing a multi-task learning framework, wherein part of the network structure is shared to learn the general features in each labeling task, and a corresponding task network layer is reserved for each task to capture the special features of the task;
S63: in the multi-task learning process, information is transmitted through a sharing layer, so that information flow and interaction between different labeling tasks are allowed;
S64: by means of the cross-validation technology, information is cross-validated among different labeling tasks, labeling errors or contradictions are identified and corrected by comparing labeling results of the different tasks, and overall accuracy and consistency of labeling are improved;
S65: in the training process, a joint optimization strategy is adopted, the loss functions of all labeling tasks are optimized at the same time, and the mutual influence and constraint among different tasks are considered.
7. The method for sequence annotation in natural language processing as claimed in claim 6, wherein the multi-task learning framework is provided withEach task/>, with different sequences labeling the tasksFor a shared network structure, text/>, corresponding to a specific labeling target, is inputConverted to shared feature representation/>：/>Wherein/>Is a transfer function of the shared layer;
Conversion function of the shared layer The definition is as follows:
Input embedding: for a given input text sequence First, each word/>Embedded vector/>, converted into high-dimensional space；
Position coding: to preserve the order information of the words in the sequence, for each embedded vectorAdding position coding/>Generating location-aware embeddings/>；
Transformer layer: embedding location awarenessInput to the transducer layer, computation of shared context sensitive feature representation by self-attention and feed forward network/>：
。
8. The method of claim 7, wherein the corresponding task network layer, for each labeling taskAre all provided with a network layer/>To handle shared features/>And outputs task-specific labeling results/>：/>Wherein/>Is/>A plurality of task-specific transfer functions;
The transfer function The definition is as follows:
task-specific feed forward network: context sensitive features obtained from shared layers Processing features using one or more feed forward network layers/>To capture task-specific patterns and relationships: /(I)Wherein/>Represents the/>A task specific feed forward network;
And (3) output processing: depending on the nature of the task, The probability distribution is converted by the softmax layer for classification tasks.
9. The method for sequence labeling in natural language processing according to claim 8, wherein the joint optimization strategy comprises assigning all labeling tasks to loss functionsTogether optimize, total loss/>Is a weighted sum of the individual task losses: /(I)Wherein/>Is/>Weights of individual tasks,/>Is a true annotation,/>Is to calculate prediction labels/>And true annotation/>A loss function of the difference between them.
10. The method for sequence labeling in natural language processing according to claim 9, wherein S7 specifically comprises:
Integrating the cross-validation results: utilizing results in a cross sequence labeling mechanism, wherein the output of each labeling task is optimized through a cross verification and fusion process, and consistent labeling information is provided for each word or phrase;
Constructing a comprehensive annotation frame: for each word or phrase in the text, aggregating the labeling results after optimizing each task into a comprehensive labeling set, wherein the comprehensive labeling set comprises information of part of speech, entity category and syntactic relation;
generating depth annotation output: and synthesizing the aggregated information to generate a depth annotation output for the whole text.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410431577.1A CN118036577A (en) | 2024-04-11 | 2024-04-11 | Sequence labeling method in natural language processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410431577.1A CN118036577A (en) | 2024-04-11 | 2024-04-11 | Sequence labeling method in natural language processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118036577A true CN118036577A (en) | 2024-05-14 |
Family
ID=90989680
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410431577.1A Pending CN118036577A (en) | 2024-04-11 | 2024-04-11 | Sequence labeling method in natural language processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118036577A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018072563A1 (en) * | 2016-10-18 | 2018-04-26 | 中兴通讯股份有限公司 | Knowledge graph creation method, device, and system |
| CN113255320A (en) * | 2021-05-13 | 2021-08-13 | 北京熙紫智数科技有限公司 | Entity relation extraction method and device based on syntax tree and graph attention machine mechanism |
| WO2022078102A1 (en) * | 2020-10-14 | 2022-04-21 | 腾讯科技（深圳）有限公司 | Entity identification method and apparatus, device and storage medium |
| CN114417872A (en) * | 2021-12-29 | 2022-04-29 | 航天科工网络信息发展有限公司 | Contract text named entity recognition method and system |
| CN116822517A (en) * | 2023-08-29 | 2023-09-29 | 百舜信息技术有限公司 | Multi-language translation term identification method |
| CN116992881A (en) * | 2023-07-14 | 2023-11-03 | 西华大学 | Method for extracting entity relationship based on context dependency perception graph convolution network |
-
2024
- 2024-04-11 CN CN202410431577.1A patent/CN118036577A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018072563A1 (en) * | 2016-10-18 | 2018-04-26 | 中兴通讯股份有限公司 | Knowledge graph creation method, device, and system |
| WO2022078102A1 (en) * | 2020-10-14 | 2022-04-21 | 腾讯科技（深圳）有限公司 | Entity identification method and apparatus, device and storage medium |
| CN113255320A (en) * | 2021-05-13 | 2021-08-13 | 北京熙紫智数科技有限公司 | Entity relation extraction method and device based on syntax tree and graph attention machine mechanism |
| CN114417872A (en) * | 2021-12-29 | 2022-04-29 | 航天科工网络信息发展有限公司 | Contract text named entity recognition method and system |
| CN116992881A (en) * | 2023-07-14 | 2023-11-03 | 西华大学 | Method for extracting entity relationship based on context dependency perception graph convolution network |
| CN116822517A (en) * | 2023-08-29 | 2023-09-29 | 百舜信息技术有限公司 | Multi-language translation term identification method |
Non-Patent Citations (2)
| Title |
|---|
| 原旎;卢克治;袁玉虎;舒梓心;杨扩;张润顺;李晓东;周雪忠;: "基于深度表示的中医病历症状表型命名实体抽取研究", 世界科学技术-中医药现代化, no. 03, 20 March 2018 (2018-03-20) * |
| 张乐;唐亮;易绵竹;: "融合多策略的军事领域中文术语抽取研究", 现代计算机, no. 26, 15 September 2020 (2020-09-15) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220147836A1 (en) | Method and device for text-enhanced knowledge graph joint representation learning | |
| CN110020438A (en) | Enterprise or tissue Chinese entity disambiguation method and device based on recognition sequence | |
| CN112380863A (en) | Sequence labeling method based on multi-head self-attention mechanism | |
| CN113743099B (en) | System, method, medium and terminal for extracting terms based on self-attention mechanism | |
| CN113255320A (en) | Entity relation extraction method and device based on syntax tree and graph attention machine mechanism | |
| CN113987169A (en) | Text abstract generation method, device and equipment based on semantic block and storage medium | |
| CN116204674B (en) | Image description method based on visual concept word association structural modeling | |
| CN115906815B (en) | Error correction method and device for modifying one or more types of error sentences | |
| CN114239574A (en) | Miner violation knowledge extraction method based on entity and relationship joint learning | |
| CN115062140A (en) | Method for generating abstract of BERT SUM and PGN fused supply chain ecological district length document | |
| CN114757184B (en) | Method and system for realizing knowledge question and answer in aviation field | |
| Xu et al. | Match-prompt: Improving multi-task generalization ability for neural text matching via prompt learning | |
| CN117313728A (en) | Entity recognition method, model training method, device, equipment and storage medium | |
| CN114048314A (en) | Natural language steganalysis method | |
| Wu et al. | Joint intent detection model for task-oriented human-computer dialogue system using asynchronous training | |
| Han et al. | A survey of unsupervised dependency parsing | |
| CN114388108A (en) | User feedback analysis method based on multi-task learning | |
| Zhou et al. | An attention-based model for joint extraction of entities and relations with implicit entity features | |
| CN116432611A (en) | Manuscript writing auxiliary method, system, terminal and storage medium | |
| CN116306653A (en) | Regularized domain knowledge-aided named entity recognition method | |
| Wang et al. | Orientation analysis for Chinese news based on word embedding and syntax rules | |
| CN115730232A (en) | Topic-correlation-based heterogeneous graph neural network cross-language text classification method | |
| CN118036577A (en) | Sequence labeling method in natural language processing | |
| CN115169429A (en) | Lightweight aspect-level text emotion analysis method | |
| Zhang et al. | Exploring aspect-based sentiment quadruple extraction with implicit aspects, opinions, and ChatGPT: a comprehensive survey |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |