CN117271701A - Method and system for extracting system operation abnormal event relation based on TGGAT and CNN - Google Patents

Abstract

The invention discloses a system operation abnormal event relation extraction method and system based on TGGAT and CNN, which are used for acquiring text data and preprocessing; processing the preprocessed text data by using a BERT model to generate a feature vector matrix; inputting the feature vector matrix into a multi-scale convolutional neural network CNN, and extracting the local features of sentences; inputting the feature vector matrix into a type guidance graph annotation network TGGAT, and extracting the global features of sentences; splicing the local features and the global features, and obtaining a characterization vector matrix weighted by the attention value through self-attention; the weighted token vector matrix is event relationship classified using a softmax classifier. And the type guidance graph annotation network is used for capturing long-range syntax dependency relationships and simultaneously capturing type information by considering different contributions of different types of dependency relationships, so that global event knowledge is accurately captured.

Description

A method for extracting abnormal event relationships in system operation based on TGGAT and CNN and system

Technical field

The present invention relates to text data recognition, specifically to a method and system for extracting abnormal event relationships in system operation based on TGGAT and CNN.

Background technique

Event relationship extraction is an important step in the event knowledge graph, which is of great benefit to natural language processing applications such as information extraction, investment strategies, and question answering. By capturing related relationships, related events can complement each other and fully realize their value.

Currently, in the task of event relationship extraction, most research focuses on extracting causal relationships and temporal relationships. Machine learning is widely used in the task of extracting event relationships. Kruengkrai et al. employed multi-column convolutional neural networks to process multiple sources of background knowledge to extract causal relationships that lack clear clues indicating the existence of event causality. Ning et al. combined integer linear programming and common sense knowledge to extract temporal relationships. Hu et al. use pre-trained language models to adaptively cluster temporal relationship features. Jiang et al. proposed a model that incorporates syntactic tree structures in classification. In order to improve the classification effect, Fan et al. proposed a relationship classification model that uses a dual-line neural network to extract syntactic information to complete the relationship classification task. Different ideas for handling document-level event causality have also been proposed, and Trong et al. designed a reinforcement learning mechanism to select key context from documents.

Since rich language knowledge is contained in syntactic dependencies, many methods based on syntactic dependencies have been proposed. Aldawsari et al. use an ancestor event of another event in a dependency tree while fusing discourse and narrative features to obtain a richer event representation. Meng et al. use a sequence encoder to encode the shortest dependency paths between events to identify relationships between them. Since sequence encoders cannot learn structural information well, some graph-based methods have been proposed to accomplish this task. Wang et al. proposed different constraints to extract temporal relationships and sub-event relationships. Zhang et al. proposed using graph transformers to capture temporal knowledge in syntax graphs. While using syntactic dependencies, Mathur et al. and Tran et al. also introduced other knowledge such as rhetoric, discourse, and semantics to enrich the representation of events through interactions between nodes in different graphs.

Some of these relationship extraction models are general relationship extraction methods, and some are relationship extraction models used in the financial field. There are no relationship extraction methods designed for the database operation and maintenance field. In addition, most existing dependency-based methods usually treat different types of dependencies equally when modeling the semantic context between events, leading to performance degradation in event relationship extraction.

Contents of the invention

Purpose of the invention: In view of the above shortcomings, the present invention provides a method for extracting abnormal event relationships in system operation based on TGGAT and CNN, which includes the following steps:

(1) Obtain text data and perform preprocessing;

(2) Use the BERT model to process the preprocessed text data and generate a feature vector matrix;

(3) Input the feature vector matrix into the multi-scale convolutional neural network CNN to extract the local features of the sentence; input the feature vector matrix into the type guided graph attention network TGGAT to extract the global features of the sentence;

(4) Splice local features and global features, and then obtain a representation vector matrix weighted by attention values through self-attention;

(5) Use the softmax classifier to classify event relationships on the weighted representation vector matrix.

Further, in step (1), the database system alarm log text data is obtained for preprocessing: tens of thousands of text data of alarm log records for more than ten years are obtained from databases on multiple business systems.

Further, the expression of the feature vector matrix V generated in step (2) is as follows:

Among them, t represents the sequence length, 768 represents the dimension of the word vector, cls represents the initial label of the sentence, seq represents the end label of the sentence, and v represents the word vector matrix.

Further, in step (3), the feature vector matrix is input into the multi-scale convolutional neural network CNN, and the extraction of local features of the sentence specifically includes:

(3.11) Perform one-dimensional convolution on the input eigenvector matrix to form local eigenvector f _i :

fi＝ReLu(k _i v _t:t+j-1 +b)

Among them, v represents the input word vector matrix, j represents the window size of the convolution kernel k _i , and b represents the bias value;

(3.12) Perform n convolution operations on the input feature vector matrix for n convolution kernels to form a high-dimensional vector F _t of the word vector v _t context feature set:

F _t ={f ₁ ,f ₂ ,...,f _n }

(3.13) Perform a maximum pooling operation to reduce the dimensionality of the high-dimensional vector F _t to form the local context feature M _t of the word vector v _t :

M _t =max(F _t ),

(3.14) For an input feature vector matrix of length t, use the convolution set K to scan the entire text to form a local feature set S of the entire text:

S＝{M ₁ ,M ₂ ,...,M _t }

Further, in step (3), the Stanford CoreNLP tool is used to obtain the syntactic dependency tree of the sentence in the input feature vector matrix, and by constructing the syntactic structure of the input sentence in the syntactic dependency tree, the sentence is converted into a graph structure for representation, where Words are represented as nodes, and the dependencies between words constructed from a syntactic dependency tree are represented as edges;

Input the feature vector matrix into the type guided graph attention network TGGAT, and extract the global features of the sentence, including:

(3.21) Use TGGAT to model the syntactic dependency tree of the sentence through type guidance, obtain the syntactic knowledge related to each word in the sentence, and then use the dependency path to transfer and aggregate the word information in the sentence to obtain each node and all The attention value of adjacent words is:

Among them, W represents the learnable weight matrix, a represents a single-layer feedforward feedback network, Represents the semantic vector of node i, /> Represents the semantic vector of node j, /> Represents the dependency path of node i and node j;

(3.22) Use the SoftMax function to normalize the attention value and obtain the attention coefficient:

α _i,r,j =softmax(d _i,r,t )

(3.23) Through the weighted summation of attention coefficients and the addition of the initial information of the original node i, the new vector feature of the node i is obtained by aggregation calculation

Among them, σ represents the activation function, W _r and W ₀ represent the learnable weights, _Ni is the set of all neighbor nodes j of node i in the syntactic dependency graph, and R is the set of all edges of node i in the syntactic dependence graph;

(3.24) The global characteristics of the sentence are as follows:

Further, in step (4), local features and global features are spliced, and then a representation vector matrix weighted by attention values is obtained through self-attention. The specific formula is as follows:

HS＝contact(H,S)

Q＝W ^q ·HS

K＝ ^Wk ·HS

V＝ ^Wv ·HS

A′=softmax(K ^T ·Q)

O＝V·A′

Among them, contact represents the splicing function, Q represents the query vector, W ^q represents the query matrix, K represents the key vector, W ^k represents the key matrix, V represents the value vector, W ^v represents the value matrix, and A′ represents the calculation of the query vector and key vector. Inner product and scale the weights obtained by normalization, O represents the representation vector matrix.

Further, in step (5), a softmax classifier is used to predict the relationship between event pairs in the text, and y is output to obtain event relationship classification:

y＝relu(O·w+b)

Among them, w and b are the parameters and bias entries of the fully connected layer.

The present invention also adopts a system operation abnormal event relationship extraction system based on TGGAT and CNN, including:

Obtain module, used to obtain text data and perform preprocessing;

The processing module is used to process the preprocessed text data using the BERT model and generate a feature vector matrix; input the feature vector matrix into the multi-scale convolutional neural network CNN to extract the local features of the sentence; input the feature vector matrix into the type guidance graph attention network TGGAT, extracts global features of sentences;

The splicing module is used to splice local features and global features, and then obtain a representation vector matrix weighted by attention values through self-attention;

The classification module is used to classify event relationships using a softmax classifier on the weighted representation vector matrix.

Beneficial effects: Compared with the existing technology, the significant advantage of the present invention is that the type-guided graph attention network can not only capture long-range syntactic dependencies, but also capture type information, which is crucial for capturing global contextual semantic information of events. Most sentences in the event relationship extraction task are long-range difficult sentences, in which two related words may be far apart, the syntactic structure is complex, and different types of dependencies may have different contributions, making it difficult to accurately capture the overall situation using only the surface information of the sentence. Event knowledge. Therefore, syntactic information is introduced to organize the structure of sentences, and type-related knowledge in syntax is modeled to further capture global features.

Description of the drawings

Figure 1 is a schematic diagram of the database abnormal event relationship extraction process of the present invention;

Figure 2 is a structural block diagram of the event relationship extraction system of the present invention;

Figure 3 is a schematic diagram of a syntactic dependency tree of an example sentence of the present invention;

Figure 4 is a schematic structural diagram of the attention mechanism of TGGAT in the present invention.

Detailed ways

Example 1

As shown in Figure 1, in this embodiment, a system operation abnormal event relationship extraction method based on TGGAT and CNN includes the following steps:

(1) Obtain tens of thousands of text data recorded in alarm logs for more than ten years from Oracle databases on multiple business systems.

(2) Perform data cleaning and preprocessing on the Oracle database log text data obtained in step (1). Data cleaning mainly includes removing unnecessary fields and text with inconsistent formats; preprocessing the cleaned database system log text data, dividing daily alarm logs into sentences, and using the BERT+CRF model to extract trigger words. For example, "The queue resource [is] deadlocked due to the transaction [waiting] for the resource", where "waiting" and "trapped" are the trigger words. Then, according to the type, cause, accompanying, disposition, carrying and sub-event categories of the event pairs, they are marked into five categories, respectively represented by numbers 0, 1, 2, 3 and 4 to form a data set. Finally, the data set is divided into a training set, a validation set and a test set in a ratio of 8:1:1 for model training of Oracle database event relationship extraction.

(3) Input the preprocessed data into the encoder. Use the BERT model to process preprocessed text data. Generate the input alarm text sentence {w ₁ , w ₂ ,..., w _t } into a feature vector matrix V. The expression of the eigenvector matrix V is as follows:

Among them, t is the sequence length, 768 is the dimension of the word vector, cls is the initial label of the sentence, seq is the end label of the sentence, v represents the word vector matrix, and each input sentence can be converted into a feature vector matrix through the Bert model .

(4) Input the feature vector matrix V generated in step (3) to the multi-scale convolutional neural network CNN layer to extract the local features of the sentence. The convolution kernel K is used to perform convolution calculation on the feature vector matrix V, and a pooling layer is designed to reduce the dimension of the obtained local feature F _t , and finally the syntactic features of the word vector are obtained. In the event relationship extraction task, K = {k ₁ , k ₂ ,..., k _n } is a set of convolution kernels, and n is the number of convolution kernels. Specifically include:

(4.11) Perform one-dimensional convolution on the input feature vector matrix, and the target word vector can form a local feature vector _fi . The specific formula is as follows:

f _i =ReLu(k _i v _t:t+j-1 +b)

Among them, v represents the input word vector matrix, j represents the window size of the convolution kernel k _i , and b represents the bias value.

(4.12) The entire set K of convolution kernels acts on the window center word vector and forms different local features of the word vector v _t , obtaining a high-dimensional vector F _t of the context feature set of the word vector v _t . n convolution operations are expressed as follows:

v _t =F _t ={f ₁ ,f ₂ ,...,f _n }

Among them, F _t is the set of contextual features formed by the target word v _t after n convolution operations.

(4.13) Since F _t is a high-dimensional vector with multiple features, pooling operations are used to reduce its dimensionality. Causal semantic role words as salient features can be retained as features using the max pooling operation, and its expression is:

M _t =max(F _t )

(4.14) The features m retained after each local feature vector f _i after the max pooling operation are fully connected to fix its dimension output, and finally form the local context representation of the central word vector v _t , whose expression is:

V _t =M _t ={m ₁ ,m ₂ ,...,m _n }

(4.15) For an input feature vector matrix of length t, the entire text is scanned using the convolution set K to form a local feature set S of the entire text with the following expression:

S={V ₁ , V ₂ ,..., V _t }={M ₁ , M ₂ ,..., M _t }

(5) Input the feature vector matrix V generated in step (3) into the TGGAT model, add type dependency to the calculation of the graph attention mechanism, and aggregate the key information of the text. Use the Stanford CoreNLP tool to obtain the syntactic dependency tree of the input sentence. For example, the syntactic dependency tree of "queue resources will be deadlocked due to transactions waiting for resources" is shown in Figure 3. Syntactic information is introduced to organize the structure of sentences, and type-related knowledge in syntax is modeled to further capture global features. After the BERT encoding layer, TGGAT is used to encode the syntactic tree to extract not only the semantic features of the words, but also the syntactic dependency features to enhance the model's understanding of the sentence and make the relationship extraction more accurate.

By constructing the syntactic structure of the input sentence in a syntactic dependency tree, representing words as nodes, and expressing the dependency relationships between words constructed in the syntactic dependency tree as edges, the sentence can be converted into a graph structure for representation. That is to say, in the figure, this article converts the type dependency information in the text into the corresponding adjacency matrix A through the syntactic dependency tree. Each element a _i,j in the matrix indicates whether there is a dependency relationship between the i-th word and the j-th word. If there is a dependency relationship between two words, then a _i,j =1, otherwise a _i,j =0, as shown in Figure 4.

Input the feature vector matrix into the type guided graph attention network TGGAT, and extract the global features of the sentence, including:

(5.11) In order to fully utilize the syntactic information of the sentence, use TGGAT to model the syntactic dependency tree of the sentence through type guidance to obtain the syntactic knowledge related to each word in the sentence, and then effectively utilize the dependency path to transfer and aggregate the sentences word information in . This enhances the feature representation of words. The attention value of each node and all adjacent words is:

Among them, W represents the learnable weight matrix, a represents a single-layer feedforward feedback network, Represents the semantic vector of node i, /> Represents the semantic vector of node j, /> Represents the dependency path of node i and node j.

(5.12) Use the SoftMax function to normalize the attention values of the central node and all adjacent entities. The normalized attention weight is the final attention coefficient:

(5.13) The new vector representation of a node is a weighted summation of the calculated attention coefficients, summed with the initial information of the original node. Aggregation calculation obtains the new vector feature of node i

Among them, σ represents the activation function, W _r and W ₀ represent the learnable weights, _Ni is the set of all neighbor nodes j of node i in the syntactic dependency graph, and R is the set of all edges of node i in the syntactic dependence graph.

(5.14) In order to further distinguish the importance of different contextual features, this solution adds a multi-head attention mechanism. The attention mechanism can meet the need to distinguish different contexts and help the model focus on some important information in the sentence while ignoring irrelevant contextual information:

(5.15) The global feature matrix of the input text is as follows:

(6) Splice the local feature vector obtained in step (4) and the global feature vector obtained in step (5), and use self-attention to calculate the attention value between each word and other words in the text, so that Words are related to each other, so that each word has different importance, and a weighted representation vector matrix is obtained. The specific formula is as follows:

HS＝contact(H,S)

Q＝W ^q ·HS

K＝ ^Wk ·HS

V＝ ^Wv ·HS

A′=softmax(K ^T ·Q)

O＝V·A′

Among them, contact represents the splicing function, Q represents the query vector, W ^q represents the query matrix, K represents the key vector, W ^k represents the key matrix, V represents the value vector, W ^v represents the value matrix, and A′ represents the calculation of the query vector and key vector. Inner product and scale the weights obtained by normalization, O represents the representation vector matrix.

(7) Use the softmax classifier to classify the event relationship on the weighted representation vector matrix, and output y to obtain the event relationship classification:

y＝relu(O·w+b)

Among them, w and b are the parameters and bias entries of the fully connected layer.

Example 2

As shown in Figure 2, in this embodiment, a system based on type-guided graph attention network (TGGAT) and CNN runs an abnormal event relationship extraction system, including an acquisition module for acquiring text data and performing Preprocessing; processing module, used to use the BERT model to process preprocessed text data and generate a feature vector matrix; input the feature vector matrix into the multi-scale convolutional neural network CNN to extract the local features of the sentence; guide the input type of the feature vector matrix The graph attention network TGGAT extracts the global features of the sentence; the splicing module is used to splice local features and global features, and then obtains the representation vector matrix weighted by the attention value through self-attention; the classification module is used to classify using softmax The device classifies event relationships on the weighted representation vector matrix.

Use the BERT model to encode the input text and obtain a semantic feature vector matrix. Then, in order to extract global information, the type-guided graph attention network uses type dependency knowledge as the guiding signal to encode the syntactic dependency tree of the text. At the same time, CNN is used to extract local information to enhance the representativeness of the text.

The type-guided graph attention network not only captures long-range syntactic dependencies but also type information, which is crucial for capturing global contextual semantic information of events. Most sentences in the event relationship extraction task are long-range difficult sentences, in which two related words may be far apart and have complex syntactic structures, making it difficult to accurately capture global event knowledge using only the surface information of the sentence. Therefore, syntactic information is introduced to organize the structure of sentences, and type-related knowledge in syntax is modeled to further capture global features.

Claims (10)

1. The system operation abnormal event relation extraction method based on the TGGATT and the CNN is characterized by comprising the following steps:

(1) Acquiring text data and preprocessing;

(2) Processing the preprocessed text data by using a BERT model to generate a feature vector matrix;

(3) Inputting the feature vector matrix into a multi-scale convolutional neural network CNN, and extracting the local features of sentences; inputting the feature vector matrix into a type guide graph annotation network TGGATT, and extracting the global features of sentences;

(4) Splicing the local features and the global features, and obtaining a characterization vector matrix weighted by the attention value through self-attention;

(5) The weighted token vector matrix is event relationship classified using a softmax classifier.

2. The system operation abnormal event relation extraction method according to claim 1, wherein in the step (1), database system alarm log text data is obtained for preprocessing: tens of thousands of pieces of text data of the alarm log records are acquired in databases on a plurality of business systems.

3. The system operation abnormal event relation extracting method according to claim 1, wherein the expression of the feature vector matrix V generated in the step (2) is as follows:

where t represents the sequence length, 768 represents the dimension of the word vector, cls represents the initial tag of the sentence, seq represents the ending tag of the sentence, and v represents the word vector matrix.

4. The system operation abnormal event relation extraction method according to claim 3, wherein in the step (3), the feature vector matrix is input into a multi-scale convolutional neural network CNN, and the extracting of the local feature of the sentence specifically comprises:

(3.11) one-dimensional convolution of the input eigenvector matrix to form a local eigenvector f _i ：

f _i ＝ReLu(k _i v _t:t+j-1 +b)

Where v represents the input word vector matrix and j represents the convolution kernel k _i B represents the offset value;

(3.12) performing n convolution operations on the feature vector matrix input by the n convolution checks to form a word vector v _t High-dimensional vector F of context feature set _t ：

F _t ＝{f ₁ ,f ₂ ,...,f _n }

(3.13) for high-dimensional vector F _t Performing maximum pooling operation with reduced dimension to form word vector v _t Local context feature M of (2) _t ：

M _t ＝max(F _t ),

(3.14) for a feature vector matrix of input length t, scanning the entire text using a convolution set K to form a local feature set S of the entire text:

S＝{M ₁ ,M ₂ ,...,M _t }。

5. the system operation abnormal event relation extraction method according to claim 4, wherein in the step (3), a syntax dependency tree of the sentence in the input feature vector matrix is obtained through a Stanford CoreNLP tool, the sentence is converted into a graph structure by constructing a syntax structure of the input sentence in the syntax dependency tree, wherein the word is represented as a node, and the dependency relation between the words constructed by the syntax dependency tree is represented as an edge;

inputting the feature vector matrix into a type guide graph meaning network TGGATT, wherein the extracting of the global features of sentences specifically comprises the following steps:

(3.21) modeling a syntactic dependency tree of the sentence through type guidance using TGGATT to obtain syntactic knowledge related to each word in the sentence, and then transferring and aggregating word information in the sentence using dependency paths to obtain the attention value of each node and all adjacent words as:

where W represents a learnable weight matrix, a represents a single layer feed-forward feedback network,semantic vector representing node i, ++>Semantic vector representing node j, +.>Representing the dependency paths of node i and node j;

(3.22) normalizing the attention value using a SoftMax function to obtain an attention coefficient:

α _i,r,j ＝softmax(d _i,r,t )

(3.23) obtaining new vector characteristics of the node i by aggregate calculation through weighted summation of attention coefficients and addition of initial information of the original node i

Wherein sigma represents an activation function, W _r 、W ₀ Representing a learnable weight, N _i Is the set of all neighbor nodes j of node i in the syntax dependency graph, R is the nodeThe point i is a set of all edges in the syntax dependency graph;

the global features of the sentence (3.24) are as follows:

。

6. the method for extracting abnormal event relation in system operation according to claim 5, wherein in the step (4), the local feature and the global feature are spliced, and then the characterization vector matrix weighted by the attention value is obtained through self-attention, and the specific formula is as follows:

HS＝contact(H,S)

Q＝W ^q ·HS

K＝W ^k ·HS

V＝W ^v ·HS

A′＝softmax(K ^T ·Q)

O＝V·A′

wherein contact represents a splicing function, Q represents a query vector, W ^q Represents a query matrix, K represents a key vector, W ^k Representing a key matrix, V representing a value vector, W ^v Representing a value matrix, A' representing the weight obtained by calculating the inner product of the query vector and the key vector and scaling the normalization, and O representing the characterization vector matrix.

7. The system operation anomaly event relationship extraction method of claim 6, wherein the step (5) uses a softmax classifier to predict the relationship of event pairs in text and outputs y to obtain event relationship classifications:

y＝relu(O·w+b)

where w and b are parameters and offset entries for the fully connected layer.

8. A system operation abnormal event relation extraction system based on TGGAT and CNN, comprising:

the acquisition module is used for acquiring text data and preprocessing the text data;

the processing module is used for processing the preprocessed text data by using the BERT model to generate a feature vector matrix; inputting the feature vector matrix into a multi-scale convolutional neural network CNN, and extracting the local features of sentences; inputting the feature vector matrix into a type guide graph annotation network TGGATT, and extracting the global features of sentences;

the splicing module is used for splicing the local features and the global features, and then obtaining a characterization vector matrix weighted by the attention value through self-attention;

and the classification module is used for classifying the event relationship of the weighted characterization vector matrix by using a softmax classifier.

9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any one of claims 1 to 7 when the computer program is executed by the processor.

10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.