CN114896388A - A Hierarchical Multi-Label Text Classification Method Based on Mixed Attention - Google Patents

Abstract

The invention provides a hierarchical multi-label text classification method based on mixed attention, which uses pre-trained word vectors as word embedding and uses Bi-GRU to perform primary feature extraction on the input word embedding; modeling a hierarchical label structure system by using a graph convolution neural network, and generating a label representation containing label relevance; local feature extraction with different granularities is further carried out on the output of the Bi-GRU by using a plurality of convolution neural networks with different convolution kernel sizes, the output is spliced into a text feature after being subjected to maximum pooling, and the text feature is further subjected to feature extraction by using attention expressed based on labels; and meanwhile, performing global feature extraction on the output of the Bi-GRU by using a self-attention mechanism, performing self-adaptive fusion on the text features expressed based on the labels and the text features expressed based on the self-attention mechanism to obtain text expression based on mixed attention, extracting information among the labels through a relation network, and obtaining a final classification result through a multilayer perceptron.

Description

A Hierarchical Multi-Label Text Classification Method Based on Mixed Attention

technical field

The present invention relates to the field of computer information technology and natural language processing, and more particularly, to a method for hierarchical multi-label text classification based on mixed attention.

Background technique

The advent of the Internet era has enabled people to access various types of information more conveniently. At the same time, various media data are also continuously generated, which provides the basic conditions for mining valuable data on the Internet. The lack of efficient data management methods and means of acquiring knowledge is undoubtedly a waste. In data mining, text classification is one of the core problems.

The task of multi-label text classification is to select the subset most relevant to the text content within a given set of labels. In practical scenarios, a lot of data is related to multiple tags in the tag set, and these tags can concisely display the specific content of the data, enabling people to manage massive data more conveniently and effectively, and further analyze the data. Hierarchical multi-label text classification is a special case of multi-label text classification. Its label system has a hierarchical structure. The general multi-label text classification algorithm does not take into account the impact of the hierarchical label structure on the classification effect, and does not make full use of text labels. The correlation information between the data leads to inaccurate identification of the category to which it belongs during classification, especially for data with long-tailed distribution, the classification effect still has a large room for improvement. At the same time, most of the existing models focus on the local features or global features of the text, and lack comprehensive consideration of local features and global features, so that the important features involved in classification are insufficiently captured.

SUMMARY OF THE INVENTION

Aiming at the deficiencies of the prior art, the present invention provides a mixed attention-based hierarchical multi-label text classification method, which improves the hierarchical multi-label by using the label hierarchy for label semantic representation and making full use of the global and local semantic information of the text. Performance issues for text classification.

In order to achieve the above-mentioned purpose, the technical solution adopted in the present invention is: a method for classifying multi-label text based on mixed attention, the method comprises the following steps:

S1, multi-label text data preprocessing; text data is used to train the model, which consists of text content and its corresponding label set. All label categories in the entire data set are a dendrogram with a hierarchical relationship. For the dendrogram In terms of it, it consists of many nodes, each node represents a label category, and the label corresponding to each sample text in the dataset comes from the node on this label tree graph;

S2, for the text label, obtain the prior level information of the hierarchical classification system, the prior level information refers to the prior probability of the interdependence between the labels, which can be obtained by calculating the transition probability between the parent label and the child label;

S3, build a deep learning hierarchical multi-label text classification model;

The deep learning multi-label text classification model includes a word embedding module, a text encoding module, a label encoding module, a text representation module based on a label attention mechanism, a text representation module based on a self-attention mechanism, a feature fusion module, a vector regression layer, and a relationship. Network module, label probability prediction layer;

S4, input the preprocessed text data of the data set to the model training; after the model training is completed, use the trained model to classify the multi-label text.

In the above technical solution, step S1 includes performing data preprocessing on the samples in the data set D, which specifically includes the following steps: Step 1.1, performing word segmentation, removing stop words, and removing punctuation marks; The word frequency word_frequence, delete words whose occurrences are less than X1, record the filtered words, and build a word list. After the data set D is preprocessed, the data set D is divided into a training set, a validation set, and a test set according to a certain proportion.

In the above technical solution, step S2 includes: for the data of the training set in the data set D, assuming that there is a hierarchical path e _{i, j} between the parent node v _i and the child node v j , then the feature f of the edge formed by the parent-child node path (e _i,j ) is represented by the prior probabilities p(U _j |U _i ) and p(U _i |U _j ):

f(e _i,j ) represents the relationship between two nodes. This relationship is described by the transition probability or co-occurrence probability of the two nodes. The transition probability of the two nodes includes the relationship between the parent node and a child node. The transition probability p(U _j |U _i ), the transition probability p(U _i |U _j ) from the child node to the parent node, a parent label node may contain multiple child label nodes, the parent label node to each child label node The sum of the transition probabilities is 1. If there is only one child node under the parent label node, the value is 1 at this time; if there are multiple child labels, the value is less than 1 at this time, but their sum is 1; in the formula, U _j and U _i denote that the text sample is labeled as v _j node label and labeled as v _i node label, respectively, p(U _j |U _i ) is the conditional probability of being labeled as v _j node label given v _i . P(U _j ∩U _i ) is the probability that {v _j ,v _i } are marked simultaneously. N _j and N _i represent the number of v _j node labels and v _i node labels in the training set, respectively.

In the above technical solution, step 3 further includes performing word embedding processing on the input text and its tags through the word embedding module, and the word embedding processing method is specifically:

Step 2.1. Obtain the preprocessed text sequence, and convert the words {x ₁ ,x ₂ ,...,x _n } in the text into word vector representations {w ₁ ,w ₂ ,... by querying the word embedding dictionary table. .,w _n }, where n refers to the number of words in the preprocessed text.

Step 2.2. Obtain the label set {l ₁ ,l ₂ ,...,l _n } of hierarchical multi-label text classification, and convert the label set into a label embedding set {c ₁ , c ₂ ,...,c _n }.

In the above technical solution, step S3 further includes, performing encoding processing on the word vector representation {w ₁ , w ₂ , . . . , _wn } through the text encoding module, and the encoding processing method is specifically:

The word vector representations {w ₁ ,w ₂ ,...,w _n } of the text are encoded using the Bi-GRU network to generate implicit representations {h ₁ ,h ₂ ,...,h _n with contextual semantic information }. Then, the implicit representation {h ₁ , h ₂ ,...,h _n } is further sent to three convolutions with different kernel sizes, and three semantic vectors under different receptive fields are obtained. Finally, the three semantic The vectors are concatenated into a new semantic representation vector S={s ₁ , s ₂ ,...,s _n }.

In the above technical solution, step S3 further includes: performing encoding processing on the label vector representation {c ₁ ,c ₂ ,...,c _n } by the label encoding module, and the label encoding processing method is specifically:

The label vector representation {c ₁ ,c ₂ ,...,c _n } is encoded using a single-layer GCN to generate an implicit representation M={m ₁ ,m ₂ ,...,m _n with label-level association information }. The implementation process is as follows:

Hierarchical GCNs aggregate top-down, bottom-up, and self-loop data flows within edges. In hierarchical GCNs, each directed edge represents a pair of label-related features, and these data streams are transformed at nodes using linear transformations along the edges.

In order to realize the node transformation, the present invention uses a weighted adjacency matrix to represent this linear transformation, and the initial value of the weighted adjacency matrix comes from the prior level information of the hierarchical classification system in step two. Formally, a hierarchical GCN encodes its hidden state according to the relevant neighborhood of node k, where the neighborhood N(k) = {n _k , child(k), parent(k)}, n _k refers to the hierarchical label tree The kth label node in , child(k) refers to the child label node of the kth node, parent(k) refers to the parent label node of the kth node, and the hidden state of node k is calculated as follows:

及

是可训练的偏置参数；对于u_k,j及g_k,j而言，可以将u_k,j理解成结点k，j之间的信息，g_k,j理解成门控值，控制u_k,j最后对In the above formula, v _j , v _k are parameters that can be trained,

and

is a trainable bias parameter; for uk _,j and gk _,j , uk _,j can be understood as the information between nodes k, j, _gk,j as the gate value, control u _k,j last pair

b_l∈R^N×dim，及b_g∈R^N，dim为向量的维度大小，属于预先定义的超参数。d(j，k)表示从节点j到节点k的层次方向，包括自顶向下、自下而上和自循环边。其中，a_k,j∈R表示的是层次概率f_d(k,j)(e_kj)，f_d(k,j)(e_kj)指的是从第k个节点到第j个标签节点间的转移概率。它是通过上文中的f(e_(i,j))得到，自循环边采用a_k,k＝1，自上而下的边使用

自下而上的边使用f_p(e_j,k)＝1。上述边的特征矩阵F＝{a_0,0,a_0,1,…,a_c-1,c-1}表示的是文本标签有向层次图的加权邻接矩阵。最后，节点k的输出隐藏状态h_k表示其对应于层次结构信息的标签表示。The influence of node k; σ refers to the activation function in deep learning, which can be taken as the sigmoid function,

b _l ∈R ^N×dim , and b _g ∈R ^N , dim is the dimension of the vector and belongs to the predefined hyperparameters. d(j, k) represents the hierarchical direction from node j to node k, including top-down, bottom-up and self-loop edges. Among them, a _k,j ∈R represents the hierarchical probability f _d(k,j) (e _kj ), and f _d(k,j) (e _kj ) refers to the node from the kth node to the jth label node the transition probability between. It is obtained by f(e_(i,j)) above, using a _k,k = 1 for the self-loop edge, and using the top-down edge

Bottom-up edges use f _p (e _j,k )=1. The feature matrix F={a _0,0 ,a _0,1 ,..., _ac-1,c-1 } of the above edge represents the weighted adjacency matrix of the directed hierarchical graph of text labels. Finally, the output hidden state h _k of node k represents its label representation corresponding to the hierarchical information.

以及来自标签编码层的标签表示

d_c表示的是文本编码向量的维度大小，是一个预先确定的固定的值，通过以下公式计算基于标签注意力的文本表示：In the above technical solution, step S3 also includes a text representation module based on the label attention mechanism: representing the text from the text encoding layer

and the label representation from the label encoding layer

d _c represents the dimension size of the text encoding vector, which is a predetermined fixed value. The text representation based on label attention is calculated by the following formula:

Among them, α _kj represents the information amount of the j-th text feature vector for the k-th label. v _k is the text representation based on label attention.

通过以下公式计算基于自注意力机制的文本表示：In the above technical solution, step S3 further includes a text representation module based on a self-attention mechanism: representing the hidden layer text output from the Bi-GRU in the text encoding layer

The text representation based on the self-attention mechanism is calculated by the following formula:

Where w ₁ , w ₂ are parameters, h is the text representation, α _kt is the weight occupied by the t-th vector in the text representation, and _uk is the text representation based on the self-attention mechanism.

In the above technical solution, step S3 also includes a feature fusion module: adaptively fuse with the text feature based on the label attention mechanism and the text feature based on the self-attention mechanism to obtain the final text feature d _ik-fusio , the calculation method is as follows :

where w ₁ and w ₂ are parameters, v _k is the text representation based on label attention, _uk is the text representation based on self-attention, and β _k is the weight occupied by v _k .

In the above technical solution, step S3 also includes the relationship network module further mining the association information between the tags: the mining method is to input the text feature d _ik-fusion generated by the feature fusion module into the fully connected layer, and obtain the logits vector corresponding to each tag O. = _{ o ₁ ,o ₂ ,...,on }, then input the vector O to the relational network module to get the prediction vector y={y ₁ ,y ₂ ,...,y _n }, and finally the prediction vector y Input to the multi-layer perceptron, the label prediction probability can be obtained, and the relationship network is essentially a residual network.

In the above technical solution, step S4 includes the need to use a cross-entropy loss function in the training process, and use the Adam optimizer for training. The cross-entropy loss function for multi-label text classification is as follows:

为第i个样本对于第j个标签的预测概率，最终得到训练好的深度学习多标签文本分类模型，L指的是标签类别的数量，N指的是样例文本的数量。Among them, y _ij is the actual probability of the i-th sample for the j-th label,

is the predicted probability of the i-th sample for the j-th label, and finally a trained deep learning multi-label text classification model is obtained, where L refers to the number of label categories, and N refers to the number of sample texts.

The advantages and beneficial effects of the present invention are as follows:

The present invention uses Bi-GRU combined with CNN to extract the semantic representation of the text, which can sufficiently obtain the local semantic information of the text. In the present invention, the hierarchical information of hierarchical multi-label classification is represented by the graph neural network, and the label representation with hierarchical correlation information can be obtained. The present invention uses the self-attention mechanism to extract the semantic representation of the text, and can obtain the semantic representation of the global association of the text. The present invention uses the self-adaptive fusion of the text feature based on label representation and the text feature based on self-attention representation, and can obtain text representation that links global, local text and label information. The present invention uses a relational network in the last layer of the model, so that the original label prediction vector can further obtain the label correlation.

The present invention includes four aspects: firstly, using graph convolutional neural network to extract label representations containing hierarchical relationships; secondly, using multiple convolutions with different granularities to extract local features; thirdly, using label-based attention mechanism and Based on the self-attention mechanism, the text features are further extracted and adaptively fused (FA). The fourth is to use the relational network to further extract label associations. A method for hierarchical multi-label classification based on mixed attention provided by the present invention can mark the text with one or more labels by extracting text features from the input text to be classified, and then classifying the text through a multi-layer perceptron. Widely used in e-commerce, news, scientific papers and other fields.

Description of drawings

Fig. 1 is the flow chart of the hierarchical multi-label text classification method based on mixed attention of the present invention;

Fig. 2 is the network structure diagram of the hierarchical multi-label text classification model based on mixed attention of the present invention;

3 is a schematic diagram of the hierarchical structure of the hierarchical multi-label text classification label of the present invention;

4 is a schematic diagram of calculation of a hierarchical multi-label text classification graph convolutional neural network according to the present invention;

FIG. 5 is a schematic diagram of the relationship network of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

In order to achieve the above-mentioned purpose, the technical solution adopted in the present invention is: a method for classifying multi-label text based on mixed attention, the method comprises the following steps:

Step S1, preprocessing the multi-label text data in the dataset D;

Step S2, for the text label, obtain the prior level information of the hierarchical classification system, the prior level information refers to the prior probability of the interdependence between the labels, which can be obtained by calculating the transition probability between the parent label and the child label. ;

Step S3, constructing a deep learning hierarchical multi-label text classification model;

The deep learning multi-label text classification model includes a word embedding module, a text encoding module, a label encoding module, a text representation module based on a label attention mechanism, a text representation module based on a self-attention mechanism, a feature fusion module, a vector regression layer, and a relationship. Network module and label probability prediction layer;

In step S4, the preprocessed text data of the dataset is input to the model training, and after the model training is completed, the multi-label text is classified by using the trained model.

Preferably, step S1 includes the following steps:

Data preprocessing is performed on the samples in the dataset D, which includes the following steps:

Step 1-1, perform word segmentation on the text in data set D, remove stop words, and remove punctuation;

Step 1-2, count the word frequency word_frequence in the text in the data set D, delete the words whose occurrence times are less than X1, record the filtered words, and construct a word list.

Steps 1-3, after the data set D is preprocessed, the data set D is divided into training set, validation set and test set according to 3:1:1.

Preferably, step S2 includes the following steps:

For the data of the training set in dataset D, assume that there is a hierarchical path e _{i,j between} the parent node v _i and the child node v j . Then the feature f(e _i,j ) of the edge formed by the parent-child node path is represented by the prior probabilities p(U _j |U _i ) and p(U _i |U _j ):

f(e _i,j ) represents the relationship between two nodes. This relationship is described by the transition probability or co-occurrence probability of the two nodes. The transition probability of the two nodes includes the transition from the parent node to a certain child node. The probability p(U _j |U _i ), the transition probability p(U _i |U _j ) from the child node to the parent node, where U _j and U _i represent that the text data is marked as v _j node label and marked as ν _i node label, p(U _j |U _i ) is the conditional probability of being labelled as v _j node label given v _i , P(U _j ∩U _i ) is {v _j ,v _i } simultaneously labelled The probability of , N _j and N _i represent the number of v _j node labels and v _i node labels in the training set, respectively.

Preferably, step S3 also includes the following steps:

The word embedding module is used to perform word embedding processing on the input text and its tags. The word embedding processing method is as follows:

Step 2-1. Obtain the preprocessed text sequence, and convert the words {x ₁ , x ₂ ,...,x _n } in the text into word vector representation {w ₁ by querying the word embedding table (Glove-300d) ,w ₂ ,...,w _n }.

Step 2-2. Obtain the label set {l ₁ , l ₂ , ..., l _n } of hierarchical multi-label text classification, and convert the label set into a label embedding set with dimension 300 by means of kaiming coding {c ₁ ,c ₂ ,...,c _n }, where n refers to the number of words in the preprocessed text.

Preferably, step S3 also includes the following steps:

The word vector representation {w ₁ ,w ₂ ,...,w _n } is encoded by the encoding module, and the encoding processing method is as follows:

The word vector representations {w ₁ ,w ₂ ,...,w _n } of the text are encoded using the Bi-GRU network to generate implicit representations {h ₁ ,h ₂ ,...,h _n with contextual semantic information }. Then the implicit representation {h ₁ , h ₂ ,...,h _n } is further fed into three convolutions with kernel sizes of 2, 3, and 4, and the number of hidden layers is 100, and three different convolutions are obtained. Semantic vectors under the receptive field are respectively subjected to maximum pooling, and then three semantic vectors are spliced into a new semantic representation vector S={s ₁ , s ₂ ,...,s _n } with a dimension of 300.

Preferably, step S3 also includes the following steps:

The label vector representation {c ₁ ,c ₂ ,...,c _n } is encoded by the label encoding module. The label encoding processing method is specifically to use a single-layer GCN to encode the label vector representation {c ₁ ,c ₂ ,... ,c _n } is encoded to generate an implicit representation M={m ₁ ,m ₂ ,...,m _n } with label-level association information. The implementation process is as follows:

Hierarchical GCNs aggregate top-down, bottom-up, and self-loop data flows within edges. In hierarchical GCNs, each directed edge represents a pair of label-related features, and these data streams are transformed at nodes using linear transformations along the edges.

In order to realize the node transformation, the present invention uses a weighted adjacency matrix to represent this linear transformation, and the initial value of the weighted adjacency matrix comes from the prior level information of the hierarchical classification system in step S2. Formally, a hierarchical GCN encodes its hidden state according to the relevant neighborhood of node k, where the neighborhood N(k) = {n _k , child(k), parent(k)}, n _k refers to the hierarchical label tree The kth label node in , child(k) refers to the child label node of the kth node, parent(k) refers to the parent label node of the kth node, and the hidden state of node k is calculated as follows:

及

是可训练的偏置参数；对于u_k,j及g_k,j而言，可以将u_k,j理解成结点k，j之间的信息，g_k,j理解成门控值，控制u_k,j最后对节点k的影响大小；σ是指深度学习中的激活函数可以取为sigmoid函数，

b_l∈R^N×dim，及b_g∈R^N，dim为向量的维度大小，属于预先定义的超参数；d(j，k)表示从节点j到节点k的层次方向，包括自顶向下、自下而上和自循环边；其中，a_k,j∈R表示的是层次概率f_d(k,j)(e_kj)，f_d(k,j)(e_kj)指的是从第k个节点到第j个标签节点间的转移概率，它是通过f(e_i,j)得到，自循环边采用a_k,k＝1，自上而下的边使用

自下而上的边使用f_p(e_j,k)＝1；上述边的特征矩阵F＝{a_0,0,a_0,1,…,a_c-1,c-1}表示的是文本标签有向层次图的加权邻接矩阵，最后，节点k的输出隐藏状态h_k表示其对应于层次结构信息的标签表示。In the above formula, v _j , v _k are parameters that can be trained,

and

is a trainable bias parameter; for uk _,j and gk _,j , uk _,j can be understood as the information between nodes k, j, _gk,j as the gate value, control The final influence of u _{k, j} on node k; σ means that the activation function in deep learning can be taken as the sigmoid function,

b _l ∈R ^N×dim , and b _g ∈R ^N , dim is the dimension of the vector, which is a predefined hyperparameter; d(j, k) represents the hierarchical direction from node j to node k, including top-to-bottom Bottom, bottom-up, and self-loop edges; where a _k,j ∈ R denotes the hierarchical probability f _d(k,j) (e _kj ), and f _d(k,j) (e _kj ) refers to The transition probability from the kth node to the jth label node, which is obtained by f(e _i,j ), the self-loop edge adopts a _k,k = 1, and the top-down edge uses

The bottom-up edge uses f _p (e _j,k )=1; the characteristic matrix F={a _0,0 ,a _0,1 ,...,a _c-1,c-1 } of the above edge represents A weighted adjacency matrix of a directed hierarchical graph of text labels, and finally, the output hidden state h _k of node k represents its label representation corresponding to the hierarchical information.

Preferably, step S3 also includes the following steps:

以及来自标签编码层的标签表示

通过以下公式计算基于标签注意力的文本表示：The extraction method of the text representation module based on the label attention mechanism is: the text representation from the text encoding layer

and the label representation from the label encoding layer

The label attention based text representation is calculated by the following formula:

Among them, α _kj represents the information amount of the j-th text feature vector for the k-th label. v _k is the text representation based on label attention.

Preferably, step S3 also includes the following steps:

通过以下公式计算基于自注意力机制的文本表示：The extraction method of the text representation module based on the self-attention mechanism is specifically: the hidden layer text representation from the Bi-GRU output in the text encoding layer

The text representation based on the self-attention mechanism is calculated by the following formula:

Where w ₁ , w ₂ are parameters, h is the text representation, α _kt is the weight occupied by the t-th vector in the text representation, and _uk is the text representation based on the self-attention mechanism.

Preferably, step S3 also includes the following steps:

The feature fusion module is: adaptively fuse with the text features based on the label attention mechanism and the text features based on the self-attention mechanism to obtain the final text feature d _ik-fusion , and the calculation method is as follows:

where w ₁ and w ₂ are parameters, v _k is the text representation based on label attention, _uk is the text representation based on self-attention, and β _k is the weight occupied by v _k .

Preferably, step S3 also includes the following steps:

Use the relational network module to further mine the association information between tags: the mining method is to input the text feature d _ik-fusion generated by the feature fusion module into the fully connected layer, and obtain the logits vector corresponding to each tag O={o ₁ ,o ₂ , ...,on }, then input the vector O to the relational network module to get the predicted vector y={y ₁ ,y ₂ ,...,y _n }, and finally input the predicted vector y to the multi _- layer perceptron, namely The label prediction probability can be obtained, and the relationship network is essentially a residual network.

Preferably, step S4 includes the following steps:

In the training process, the cross-entropy loss function needs to be used, and the Adam optimizer is used for training. The cross-entropy loss function of multi-label text classification is as follows:

Among them, y _ij is the actual probability of the i-th sample for the j-th label, and is the predicted probability of the i-th sample for the j-th label, and finally the trained deep learning multi-label text classification model is obtained, and L refers to the label The number of categories, N refers to the number of text samples.

The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which the present invention pertains can make various modifications or additions to the described specific embodiments or substitute in similar manners, but will not deviate from the spirit of the present invention or go beyond the definitions of the appended claims range.

Claims (10)

1. A hierarchical multi-label text classification method based on mixed attention is characterized by comprising the following steps:

step S1, preprocessing the multi-label text data in the data set D;

step S2, acquiring prior hierarchy information of a hierarchy classification system aiming at the text labels, wherein the prior hierarchy information refers to prior probability of mutual dependence between the labels and can be obtained by calculating transition probability between a father label and a son label;

step S3, constructing a deep learning level multi-label text classification model;

the deep learning multi-label text classification model comprises a word embedding module, a text coding module, a label coding module, a text representation module based on a label attention mechanism, a text representation module based on an attention mechanism, a feature fusion module, a vector regression layer, a relation network module and a label probability prediction layer;

and step S4, inputting the text data after the data set preprocessing to model training, and after the model training is finished, classifying the multi-label text by using the trained model.

2. The method of claim 1, wherein the hierarchical multi-label text classification based on label attention comprises: in step S1, preprocessing the text data in the data set D includes the following steps:

step 1.1, performing word segmentation, removing stop words and removing punctuation marks;

step 1.2, counting word frequency word _ frequency in the text in the data set D, deleting words with the occurrence frequency less than X1, recording the filtered words, and constructing a word list.

After the data set D is preprocessed, the data set D is divided into a training set, a verification set and a test set according to a certain proportion.

3. The method of claim 1, wherein the hierarchical multi-label text classification based on label attention comprises: the specific implementation of the step S2 includes;

for data in dataset D, assume parent node v _i And child node v _j There is a hierarchical path e between _i，j Then the characteristic f (e) of the edge formed by the path of the parent and child nodes _i，j ) From a priori probability p (U) _j |U _i ) And p (U) _i |U _j ) Represents:

f(e _i，j ) The relation of two nodes is represented, the relation is described by the transition probability or the co-occurrence probability of the two nodes, and the transition probability of the two nodes respectively comprises the transition probability p (U) from a parent node to a child node _j |U _i ) Transition probability p (U) of child node to parent node _i |U _j ) In the formula, U _j And U _i Respectively representing text data marked v _j Node label and is marked as v _i Node label, p (U) _j |U _i ) Is given by v _i In the case marked v _j Conditional probability of node label, P (U) _j ∩U _i ) Is { v _j ，v _i Probability of being marked simultaneously, N _j And N _i Respectively representing a training set v _j Node labels and v _i The number of node labels.

4. The method of claim 3, wherein the hierarchical multi-label text classification based on label attention comprises: in step S3, word embedding processing is performed on the input text and the tag thereof by the word embedding module, and the word embedding processing method specifically includes:

step 2.1, obtaining the preprocessed text sequence, and embedding words { x ] in the text into a dictionary table through the query words ₁ ，x ₂ ，...，x _n Convert to word vector representation w ₁ ，w ₂ ，...，w _n }；

Step 2.2, obtaining a label set { l ] of the hierarchical multi-label text classification ₁ ，l ₂ ，...，l _n Converting the label set into a label embedding set with dimension d { c } by means of kaiming coding ₁ ，c ₂ ，...，c _n N refers to the number of words of the preprocessed text;

5. the method of claim 4, wherein the hierarchical multi-label text classification based on label attention comprises: in step S3Representing the word vector by the text encoding module w ₁ ，w ₂ ，...，w _n Performing coding processing, wherein the coding processing method specifically comprises the following steps:

word vector representation of text using Bi-GRU network w ₁ ，w ₂ ，...，w _n Encode, generating an implicit representation h with context semantic information ₁ ，h ₂ ，...，h _n Will then implicitly represent h ₁ ，h ₂ ，...，h _n Sending the data into three convolutions with different convolution kernel sizes, obtaining semantic vectors under three different receptive fields, and finally splicing the 3 semantic vectors into a new semantic expression vector S ═ S ₁ ，s ₂ ，...，s _n }。

In step S3, label vector is represented by label encoding module { c ₁ ，c ₂ ，...，c _n Performing encoding processing, wherein the label encoding processing method specifically comprises the following steps:

using single-layer GCN to represent the tag vector c ₁ ，c ₂ ，...，c _n Encoding to generate an implicit expression M ═ M with label hierarchy association information ₁ ，m ₂ ，...，m _n The implementation process is as follows:

hierarchical GCN aggregates data flows from top to bottom, bottom to top and within self-looping edges, in hierarchical GCN each directed edge represents a pair of label related features, these data flows are node transformed using edgewise linear transformation;

to implement the node transformation, a weighted adjacency matrix is used to represent the linear transformation, and the initial value of the weighted adjacency matrix comes from the prior hierarchical information of the hierarchical classification system in step S2, and the hierarchical GCN formally encodes its hidden state according to the relevant neighborhood of node k, where the neighborhood n (k) { n } n _k ，child(k)，parent(k)}，n _k Referring to the kth label node in the hierarchical label tree, child (k) refers to the child label node of the kth node, parent (k) refers to the parent label node of the kth node, and the hidden state of the node k is calculated as follows:

in the above formula, v _j ，v _k Is a parameter that can be trained on,

and

is a trainable bias parameter; for u _k，j And g _k，j In other words, u may be _k，j Understood as information between nodes k, j, g _k，j Understood as a gated value, control u _k，j Finally, the influence on the node k is large; sigma means that the activation function in deep learning can be taken as sigmoid function,

b _l ∈R ^N×dim and b is _g ∈R ^N Dim is the dimension of the vector and belongs to a predefined hyperparameter; d (j, k) represents the hierarchical direction from node j to node k, including top-down, bottom-up, and self-circulating edges; wherein, a _k，j E.r denotes the hierarchical probability f _d(k，j) (e _kj )，f _d(k，j) (e _kj ) Refers to the transition probability from the kth node to the jth tag node, which is determined by f (e) _i，j ) Is obtained by self-circulation side using a _k，k 1, used from top to bottom

Using from bottom to top f _p (e _j，k ) 1 is ═ 1; the feature matrix F of the above edge is { a ═ a _0，0 ，a _0，1 ，...，a _c-1，c-1 The expression is the weighted adjacency matrix of the text label directed hierarchical graph, and finally, the output hidden state h of the node k _k Indicating that it corresponds to a tag representation of the hierarchy information.

6. The method of claim 5, wherein the hierarchical multi-label text classification based on label attention comprises: the extraction method of the text representation module based on the label attention mechanism in the step S3 is as follows: for text representation from text coding layer

And label representation from a label coding layer

d _c The dimensional size of the text encoding vector is represented, and is a predetermined fixed value, and the text representation based on the attention of the label is calculated by the following formula:

wherein alpha is _kj Represents the amount of information, v, of the jth text feature vector to the kth tag _k I.e. a text representation based on the tag attention.

7. The method of claim 6, wherein the hierarchical multi-label text classification based on label attention comprises: the extraction method based on the self-attention mechanism text representation module in step S3 specifically includes: hidden layer text representation of Bi-GRU output from text encoding layer

A textual representation based on the self-attention mechanism is calculated by the following formula:

wherein w ₁ ，w ₂ H is a text representation, α, as a parameter _kt In a text representationWeight, u, occupied by the t-th vector _k Is a textual representation based on the self-attention mechanism.

8. The method of claim 7, wherein the hierarchical multi-label text classification based on label attention comprises: the feature fusion module in step S3 is: the text features based on the label attention mechanism and the text features based on the self-attention mechanism are subjected to self-adaptive fusion to obtain final text features d _ik-fusion The calculation method is as follows:

wherein w ₁ ，w ₂ Is a parameter, v _k For text representation based on tag attention, u _k For self-attention based text representation, β _k Is v is _k The occupied weight.

9. The method of claim 8, wherein the hierarchical multi-label text classification based on label attention comprises: the relationship network module in step S3 further mines the association information between the tags: the mining method is to fuse the text features d generated by the feature fusion module _ik-fusion Inputting the data into a full connection layer to obtain a logits vector O ═ O corresponding to each label ₁ ，o ₂ ，...，o _n And inputting the vector O into a relational network module to obtain a prediction vector y ═ y ₁ ，y ₂ ，...，y _n And finally, inputting the prediction vector y into a multilayer perceptron to obtain the label prediction probability, wherein the nature of the relational network is a residual network.

10. The label attention based hierarchical multi-label text classification method according to claim 1, characterized in that: in the training process in the step S4, a cross entropy loss function is used, and an Adam optimizer is used for training, and the cross entropy loss function of multi-label text classification is as follows:

wherein, y _ij The actual probability of the ith sample to the jth label,

and finally obtaining a trained deep learning multi-label text classification model for the prediction probability of the ith sample to the jth label, wherein L refers to the number of label categories, and N refers to the number of text samples.

Images