CN107577785A - A Hierarchical Multi-Label Classification Approach for Legal Identification - Google Patents

Abstract

The invention discloses a kind of level multi-tag sorting technique suitable for law identification, comprise the following steps：Step 1, case facts and its legal provision are extracted from the judgement document by pretreatment；Step 2, the hierarchical structure based on Label space, legal provision corresponding to extension case facts, the class label for making case sample are a subset of Label space；Step 3, case facts text is segmented and part-of-speech tagging, feature selecting is carried out to word segmentation result, choose the Feature Words construction feature vector that can fully represent case facts；Step 4, forecast model is built：Find out the k neighbour sample set N (x) for having no example x in multi-tag training set is extended, to each neighbour's sample, weight is set, the classified weight of each classification is calculated according to k neighbour's sample and has no that example belongs to the confidence level of each classification, finally prediction has no the class label set of example.

Description

A Hierarchical Multi-Label Classification Approach for Legal Identification

technical field

The invention belongs to the field of computer data analysis and mining, and relates to a hierarchical multi-label classification method suitable for legal identification.

Background technique

Hierarchical multi-label classification is a special case of multi-label classification. Different from general multi-label classification, in hierarchical multi-label classification problems, each sample can have multiple class labels, and the sample label space is organized in a tree or directed acyclic graph hierarchy. In a directed acyclic graph, a node may have multiple parent nodes, which is more complex than the tree structure, and the design of the algorithm is more difficult. Therefore, the current research on hierarchical multi-label classification mainly focuses on the tree-shaped category label structure. . Hierarchical multi-label classification algorithms can be divided into local algorithms and global algorithms according to the different ways in which the algorithms examine the category hierarchy.

The local algorithm examines the local classification information of each internal node in the category hierarchy one by one, and transforms the hierarchical multi-label classification problem into multiple multi-label classification problems. Moreover, when training a multi-label classifier on an internal node, it is necessary to select an appropriate local sample set. In the forecasting stage, top-down forecasting methods are used to make the forecasting results meet the hierarchical requirements. Literature ESULI A, FAGNI T, SEBASTIANI F.TreeBoost.MH: A boosting algorithm formulti-labelhierarchical text categorization[C]//String Processing andInformationRetrieval.2006:13–24. The TreeBoost.MH algorithm is proposed to deal with hierarchical multi-label text classification question. The algorithm recursively trains a multi-label classifier on each non-leaf node in the category label tree. The base classifier selects AdaBoost.MH. During the training process of each multi-label classifier, feature selection and training sample selection are locally conduct. Experimental results prove that TreeBoost.MH algorithm is better than AdaBoost.MH algorithm in terms of time efficiency and prediction performance. Literature CERRI R, BARROS R C, DE CARVALHO AC. Hierarchical multi-label classification using local neural networks [J]. Journal of Computer and System Sciences, 2014, 80(1): 39–56. Proposed local hierarchical multi-label classification based on multi-layer perceptron The label classification algorithm trains a multi-layer perceptron network at each layer of the category hierarchy. Each neural network is associated with a category hierarchy and is used to predict the category label at this level. The prediction result of the neural network on a certain layer will be used as Input to the next layer of neural network. Since each layer of the neural network is trained on the same sample set, the prediction results may not meet the hierarchical constraints, and subsequent processing of the prediction results is required to ensure that they meet the hierarchical constraints.

The disadvantage of the local algorithm is that on the one hand, it needs to train multiple classifiers, which makes the model more complex and affects the intelligibility of the model; on the other hand, there will be blocking problems in the prediction process, that is, samples that are misclassified in the upper layer cannot reach the lower layer Although some people have proposed three strategies of lowering the threshold, limiting voting and expanding the threshold multiplication to deal with the blocking problem of the local algorithm, the local algorithm is often not ideal in terms of prediction accuracy.

Global algorithms consider the hierarchy of categories holistically and train a single hierarchical multi-label classifier to make predictions on unseen instances. The global algorithm can be mainly divided into the following types according to the way it deals with the category label hierarchy: a global algorithm uses category clustering, first calculates the similarity between the test sample and each category, and then classifies the test sample to the nearest category . Another method is to convert the hierarchical multi-label classification problem into a multi-label classification problem for processing: Document KIRITCHENKO S, MATWIN S, FAMILI F. Functional annotation of genes using hierarchical text categorization[J], 2005. Extend the category labels of training samples , increase its ancestor category label, and convert the hierarchical multi-label classification problem into a multi-label classification problem for processing. In the test phase, because the multi-label classification algorithm AdaBoost.MH adopted does not consider the hierarchical structure of categories, it faces the same problem as the local algorithm, that is, the prediction results will have hierarchical inconsistencies, and the output of the model needs to be corrected to ensure Hierarchy constraints are met. Another global algorithm is to transform the existing non-hierarchical classification algorithm so that it can directly process hierarchical information and use hierarchical information to improve performance. Literature VENS C, STRUYF J, SCHIETGAT L, et al. Decision trees for hierarchical multilabel classification [J]. Machine Learning, 2008, 73(2): 185–214. Based on the predictive clustering tree (PCT), the Clus-HMC algorithm was proposed. A decision tree is trained to handle hierarchical multi-label classification problems and compared with the Clus-HSC and Clus-SC methods, Clus-SC ignores the hierarchy of class labels and trains a separate classifier for each class label, Clus - The HSC method is a hierarchical Clus-SC, and the prediction results meet the hierarchical constraints. Experimental results show that the global Clus-HMC algorithm is not only better than the Clus-SC and Clus-HSC algorithms in prediction performance, but also better in time efficiency.

In general, the global algorithm has two characteristics: it considers the hierarchical structure of the category as a whole at one time; it does not have the unique modularity of the local algorithm. The key difference between the global algorithm and the local algorithm is the training process. In the test phase, the global algorithm can even use a top-down method to predict the category of unseen instances like the local algorithm.

Since the category labels are organized in a hierarchical structure in hierarchical multi-label classification problems, if a sample has a category label _ci , the sample also implicitly has all the ancestor category labels of _ci ; on the other hand, when predicting unseen instances When the category of , it must also satisfy the hierarchical restriction, that is, it cannot appear that no instance belongs to a certain category but does not belong to the ancestor category of this category. The general hierarchical multi-label classification algorithm often cannot guarantee that its prediction results meet the hierarchical constraints, or cannot achieve the optimal learning effect because it does not utilize the hierarchical structure features of the label space. Therefore, the hierarchical multi-label classification algorithm should not only make full use of the association and hierarchical structure between category labels to improve the prediction performance of the classification model, but also make the prediction results meet the hierarchical constraints.

The problem of automatic identification of the applicable laws of a case is essentially a hierarchical multi-label classification problem. The category labels of the samples, that is, the legal provisions applicable to the case, are organized in a tree structure. A case may apply to multiple legal provisions, and the applicable legal provisions of the case are specific. The extent may vary. The corresponding hierarchical multi-label classification algorithm used to solve the problem of automatic identification of the applicable law of the case needs to be able to deal with the tree-shaped category hierarchy, and it is a non-mandatory leaf node prediction algorithm, and the predicted category label can correspond to any node in the category hierarchy .

Contents of the invention

Purpose of the invention: The technical problem to be solved by the present invention is to provide an effective hierarchical multi-label classification method suitable for legal identification in view of the deficiencies of the prior art.

Technical solution: the present invention discloses a hierarchical multi-label classification method suitable for legal identification, including the following steps:

Step 1: Crawl the required original text data set of referee documents from the Internet using jsoup-based crawler technology. One referee document corresponds to one sample, and it is randomly divided into a training set and a test set at a ratio of 7:3. Then pre-process the judgment documents: extract the case facts and applicable legal provisions according to the text structure of the judgment documents. The case facts are used to generate the feature vectors of the case samples, and the applicable legal provisions are used to represent the category labels of the case samples. The original text data set is converted into a semi-structured multi-label training set and test set. The semi-structured sample form is: (case fact description, legal text); correct the errors and inconsistent formats in the applicable legal provisions of the case; Utilize the language technology platform LTP of Harbin Institute of Technology as a language processing tool (LTP is a set of Chinese language processing system, formulate an XML-based language processing result representation, and provide a set of bottom-up rich and efficient Chinese language The processing module (including six Chinese processing core technologies such as lexical, syntactic, and semantic), as well as application programming interfaces and visualization tools based on dynamic link libraries (DLL), and can be used in the form of network services) performs word segmentation for case fact descriptions and part-of-speech tagging.

Step 2. Since the organization of legal provisions in the legal system is in a tree structure, correspondingly, the label space formed by the category labels in the multi-label training set is in a tree structure. Based on the hierarchical structure of the label space label space formed by the category labels in the multi-label training set, expand the legal provisions corresponding to the case facts of all samples, so that the category labels corresponding to each case fact are a subset of the label space and meet the hierarchical constraints;

Step 3, perform feature selection on the word segmentation results from the training set in step 1 (referring to the word segmentation results of the case fact part of the semi-structured multi-label training set described in step 1), and select feature words that can fully represent the case facts Construct feature vectors; through text representation, a structured extended multi-label training set Tr and test set Te are obtained;

Step 4, build a prediction model: find out the k-nearest neighbor sample set N(x) of the unseen instance x from the extended multi-label test set Te in the extended multi-label training set Tr, the unseen instance is the fact of the case to be classified, given Set weights for each neighbor sample, calculate the confidence that the unseen instance belongs to each category in the label space according to the classification weights of k neighbor samples for each category in the label space, and predict the category label set h(x) of the unseen instance, and h (x) Satisfy the hierarchical constraints. Finally, according to the tree structure of the label space, the hierarchical restrictions in the predicted category label set h(x) are removed (that is, the inverse process of label expansion), and the specific applicable legal provisions of the unseen examples are obtained. .

Step 2 includes:

Step 2-1, in the hierarchical multi-label classification problem, given a d-dimensional instance space ( is a set of real numbers), and the label space Y={y ₁ ,y ₂ ,…,y _q } containing q categories, y _i represents the i-th category, then the category label space hierarchy can be used as a binary group (Y, <) means that if there is y _i , y _j ∈ Y and y _i < y _j , then category y _i belongs to category y _j , y _i is the descendant category of y _j , y _j is the ancestor category of y _i , < means category The partial order relationship of the label, the partial order relationship < can be understood as the "belongs to" relationship, that is, if there is y _i , y _j ∈ Y and y _i < y _j , then category y _i belongs to category y _j , and y _i belongs to y _j Descendant category, y _j is the ancestor category of y _i . Partial order < is asymmetric, non-reflexive and transitive, and can be described by the following four characteristics:

a) The only root node in the category label hierarchy is represented by a virtual category label R, and for any y _i ∈ Y, y _i <R;

b) For any y _i , y _j ∈ Y, if y _i < y _j , then

c) For any y _i ∈ Y, we have

d) Any y _i , y _j , y _k ∈ Y, y _i <y _j and y _j <y _k , then y _i <y _k .

Any multi-label classification problem whose organizational structure of category labels satisfies the above four characteristics can be considered as a hierarchical multi-label classification problem. From the above formal definition, in the hierarchical category label space, all other category nodes (except the start node) on the unique path formed from any category node traced back to the root node are the category nodes. Ancestor class node. Therefore, if a sample has a category label y _i , the sample also implicitly has all the ancestor category labels of y _i , which requires the classifier to satisfy the hierarchical constraints on the predicted category set h(x) of unseen instances, that is, And y′<y″: y″∈h(x). Where y' is the category in h(x), and y" is an ancestor category of y';

Step 2-2, for any training sample ( _xi , h _i ) (1≤i≤m), m is the number of samples of all judgment documents obtained, and x _i ∈ X is a d-dimensional feature vector, which is used to represent the case factual part, is a set of category labels corresponding to _xi , that is, the legal provisions corresponding to _xi , so that the expanded set of category labels is Then h _i ′ contains all category labels in _hi and all their ancestor category labels. formally,

The label expansion process explicitly expresses the hierarchical relationship of category labels in the category labels of samples: if a sample is labeled as certain categories, then after label expansion, the ancestor categories of these categories will also be explicitly assigned to the sample; therefore, every The category label of each sample can be regarded as a subtree of the label space tree, and the top level of each subtree is the root node. It can be seen that if there is y _i , y _j ∈ Y and y _i < y _j , among the k-nearest neighbor samples in the expanded multi-label training set, the number of samples with class label y _i must not be less than that with The number of samples for class label _yj . Label expansion is an important step to ensure that the prediction results of this learning algorithm meet the hierarchical constraints.

Step 3 includes the following steps:

In step 3-1, the purpose of feature selection is to reduce the dimensionality of features. Since the general text feature selection algorithm cannot directly process multi-label data sets, it is necessary to convert multi-label data into single-label data for processing. The conversion method is: for each multi-label sample (x, h), use |h| to represent the number of label categories in the label category set h, and replace it with |h| new single-label samples (x, y _i )(1≤i≤|h|,y _i ∈h), the class y _i of each new sample is a category label in the original multi-label sample category label set h. Table 1 shows that according to the above strategy, An example of converting multi-label samples to single-label samples.

Table 1 Multi-label sample conversion process

Step 3-2, after the conversion process of step 3-1, the multi-label case samples are converted into multiple single-label case samples, and the general feature selection algorithm can be used to perform feature selection on the word segmentation results obtained in the original training set in step 1 , select a certain number (usually depending on the original text data set, for example, when using the information gain algorithm for feature selection, the total amount of information gain of the selected feature words should be as large as possible and the number of feature words should not be too large, generally at least Take 100 feature words) with distinguishing ability to form a feature space, and use the feature words from the feature space to represent the case fact part of each case sample. Among them, the attribute value corresponding to each feature word, that is, the feature weight, is calculated using the commonly used TF-IDF algorithm. Considering the case fact part of each case sample as a word-segmented document, the case fact part of all case samples forms a document set. The feature weight tf-idf _ij of the j-th dimension feature in the i-th document in the document collection is defined as follows:

Among them, tf _ij represents the frequency of the feature word t _j in the document d _i , idf _j represents the inverse document frequency of the feature word t _j in the document collection, N represents the total number of documents in the document collection, and n _j represents the feature word t _j The document frequency in the document collection, that is, the number of documents in which the characteristic word t _j appears in the document collection, and the denominator is the normalization factor.

Step 3-3, perform feature selection on the word segmentation results obtained in the original training set in step 1, and select about 100 feature words with the most distinguishing ability to form a feature vector. Commonly used text feature selection methods are mainly based on document frequency (DF), mutual information (MI), information gain (IG), chi-square statistics (χ ² Statistics, CHI) and other measurement indicators. The feature selection based on document frequency is too simple, and it is often impossible to select the feature words with the most classification information. The disadvantage of mutual information is that it is easily affected by the marginal probability of feature words. Therefore, the multi-label classification method of this level chooses information gain or chi-square statistical algorithm. feature selection.

Step 3-3 includes: using the information gain algorithm for feature selection: the definition of the information gain IG(t) of the feature word t is as follows:

Among them, P _r (y _i ) represents the probability of class y _i appearing, P _r (t) represents the probability of feature t appearing, P _r (y _i |t) represents the probability of class y _i appearing under the premise of feature t , Indicates the probability that feature t does not appear, Indicates the probability of category y _i appearing under the premise that feature t does not appear. For each feature word in the document collection, its information gain is calculated, and the information gain value is lower than the set threshold (for example, 0.15, when setting the threshold, the total amount of information gain of the selected feature words should be as large as possible and the feature words The number of feature words is not included in the feature space.

Step 3-3 can also use the chi-square statistical algorithm for feature selection: first assume that the feature word is not related to the category, if the test value calculated using the CHI distribution deviates from the threshold, then it is more confident to reject the null hypothesis and accept the original hypothesis. Hypothetical alternative hypothesis: That is, the feature word has a high correlation with the category.

Let A be the number of documents that contain feature word t and belong to category y, B be the number of documents that contain feature word t but not belong to category y, C be the number of documents that do not contain feature word t but belong to category y, and D be the number of documents that do not contain feature word t The number of documents with word t that does not belong to category y, N is the total number of documents, then the chi-square statistic χ ² (t,y) of feature word t and category y is defined as:

When the feature word t and category y are independent, the chi-square statistic is 0. For a feature word, calculate its chi-square statistic for each category, and then calculate the mean value χ ² _avg (t) and maximum value χ ² _max ( t), using these two methods for comprehensive consideration, select about 100 feature words with the most distinguishing ability:

χ ² _avg (t)=∑ _i=1 P _r (y _i )χ ² (t,y _i ),

χ ² _max (t)=max _{i=1, . . . , q} χ ² (t, y _i ).

P _r (y _i ) represents the probability of class y _i appearing. The main advantage of the chi-square statistical feature selection algorithm over mutual information is that it is a normalized value, so it can better measure different feature words in the same category.

In step 4, when finding k-nearest neighbors, the distance d(x, _xi ) between the unseen instance x and the sample ( _xi , _hi ) is measured by the reciprocal of the cosine similarity of their feature vectors. The cosine similarity cos(γ,λ) of the feature vector γ of the unseen instance and the feature vector λ of the neighbor sample is calculated as follows:

Among them, s represents the subscript of the vector component, that is, the position of the component in the vector, S represents the dimension of the vector, γ _s represents the s-th component of the vector γ, and λ _s represents the s-th component of the vector λ.

In step 4, use d(x, _xi ) to represent the distance between the instance x and the sample ( _xi , _hi ), and use the full label distance weight method or the entropy label distance weight method to calculate the sample (( _xi , _hi )∈ N(x)) for classification weight w _ij of category y _j in h _i :

Calculate w _ij using the full-label distance weight method:

Entropy label distance weight method to calculate w _ij :

The calculation formula of the confidence c(x,y _j ) that the instance belongs to the category y _j is as follows:

where r represents the r-th category, and _{w ir} represents the classification weight of the r-th category y _r of hi;

The category label set h(x) of the predicted unseen instance x is:

Choose 0.5 as the decision threshold, when the confidence of each category that the unseen instance belongs to is less than the decision threshold, return the category with the highest confidence as the category to which the unseen instance belongs.

As a hierarchical multi-label classification method, its prediction results need to meet the hierarchical constraints, that is, And y′<y″: y″∈h(x). The proof is given below: from the confidence calculation formula, if the algorithm predicts that no instance x has the category label y _a (y _a ∈ Y), then the confidence c(x, y _a ) that x belongs to the category y _a is greater than the threshold t , or the maximum value across all categories. Consider the ancestor category y _b of category y _a (y _b ∈ Y, y _a < y _b ), if y _b corresponds to the virtual root node in the category hierarchy, then x has the category label y _a obviously meets the hierarchical constraints; otherwise, For any neighbor sample (x _i , Y _i )∈N(x) of x, if y _a ∈ Y _i , then there is also y _b ∈ Y _i , and vice versa. The label expansion process of the training set guarantees the above The conclusion holds. Therefore, using the full label distance weight method and the entropy label distance weight method, it can be deduced that:

on the denominator remains unchanged, so the confidence c(x,y _b ) of x belonging to category y _b is not less than the confidence c(x,y _a ) of x belonging to category y _a , if c(x,y _a )>t, There must also be c(x,y _b )>t, so the prediction result satisfies the level restriction.

Finally, the hierarchical evaluation indicators used in the performance evaluation indicators of this learning method: hierarchical precision (hP), hierarchical recall (hR) and hierarchical F-measure (hF), which are defined as follows:

in, is the set of the category to which the test sample i belongs and its ancestor categories, is the set of the category that the test sample i actually belongs to and its ancestor categories, and the summation operation is to calculate the value on all test samples.

In order to make the identification of the applicable law of the case more practical, the target category predicted by the algorithm is preferably a specific legal clause, not just a broad law. Therefore, this method considers that the target category is all legal clauses and specific legal clauses. predictive performance. In the following, hP_all, hR_all, and hF_all are used to represent the hierarchical precision, recall rate, and F-measure value of the system when the target category is all legal provisions, and hP_partial, hR_partial, and hF_partial are used to represent the hierarchical precision of the algorithm when the target category is specific legal provisions , recall and F-measure.

In addition to the hierarchical evaluation index, the precision, recall rate and F measure value of each category can be calculated separately, and the mean value of the precision, recall rate and F measure value of all categories can be used as the evaluation index of system performance, that is, precision, recall rate Macro-averaging of F-measure and F-measure. For each category, let TP represent the number of true cases, FP represent the number of false positive cases, TN represent the number of true negative cases, and FN represent the number of false negative cases, then the macro of precision, recall and F value The formulas for calculating the average Macro-P, Macro-R, and Macro-F are as follows:

The present invention is a global hierarchical multi-label classification method, which considers the hierarchical structure of category labels as a whole, and ensures that the prediction results also meet the hierarchical restrictions. This learning method is an inert learning algorithm. It does not need to construct a clear prediction model on the training set, and only stores the original multi-label samples after label expansion, thus supporting incremental learning; in the prediction stage, first find the unseen The k neighbor samples of the instance in the training set, according to the classification weights of these neighbor samples for each category, determine the confidence that the instance belongs to each category, and then predict the category to which the unseen instance belongs. The model of this learning method is simple, supports incremental learning, and can be well applied to hierarchical multi-label classification problems such as automatic identification of applicable laws of cases, which contain massive amounts of data and the data is constantly growing.

Beneficial effects: the present invention provides a hierarchical multi-label classification method suitable for legal identification, which fully considers the tree-like hierarchical structure of the label space of legal provisions as a whole, so that the prediction results meet the hierarchical restrictions, and no additional processing is required for the prediction results. fix. At the same time, the model of this method is simple, supports incremental learning, and can be well applied to hierarchical multi-label classification problems such as automatic identification of applicable laws in cases, which contain massive amounts of data and the data is constantly growing.

Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the advantages of the above and other aspects of the present invention will become clearer.

Fig. 1 main flow chart of the present invention.

Figure 2 Sample referee documents.

Figure 3. The tree structure of the label space of legal provisions.

Fig. 4 Frequency distribution of combinations of legal provisions.

Figure 5. Performance comparison of hierarchical indicators under different numbers of neighbors.

Figure 6. Performance comparison of macro-averaged indicators with different numbers of neighbors.

Figure 7. Performance comparison of each index under different weighting strategies.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

The invention discloses a hierarchical multi-label classification method suitable for legal identification, which includes the following steps:

Step 1, use jsoup-based crawler technology to crawl the required original text data set of referee documents from the Internet, and divide it randomly into training set and test set at a ratio of 7:3. Then pre-process the referee documents, mainly completing the following tasks:

According to the text structure of the judgment document, the case facts and the applicable legal provisions are extracted. The former is used to generate the feature vector of the case sample, and the latter is used to represent the category label of the case sample. The original text data set is converted into a semi-structured multi- label training and test sets;

Correct errors and format inconsistencies in applicable legal provisions of the case;

Use Harbin Institute of Technology's language technology platform LTP to perform word segmentation and part-of-speech tagging for the description of the facts of the case.

Step 2. Since the organization of legal provisions in the legal system is in a tree structure, correspondingly, the label space formed by the category labels in the multi-label training set is in a tree structure. Based on the hierarchical structure of the label space, the legal provisions corresponding to the case facts of all samples are expanded, so that the category label set corresponding to each case fact is a subset of the label space and meets the hierarchical constraints;

Step 3, perform feature selection on the word segmentation results obtained in the original training set in step 1, and select feature words that can fully represent the facts of the case to construct a feature vector; after text representation, a structured extended multi-label training set Tr and test set Te are obtained;

Step 4, build a prediction model: Find the k-nearest neighbor sample set N(x) of the unseen instance x from the extended multi-label test set Te in the extended multi-label training set Tr, set weights for each neighbor sample, according to k The classification weights of neighboring samples for each category in the label space calculate the confidence that the unseen instance belongs to each category in the label space, and predict the category label set h(x) of the unseen instance, and h(x) satisfies the hierarchical constraints. Finally, according to the tree structure of the label space, the hierarchical restrictions in the predicted category set h(x) are removed (ie, the inverse process of label expansion), and the specific applicable legal provisions of the unseen examples are obtained.

Step 2 includes:

Step 2-1, in the hierarchical multi-label classification problem, given a d-dimensional instance space And the label space Y={y ₁ ,y ₂ ,…,y _q } containing q categories, y _i represents the i-th category, then the category label space hierarchy can be represented by a two-tuple (Y,<), < Represents the partial order relationship of category labels. The partial order relationship < can be understood as the "belongs to" relationship, that is, if there is y _i , y _j ∈ Y and y _i < y _j , then category y _i belongs to category y _j , and y _i is y The descendant category of _j , y _j is the ancestor category of y _i . Partial order < is asymmetric, non-reflexive and transitive, and can be described by the following four characteristics:

e) The only root node in the category label hierarchy is represented by a virtual category label R, and for any y _i ∈ Y, y _i <R;

f) For any y _i , y _j ∈ Y, if y _i < y _j , then

g) For any y _i ∈ Y, we have

h) Any y _i , y _j , y _k ∈ Y, y _i <y _j and y _j <y _k , then y _i <y _k .

Any multi-label classification problem whose organizational structure of category labels satisfies the above four characteristics can be considered as a hierarchical multi-label classification problem. From the above formal definition, in the hierarchical category label space, all other category nodes (except the start node) on the unique path formed from any category node traced back to the root node are the category nodes. Ancestor class node. Therefore, if the sample has the category label _ci , the sample also implicitly has all the ancestor category labels of _ci , which requires the classifier to satisfy the hierarchical limit for the predicted category set h(x) of unseen instances, that is, And y′<y″: y″∈h(x).

Step 2-2, for any training sample (x _i , y _i ) (1≤i≤m), m is the number of samples of all referee documents obtained, x _i ∈ X is a d-dimensional feature vector, is a set of category labels corresponding to _xi . Let the expanded category label set be y _i ′, then y _i ′ contains all the category labels in y _i and all their ancestor category labels. formally,

The label expansion process explicitly expresses the hierarchical relationship of category labels in the category labels of samples: if a sample is labeled as certain categories, then after label expansion, the ancestor categories of these categories will also be explicitly assigned to the sample; therefore, every The category label of each sample can be regarded as a subtree of the label space tree, and the top level of each subtree is the root node. It can be seen that if there is y _i , y _j ∈ Y and y _i < y _j , among the k-nearest neighbor samples in the expanded multi-label training set, the number of samples with class label y _i must not be less than that with The number of samples for class label _yj . Label expansion is an important step to ensure that the prediction results of this learning algorithm meet the hierarchical constraints.

Step 3 includes the following steps:

In step 3-1, the purpose of feature selection is to reduce the dimensionality of features. Since the general text feature selection algorithm cannot directly process multi-label data sets, it is necessary to convert multi-label data into single-label data for processing. The conversion method is: for each multi-label sample (x, h), use |h| to represent the number of label categories in the label category set h, and replace it with |h| new single-label samples (x, y _i )(1≤i≤|y|,y _i ∈h), the class y _i of each new sample is a category label in the original multi-label sample category label set h. Table 1 shows that according to the above strategy, An example of converting multi-label samples to single-label samples.

Table 1 Multi-label sample conversion process

Step 3-2, after the conversion process of step 3-1, the multi-label case samples are converted into single-label case samples, and the general feature selection algorithm can be used to perform feature selection on the word segmentation results obtained in the original training set in step 1, and select About 100 most discriminative feature words constitute the feature space. The feature words from the feature space are used to represent the case facts of each case sample, and the attribute value corresponding to each feature word, that is, the feature weight, is calculated using the commonly used TF-IDF algorithm. Considering the case fact part of each sample as a word-segmented document, the case fact part of all samples forms a document set. The feature weight tf-idf _ij of the j-th dimension feature in the i-th document is defined as follows:

Among them, tf _ij represents the frequency of the feature word t _j in the document d _i , idf _j represents the inverse document frequency of the feature word t _j in the document collection, N represents the total number of documents in the document collection, and n _j represents the feature word t _j The document frequency in the document collection, that is, the number of documents in which the characteristic word t _j appears in the document collection, and the denominator is the normalization factor.

Step 3-3, perform feature selection on the word segmentation results obtained in the original training set in step 1, and select a certain number of feature words with distinguishing ability to form a feature vector. Commonly used text feature selection methods are mainly based on document frequency (DF), mutual information (MI), information gain (IG), chi-square statistics (χ ² Statistics, CHI) and other measurement indicators. The feature selection based on document frequency is too simple, and it is often impossible to select the feature words with the most classification information. The disadvantage of mutual information is that it is easily affected by the marginal probability of feature words. Therefore, the multi-label classification method of this level chooses information gain or chi-square statistical algorithm. feature selection.

Step 3-3 includes: using the information gain algorithm for feature selection: the definition of the information gain IG(t) of the feature word t is as follows:

Among them, P _r (y _i ) represents the probability of class y _i appearing, P _r (t) represents the probability of feature t appearing, P _r (y _i |t) represents the probability of class y _i appearing under the premise of feature t , Indicates the probability that feature t does not appear, Indicates the probability of category y _i appearing under the premise that feature t does not appear. For each feature word in the document collection, its information gain is calculated, and the feature words whose information gain value is lower than the set threshold are not included in the feature space.

Step 3-3 can also use the chi-square statistical algorithm to select the features of the case fact texts in the training set: first assume that the feature words are not related to the category, if the test value calculated by using the CHI distribution deviates from the threshold, the more Confidence negates the null hypothesis and accepts the alternative hypothesis of the null hypothesis: that is, the feature words have a high correlation with the category.

Let A be the number of documents that contain feature word t and belong to category y, B be the number of documents that contain feature word t but not belong to category y, C be the number of documents that do not contain feature word t but belong to category y, and D be the number of documents that do not contain feature word t The number of documents with word t that does not belong to category y, N is the total number of documents, then the chi-square statistic χ ² (t,y) of feature word t and category y is defined as:

When the feature word t and category y are independent, the chi-square statistic is 0. For a feature word, calculate its chi-square statistic for each category, and then calculate the mean value χ ² _avg (t) and maximum value X ² _max ( t), use these two methods to comprehensively consider and select the most distinguishable feature words:

X ² _avg (t)=∑ _i=1 P _r (y _i )χ ² (t,y _i ),

χ ² _max (t)=max _{i=1, . . . , q} χ ² (t, y _i ).

P _r (y _i ) represents the probability of class y _i appearing. The main advantage of the chi-square statistical feature selection algorithm over mutual information is that it is a normalized value, so it can better measure different feature words in the same category.

In step 4, when finding k-nearest neighbors, the distance d(x, _xi ) between the unseen instance x and the sample ( _xi , _hi ) is measured by the reciprocal of the cosine similarity of their feature vectors. The cosine similarity cos(γ,λ) of the feature vector γ of the unseen instance and the feature vector λ of the neighbor sample is calculated as follows:

Among them, s represents the subscript of the vector component, that is, the position of the component in the vector, S represents the dimension of the vector, γ _s represents the s-th component of the vector γ, and λ _s represents the s-th component of the vector λ.

In step 4, use d(x, _xi ) to represent the distance between instance x and sample ( _xi , _hi ), and use the full-label distance weight method to calculate the sample (( _xi , _hi )∈N(x)) for Classification weight w _ij for class y _j :

Calculate w _ij using the full-label distance weight method:

Entropy label distance weight method to calculate w _ij :

The formula for calculating the confidence c(x,y _j ) of the unseen instance belonging to the category y _j is as follows:

The category label set h(x) of the predicted unseen instance x is:

Choose 0.5 as the decision threshold, when the confidence of each category that the unseen instance belongs to is less than the decision threshold, return the category with the highest confidence as the category to which the unseen instance belongs.

Example

As shown in Figure 1, the steps of the present invention are:

Step 1: Use jsoup-based crawler technology to crawl the required original text data set of referee documents from the Internet, and randomly divide it into a training set and a test set at a ratio of 7:3. Then pre-process the referee documents, mainly completing the following tasks:

According to the text structure of the judgment document, the case facts and the applicable legal provisions are extracted. The former is used to generate the feature vector of the case sample, and the latter is used to represent the category label of the case sample. The original text data set is converted into a semi-structured multi- label training and test sets;

Correct errors and format inconsistencies in applicable legal provisions of the case;

Use Harbin Institute of Technology's language technology platform LTP to perform word segmentation and part-of-speech tagging for the description of the facts of the case.

Step 2, based on the hierarchical structure of the label space, expand the legal provisions corresponding to the case facts of all samples, so that the category label corresponding to each case fact is a subset of the label space and meets the hierarchical constraints;

Step 3, perform feature selection on the word segmentation results obtained in the original training set in step 1, and select feature words that can fully represent the facts of the case to construct a feature vector; through text representation, a structured extended multi-label training set Tr and test set Te are obtained;

Step 4, build a prediction model: first find out the k-nearest neighbor sample set N(x) of the unseen instance x from the extended multi-label test set Te in the extended multi-label training set Tr, set weights for each neighbor sample, according to k Calculate the confidence of unseen instances belonging to each category in the label space with respect to the classification weights of each category in the label space for the nearest neighbor samples, and predict the category label set h(x) of the unseen instance, and h(x) satisfies the hierarchical constraints. Finally, according to the tree structure of the label space, the hierarchical restrictions in the predicted category set h(x) are removed (ie, the inverse process of label expansion), and the specific applicable legal provisions of the unseen examples are obtained.

The specific implementation data are taken from the judgment documents of the people's courts at all levels in Zhejiang Province published on the Zhejiang Court Open Network.

Figure 2 is a sample of adjudication documents, in which the underlined part is the fact of the case, and the underlined part is the legal provisions applicable to the case. According to the writing rules of judgment documents, extract the facts of the case and its legal provisions. The preprocessing work is mainly to clean up and correct the legal part of the case.

In Figure 3, the tree structure of the label space of legal provisions is shown. Based on such a hierarchical structure, label extensions are performed on the legal provisions corresponding to the facts of each case.

Figure 4 is a frequency distribution diagram of the combination of legal provisions. According to the frequency of citations of various legal provisions, 26 laws such as "Civil Procedure Law of the People's Republic of China" and "Contract Law of the People's Republic of China" with high frequency and 451 specific legal provisions contained in these laws were selected The label space is composed of category labels, that is, the dimension of the label space is 477. The category label set of each case sample is expressed in the form of a label vector, and each dimension of the vector represents a category label in the label space, that is, a complete legal provision. If a certain legal provision is applied to the case, the label entry value corresponding to this legal provision and all legal provisions containing this legal provision in the label vector is 1, otherwise it is 0. Therefore, the label vector of each sample corresponds to a combination of legal provisions, and the frequency of each combination is the number of corresponding case samples. The frequency of each combination of legal provisions can also reflect some properties of the case sample set. Figure 4 can be obtained by calculating each, and selecting combinations with higher frequency of occurrence and arranging them in descending order. It can be seen from the figure that the frequency of occurrence of combinations of legal clauses is generally in a long-tail distribution, and the frequency of occurrence of a few combinations of legal clauses is extremely high, indicating that a large number of case samples apply to this combination of legal clauses. In addition, most of the combinations of legal clauses appear The frequency is more balanced.

Step 3 Select the information gain algorithm for feature selection. By calculating the information gain of each characteristic word, it can be found that the words with higher information gain are mostly verbs or nouns. Table 2 shows the proportion of verbs and nouns in the characteristic words with the highest information gain value, which can be seen in the identification of applicable laws Middle nouns and verbs are more distinguishable than words of other natures. On the other hand, it also shows that words other than verbs and nouns in the text can be removed through part-of-speech tagging, thereby reducing the number of words in the text and simplifying subsequent calculations.

The proportion of verbs and nouns in the feature words in Table 2:

number of feature words Verb Noun Quantity Ratio Verb noun total information gain ratio 100 88.0% 87.9% 200 80.0% 82.3% 300 81.0% 82.5% 400 80.5% 82.0% 500 76.8% 79.7%

Table 3 Overview of the experimental training set and test set:

Number of samples Sample Average Number of Class Labels Training set 102608 7.6344 test set 44210 7.6397

Figure 5 and Figure 6 are the performance comparisons of the hierarchical index and the macro-averaged index when different numbers of neighbors are taken.

It can be seen from Figure 5 that when the number of neighbors is even, the precision of the algorithm is high, but the recall rate is low; when the number of neighbors is odd, the precision of the algorithm is low, but the recall rate is high. As the number of neighbors increases, this difference gradually becomes smaller. By analyzing the principle of the algorithm, this phenomenon can be explained: the decision threshold set by the algorithm is 0.5, and when the number of neighbors is even, due to the addition of smoothing parameters, only category labels with occurrences exceeding k=2 The class label of an unseen instance is predicted, while a class label with exactly k=2 occurrences is not assigned to an unseen instance. Therefore, when the number of neighbors is an even number, the conditions for each category label to assign unseen instances are more stringent, resulting in a higher prediction accuracy of the algorithm and a correspondingly lower recall rate. When the number of neighbors increases, this effect gradually weakens, so this difference becomes smaller. It can also be seen from the figure that when the target category is all legal provisions, all the predictive indicators of the algorithm are higher than when the target category is specific legal provisions. This is because broader legal categories contain a larger sample of cases, giving the model better predictive power in those categories. On the whole, when the number of neighbors k is 5, the comprehensive prediction performance of the algorithm is the best.

From Figure 6, it can be found that with the increase of the number of neighbors, the macro-average precision, recall rate and F-measure value of the algorithm are all decreasing. The reason may be that as the number of neighbors increases, it is more difficult for classes with a small number of samples to reach the decision threshold, thus leading to a decrease in the prediction performance of most classes, and finally leading to a corresponding decrease in the macro-averaged performance.

Figure 7 shows the performance of the algorithm on each evaluation index when the number of neighbors is fixed to 5, and the sample weight strategy is the full label distance weight method and the entropy label distance weight method. Taken together, whether it is a hierarchical index or a macro-average index, using the entropy label distance weight strategy can achieve better results in accuracy, while using the full label distance weight strategy can achieve better recall and F-measure values. Effect. The reason is that the entropy label weight strategy is biased towards samples with a small number of category labels, and in the extended hierarchical multi-label samples, the more specific the category of the sample, the more category labels it will have, resulting in the entropy label weight The classification weight under the strategy is small, so the prediction result of the entropy label weight strategy is more inclined to the upper category, resulting in a larger generalization error. Although the performance of the algorithm decreases when the target category is a specific legal clause, it still has a hierarchical precision of close to 80% and a hierarchical recall of more than 65%, indicating that the case based on this hierarchical multi-label classification algorithm is applicable to the law. Identification is valid.

Consider target category is two kinds of situations of whole legal clause and specific legal clause, in the present invention, represent the macro average precision, recall rate and F measure value of algorithm when target category is whole legal clause with mP_all, mP_all, mP_all respectively, use mP_partial, mP_partial, mP_all mP_partial and mP_partial represent the macro average precision, recall rate and F-measure value of the algorithm when the target category is a specific legal clause.

In this implementation, two commonly used hierarchical multi-label classification algorithms, TreeBoost.MH local algorithm and Clus-HMC global algorithm, are selected to compare with the prediction performance of this hierarchical multi-label classification algorithm. Table 5 shows their performance in each hierarchical index Table 6 shows the comparison of their prediction performance on each macro-averaged index.

Table 5. Performance comparison of hierarchical indicators of each algorithm:

Table 6 Macro average performance comparison of each algorithm:

Facts have proved that this hierarchical multi-label classification algorithm can achieve better results than existing methods in terms of predictive performance. Combined with the characteristics of the Lazy-HMC algorithm supporting incremental learning, the Lazy-HMC algorithm can be used to build an effective and applicable automatic identification system for applicable laws.

The present invention provides a hierarchical multi-label classification method suitable for legal identification. There are many methods and approaches to specifically realize the technical solution. The above description is only a preferred embodiment of the present invention. As far as people are concerned, some improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components that are not specified in this embodiment can be realized by existing technologies.

Claims (8)

1. A hierarchical multi-label classification method applicable to legal identification, characterized in that, comprising the following steps: Step 1. Obtain the original text data set of the judgment document, divide it into a training set and a test set, and perform preprocessing: extract case facts and applicable legal provisions from it according to the text structure of the judgment document, and use the case facts to generate case samples The feature vector of the applicable legal provisions is used to represent the category label of the case sample, and the original text data set is converted into a semi-structured multi-label training set and test set; word segmentation and part-of-speech labeling are performed on the description of the case facts; Step 2, based on the hierarchical structure of the label space formed by the category labels in the multi-label training set, expand the legal provisions corresponding to the case facts of all samples, so that the category labels corresponding to each case fact are a subset of the label space and satisfy the hierarchical constraints ; Step 3, perform feature selection on the word segmentation results from the training set in step 1, and select feature words that can fully represent the facts of the case to construct a feature vector; after text representation, a structured extended multi-label training set Tr and test set Te are obtained; Step 4, build a prediction model: find out the k-nearest neighbor sample set N(x) of the unseen instance x from the extended multi-label test set Te in the extended multi-label training set Tr, the unseen instance is the fact of the case to be classified, given Set the weight for each neighbor sample, calculate the confidence that the unseen instance belongs to each category according to the classification weights of k neighbor samples for each category, and predict the category label set h(x) of the unseen instance, and h(x) satisfies the hierarchical limit , and finally according to the tree structure of the label space, remove the hierarchical restrictions in the predicted category label set h(x), and obtain the specific applicable legal provisions of the unseen examples. 2. according to the method described in claim 1, it is characterized in that, in step 1, with the ratio of 7:3, the original text dataset of referee documents is randomly divided into training set and test set. 3. according to the method described in claim 2, it is characterized in that: step 2 comprises: Step 2-1, in the hierarchical multi-label classification problem, given a d-dimensional instance space is a set of real numbers, and a label space Y={y ₁ ,y ₂ ,…,y _q } containing q categories, y _i represents the i-th category, then the class label space hierarchy uses a tuple (Y,<) means, < means the partial order relationship of category labels, if there is y _i , y _j ∈ Y and y _i < y _j , then category y _i belongs to category y _j , y _i is the descendant category of y _j , and y _j is y _i The ancestor category of the classifier, the predicted category set h(x) of the unseen instance must satisfy the hierarchical restriction, that is, And y′<y″: y″∈h(x), where y′ is the category in h(x), and y″ is an ancestor category of y′; Step 2-2, for any sample (x _i , h _i ) (1≤i≤m), m is the number of samples of all judgment documents obtained, and x _i ∈ X is a d-dimensional feature vector, which is used to represent the facts of the case part, is a set of category labels corresponding to _xi , that is, the legal provisions corresponding to _xi , let the expanded set of category labels be h _i ′, Then h _i ′ contains all category labels in h _i and all their ancestor category labels: <mrow><msup><msub><mi>h</mi><mi>i</mi></msub><mo>&amp;prime;</mo></msup><mo>=</mo><msub><mi>h</mi><mi>i</mi></msub><mo>&amp;cup;</mo><msub><mo>&amp;cup;</mo><mrow><mi>y</mi><mo>&amp;Element;</mo><msub><mi>h</mi><mi>i</mi></msub></mrow></msub><mo>{</mo><msup><mi>y</mi><mo>&amp;prime;</mo></msup><mo>|</mo><mi>y</mi><mo>&lt;</mo><msup><mi>y</mi><mo>&amp;prime;</mo></msup><mo>}</mo><mo>.</mo></mrow> 4. according to the method described in claim 3, it is characterized in that: step 3 comprises the following steps: Step 3-1, convert multi-label data into single-label data for processing: For each multi-label sample (x, h), use |h| to represent the number of label categories in the label category set h, and replace it with | h| new single-label samples (x, y _i ) (1≤i≤|h|, y _i ∈ h), the class y _i of each new sample is one of the original multi-label sample category label set h category label; Step 3-2, after the conversion process of step 3-1, the multi-label case samples are converted into multiple single-label case samples, and the case fact part of each case sample is regarded as a word-segmented document, then all The case fact part of the case sample forms a document collection, and the feature weight tf-idf _ij of the j-th dimension feature in the i-th document in the document collection is defined as follows: Among them, tf _ij represents the frequency of the feature word t _j in the document d _i , idf _j represents the inverse document frequency of the feature word t _j in the document collection, N represents the total number of documents in the document collection, and n _j represents the feature word t _j The document frequency in the document collection, that is, the number of documents in which the characteristic word t _j appears in the document collection, and the denominator is the normalization factor; Step 3-3, use the general feature selection algorithm to perform feature selection on the word segmentation results obtained in the original training set in step 1, and select a certain number of feature words with distinguishing ability to form a feature space. 5. according to the method described in claim 4, it is characterized in that: step 3-3 comprises: adopt information gain algorithm to carry out feature selection: the definition of the information gain IG (t) of feature word t is as follows: <mrow><mtable><mtr><mtd><mrow><mi>I</mi><mi>G</mi><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow><mo>=</mo></mrow></mtd></mtr><mtr><mtd><mrow><mo>-</mo><msubsup><mi>&amp;Sigma;</mi><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>q</mi></msubsup><msub><mi>P</mi><mi>r</mi></msub><mrow><mo>(</mo><msub><mi>y</mi><mi>i</mi></msub><mo>)</mo></mrow><msub><mi>logP</mi><mi>r</mi></msub><mrow><mo>(</mo><msub><mi>y</mi><mi>i</mi></msub><mo>)</mo></mrow><mo>+</mo><msub><mi>P</mi><mi>r</mi></msub><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow><msubsup><mi>&amp;Sigma;</mi><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>q</mi></msubsup><msub><mi>P</mi><mi>r</mi></msub><mrow><mo>(</mo><mrow><msub><mi>y</mi><mi>i</mi></msub><mo>|</mo><mi>t</mi></mrow><mo>)</mo></mrow><msub><mi>logP</mi><mi>r</mi></msub><mrow><mo>(</mo><mrow><msub><mi>y</mi><mi>i</mi></msub><mo>|</mo><mi>t</mi></mrow><mo>)</mo></mrow><mo>+</mo><msub><mi>P</mi><mi>r</mi></msub><mrow><mo>(</mo><mover><mi>t</mi><mo>&amp;OverBar;</mo></mover><mo>)</mo></mrow><msubsup><mi>&amp;Sigma;</mi><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>q</mi></msubsup><msub><mi>P</mi><mi>r</mi></msub><mrow><mo>(</mo><mrow><msub><mi>y</mi><mi>i</mi></msub><mo>|</mo><mover><mi>t</mi><mo>&amp;OverBar;</mo></mover></mrow><mo>)</mo></mrow><msub><mi>logP</mi><mi>r</mi></msub><mrow><mo>(</mo><mrow><msub><mi>y</mi><mi>i</mi></msub><mo>|</mo><mover><mi>t</mi><mo>&amp;OverBar;</mo></mover></mrow><mo>)</mo></mrow></mrow></mtd></mtr></mtable><mo>,</mo></mrow> Among them, P _r (y _i ) represents the probability of class y _i appearing, P _r (t) represents the probability of feature t appearing, P _r (y _i |t) represents the probability of class y _i appearing under the premise of feature t , Indicates the probability that feature t does not appear, Indicates the probability of category y _i appearing on the premise that feature t does not appear. For each feature word in the document collection, its information gain is calculated. Feature words whose information gain value is lower than the set threshold are not included in the feature space. 6. according to the method described in claim 5, it is characterized in that: step 3-3 comprises: adopt chi square statistical algorithm to carry out feature selection: Let A be the number of documents that contain feature word t and belong to category y, B be the number of documents that contain feature word t but not belong to category y, C be the number of documents that do not contain feature word t but belong to category y, and D be the number of documents that do not contain feature word t The number of documents with word t that does not belong to category y, N is the total number of documents, then the chi-square statistic χ ² (t,y) of feature word t and category y is defined as: <mrow><msup><mi>&amp;chi;</mi><mn>2</mn></msup><mrow><mo>(</mo><mi>t</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow><mo>=</mo><mfrac><mrow><mi>N</mi><mo>&amp;times;</mo><msup><mrow><mo>(</mo><mi>A</mi><mi>D</mi><mo>-</mo><mi>B</mi><mi>C</mi><mo>)</mo></mrow><mn>2</mn></msup></mrow><mrow><mo>(</mo><mi>A</mi><mo>+</mo><mi>C</mi><mo>)</mo><mo>&amp;times;</mo><mo>(</mo><mi>B</mi><mo>+</mo><mi>D</mi><mo>)</mo><mo>&amp;times;</mo><mo>(</mo><mi>A</mi><mo>+</mo><mi>B</mi><mo>)</mo><mo>&amp;times;</mo><mo>(</mo><mi>C</mi><mo>+</mo><mi>D</mi><mo>)</mo></mrow></mfrac><mo>,</mo></mrow> When the feature word t and category y are independent, the chi-square statistic is 0. For a feature word, calculate its chi-square statistic for each category, and then calculate the mean value χ ² _avg (t) and maximum value χ ² _max ( t), using these two methods to comprehensively consider, select a certain number of feature words with distinguishing ability, where P _r (y _i ) represents the probability of category occurrence: χ ² _avg (t)=∑ _i=1 P _r (y _i )χ ² (t,y _i ), χ ² _max (t)=max _{i=1, . . . , q} χ ² (t, y _i ). 7. The method according to claim 5 or 6, characterized in that: in step 4, when looking for k-nearest neighbors, the distance d(x, _xi ) between instance x and sample ( _xi ,h _i ) is not seen, The reciprocal of the cosine similarity of their eigenvectors is used to measure the cosine similarity cos(γ,λ) of the eigenvector γ of the unseen instance and the eigenvector λ of the neighbor sample as follows: Among them, s represents the subscript of the vector component, that is, the position of the component in the vector, S represents the dimension of the vector, γ _s represents the s-th component of the vector γ, and λ _s represents the s-th component of the vector λ. 8. The method according to claim 7, characterized in that: in step 4, d(x, _xi ) is used to represent the distance between the unseen instance x and the neighbor sample ( _xi ,h _i ), and the full label distance is used The weight method or the entropy label distance weight method calculates the classification weight _{w ij of} the neighbor sample (( _xi ,hi)∈N(x)) for the category y _{j in h i :} Calculate w _ij using the full-label distance weight method: <mrow><msub><mi>w</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>=</mo>< mfenced open = "{" close = ""><mtable><mtr><mtd><mrow><mn>1</mn><mo>/</mo><mrow><mo>(</mo><msup><mi>d</mi><mn>2</mn></msup><mo>(</mo><mrow><mi>x</mi><mo>,</mo><msub><mi>x</mi><mi>i</mi></msub></mrow><mo>)</mo><mo>)</mo></mrow><mo>,</mo></mrow></mtd><mtd><mrow><msub><mi>y</mi><mi>j</mi></msub><mo>&amp;Element;</mo><msub><mi>h</mi><mi>i</mi></msub></mrow></mtd></mtr><mtr><mtd><mrow><mn>0</mn><mo>,</mo></mrow></mtd><mtd><mrow><msub><mi>y</mi><mi>j</mi></msub><mo>&amp;NotElement;</mo><msub><mi>h</mi><mi>i</mi></msub></mrow></mtd></mtr></mtable></mfenced><mo>,</mo></mrow> Entropy label distance weight method to calculate w _ij : <mrow><msub><mi>w</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>=</mo>< mfenced open = "{" close = ""><mtable><mtr><mtd><mrow><mn>1</mn><mo>/</mo><mrow><mo>(</mo><msup><mi>d</mi><mn>2</mn></msup><mo>(</mo><mrow><mi>x</mi><mo>,</mo><msub><mi>x</mi><mi>i</mi></msub></mrow><mo>)</mo><mo>&amp;times;</mo><mo>|</mo><msub><mi>h</mi><mi>i</mi></msub><mo>|</mo><mo>)</mo></mrow><mo>,</mo></mrow></mtd><mtd><mrow><msub><mi>y</mi><mi>j</mi></msub><mo>&amp;Element;</mo><msub><mi>h</mi><mi>i</mi></msub></mrow></mtd></mtr><mtr><mtd><mrow><mn>0</mn><mo>,</mo></mrow></mtd><mtd><mrow><msub><mi>y</mi><mi>j</mi></msub><mo>&amp;NotElement;</mo><msub><mi>h</mi><mi>i</mi></msub></mrow></mtd></mtr></mtable></mfenced><mo>,</mo></mrow> The formula for calculating the confidence c(x,y _j ) of the unseen instance belonging to the category y _j is as follows: <mrow><mi>c</mi><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><msub><mi>y</mi><mi>j</mi></msub></mrow><mo>)</mo></mrow><mo>=</mo><mfrac><mrow><msub><mi>&amp;Sigma;</mi><mrow><mrow><mo>(</mo><mrow><msub><mi>x</mi><mi>i</mi></msub><mo>,</mo><msub><mi>h</mi><mi>i</mi></msub></mrow><mo>)</mo></mrow><mo>&amp;Element;</mo><mi>N</mi><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow></mrow></msub><msub><mi>w</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow><mrow><msub><mi>&amp;Sigma;</mi><mrow><mrow><mo>(</mo><mrow><msub><mi>x</mi><mi>i</mi></msub><mo>,</mo><msub><mi>h</mi><mi>i</mi></msub></mrow><mo>)</mo></mrow><mo>&amp;Element;</mo><mi>N</mi><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow></mrow></msub><msub><mi>max</mi><mrow><mn>1</mn><mo>&amp;le;</mo><mi>r</mi><mo>&amp;le;</mo><mi>q</mi></mrow></msub><msub><mi>w</mi><mrow><mi>i</mi><mi>r</mi></mrow></msub></mrow></mfrac><mo>,</mo></mrow> where _{w ir} represents the classification weight of the rth category y _r of hi; The category label set h(x) of the predicted unseen instance x is: <mrow><mi>h</mi><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow><mo>=</mo><mo>{</mo><msub><mi>y</mi><mi>j</mi></msub><mo>|</mo><mi>c</mi><mrow><mo>(</mo><mi>x</mi><mo>,</mo><msub><mi>y</mi><mi>j</mi></msub><mo>)</mo></mrow><mo>&gt;</mo><mfrac><mn>1</mn><mn>2</mn></mfrac><mo>,</mo><mn>1</mn><mo>&amp;le;</mo><mi>j</mi><mo>&amp;le;</mo><mi>q</mi><mo>}</mo><mo>&amp;cup;</mo><mo>{</mo><msub><mi>y</mi><msup><mi>j</mi><mo>*</mo></msup></msub><mo>|</mo><msup><mi>j</mi><mo>*</mo></msup><mo>=</mo><mi>arg</mi><mi></mi><msub><mi>max</mi><mrow><mn>1</mn><mo>&amp;le;</mo><mi>j</mi><mo>&amp;le;</mo><mi>q</mi></mrow></msub><mi>c</mi><mrow><mo>(</mo><mi>x</mi><mo>,</mo><msub><mi>y</mi><mi>j</mi></msub><mo>)</mo></mrow><mo>}</mo><mo>,</mo></mrow> Choose 0.5 as the decision threshold, when the confidence of each category that the unseen instance belongs to is less than the decision threshold, return the category with the highest confidence as the category to which the unseen instance belongs.