CN107577785A - A kind of level multi-tag sorting technique suitable for law identification - Google Patents
A kind of level multi-tag sorting technique suitable for law identification Download PDFInfo
- Publication number
- CN107577785A CN107577785A CN201710832304.8A CN201710832304A CN107577785A CN 107577785 A CN107577785 A CN 107577785A CN 201710832304 A CN201710832304 A CN 201710832304A CN 107577785 A CN107577785 A CN 107577785A
- Authority
- CN
- China
- Prior art keywords
- label
- category
- feature
- class
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 67
- 238000012549 training Methods 0.000 claims abstract description 45
- 239000013598 vector Substances 0.000 claims abstract description 41
- 230000011218 segmentation Effects 0.000 claims abstract description 15
- 238000004422 calculation algorithm Methods 0.000 claims description 66
- 238000012360 testing method Methods 0.000 claims description 28
- 238000012545 processing Methods 0.000 claims description 13
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 238000012896 Statistical algorithm Methods 0.000 claims description 5
- 238000007781 pre-processing Methods 0.000 claims description 4
- 238000005259 measurement Methods 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 abstract 1
- 238000007635 classification algorithm Methods 0.000 description 13
- 238000005516 engineering process Methods 0.000 description 7
- 238000013528 artificial neural network Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000011156 evaluation Methods 0.000 description 5
- 238000009826 distribution Methods 0.000 description 4
- 230000008520 organization Effects 0.000 description 4
- 230000009193 crawling Effects 0.000 description 3
- 230000007423 decrease Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 238000012935 Averaging Methods 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 238000010187 selection method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 108090000623 proteins and genes Proteins 0.000 description 1
- 239000002994 raw material Substances 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of level multi-tag sorting technique suitable for law identification, comprise the following steps：Step 1, case facts and its legal provision are extracted from the judgement document by pretreatment；Step 2, the hierarchical structure based on Label space, legal provision corresponding to extension case facts, the class label for making case sample are a subset of Label space；Step 3, case facts text is segmented and part-of-speech tagging, feature selecting is carried out to word segmentation result, choose the Feature Words construction feature vector that can fully represent case facts；Step 4, forecast model is built：Find out the k neighbour sample set N (x) for having no example x in multi-tag training set is extended, to each neighbour's sample, weight is set, the classified weight of each classification is calculated according to k neighbour's sample and has no that example belongs to the confidence level of each classification, finally prediction has no the class label set of example.
Description
Technical Field
The invention belongs to the field of computer data analysis and mining, and relates to a hierarchical multi-label classification method suitable for legal identification.
Background
Hierarchical multi-label classification is a special case of multi-label classification. Unlike general multi-label classification, in a hierarchical multi-label classification problem, each sample can have multiple class labels, while the sample label space is organized in a tree or directed acyclic graph hierarchy. In the directed acyclic graph, one node may have multiple father nodes, which is more complex than a tree structure and has a greater difficulty in designing an algorithm, so that the current research on hierarchical multi-label classification mainly aims at the category label structure of the tree. According to different modes of observing category hierarchical structures by algorithms, the hierarchical multi-label classification algorithm can be divided into a local algorithm and a global algorithm.
And (3) the local algorithm inspects the local classification information of each internal node in the category hierarchy one by one, and converts the hierarchical multi-label classification problem into a plurality of multi-label classification problems. And when training the multi-label classifier on the internal node, an appropriate local sample set needs to be selected. And in the prediction stage, a top-down prediction mode and the like are adopted to enable the prediction result to meet the hierarchical requirement. Document ESULI A, FAGNI T, SEBASTIANI F.TreeBoost.MH A boosting algorithm for multi-label textual classification [ C ]// Stringprocessing and information retrieval.2006:13-24. TreeBoost.MH algorithm is proposed to handle the hierarchical multi-label textual classification problem. The algorithm recursively trains multi-label classifiers on each non-leaf node in the class label tree, the base classifier selects adaboost. The experimental effect proves that the TreeBoost.MH algorithm is better than the AdaBoost.MH algorithm in time efficiency and prediction performance. The documents CERRI R, BARROS R C, DE CARVALHO AC. Hierarchical multi-layer classification of local neural networks [ J ]. Journal of Computer and System Sciences,2014,80 (1): 39-56. A local hierarchical multi-label classification algorithm based on multi-layer perceptron is proposed, one multi-layer perceptron network is trained at each layer of the category hierarchy, each neural network is associated with one category hierarchy for predicting the category label at the level, and the prediction result of the neural network at a certain layer is used as the input of the neural network at the next layer. Because each layer of neural network is trained on the same sample set, the prediction result can not meet the hierarchical constraint, and the prediction result needs to be subjected to subsequent processing to ensure that the prediction result meets the hierarchical constraint.
The local algorithm has the disadvantages that on one hand, a plurality of classifiers are required to be trained, so that the model is relatively complex, and the understandability of the model is influenced; on the other hand, a blocking problem occurs in the prediction process, that is, the samples which are misclassified at the upper layer cannot reach the classifier at the lower layer, and although three strategies of reducing the threshold, limiting voting and expanding threshold multiplication are proposed to solve the blocking problem of the local algorithm, the local algorithm is not ideal in prediction accuracy.
And the global algorithm considers the hierarchical structure of the category as a whole, trains a single hierarchical multi-label classifier and predicts unseen examples. Global algorithms can be mainly classified into the following according to the way they process class label hierarchies: one global algorithm is to use class clustering to first calculate the similarity between the test sample and each class, and then classify the test sample into the closest class. The other method is to convert the hierarchical multi-label classification problem into a multi-label classification problem for processing: documents KIRITCHENKO S, MATWIN S, famill f.functional organization of genes using hierarchical text classification [ J ],2005. Extend the class labels of training samples, increase their ancestor class labels, convert the hierarchical multi-label classification problem into a multi-label classification problem for processing. In the testing stage, the multi-label classification algorithm AdaBoost.MH does not consider the hierarchical structure of the categories, so the same problem as the local algorithm is faced, namely, the predicted result has the condition of inconsistent hierarchies, and the output of the model also needs to be corrected to ensure that the hierarchy limit is met. There is also a global algorithm that adapts existing non-hierarchical classification algorithms to directly process hierarchical information and use the hierarchical information to improve performance. The literature VENS C, STRUFF J, SCHIETGAT L, et al, precision trees for hierarchical multilabelellar classification [ J ]. Machine Learning,2008,73 (2): 185-214, based on the Predictive Clustering Tree (PCT), proposes a Clus-HMC algorithm, trains a decision tree to handle the hierarchical multi-label classification problem, and compares with the Clus-HSC and Clus-SC methods, which ignore the hierarchical structure of class labels, trains an independent classifier for each class label, the Clus-HSC method is hierarchical Clus-SC, the prediction results satisfy the hierarchical constraint. Experimental results show that the global Clus-HMC algorithm is better than the Clus-SC and Clus-HSC algorithms in prediction performance and is better in time efficiency.
In general, global algorithms have two features: considering the hierarchical structure of the categories as a whole once; there is no modularity specific to the local algorithm. The key difference between the global algorithm and the local algorithm lies in the training process, and in the testing stage, the global algorithm can even use a top-down mode like the local algorithm to predict the category of the unseen instances.
Since the organization of class labels in the hierarchical multi-label classification problem is hierarchical, if a sample has a class label c i Then the sample implicitly has c i All ancestor category tags of (a); on the other hand, in predicting the category of the unseen instance, the hierarchical constraint is also satisfied, i.e., it cannot happen that the unseen instance belongs to a category and not to an ancestor category of the category. A general hierarchical multi-label classification algorithm cannot always ensure that a prediction result meets the hierarchical limitation, or cannot obtain the optimal learning effect because the hierarchical structure features of a label space are not utilized. Therefore, the hierarchical multi-label classification algorithm not only fully utilizes the association and the hierarchical structure among the class labels to improve the prediction performance of the classification model, but also ensures that the prediction result meets the hierarchical limitation.
The problem of automatic case-applicable law identification is essentially a hierarchical multi-label classification problem, the type labels of the samples, namely, the applicable legal provisions of the cases, are organized in a tree structure, one case may be applicable to a plurality of legal provisions, and the specific degrees of the applicable legal provisions of the cases may be different. The corresponding hierarchical multi-label classification algorithm for solving the problem of automatic identification of case applicable law needs to be capable of processing tree-shaped class hierarchical structures, and is a non-mandatory leaf node prediction algorithm, and predicted class labels can correspond to any nodes in the class hierarchical structures.
Disclosure of Invention
The invention aims to: the invention aims to solve the technical problem of providing an effective hierarchical multi-label classification method suitable for legal identification aiming at the defects of the prior art.
The technical scheme is as follows: the invention discloses a hierarchical multi-label classification method suitable for legal identification, which comprises the following steps:
step 1, crawling a required referee document original text data set from the Internet by using a crawler technology based on a jsup, wherein one referee document corresponds to one sample, and the sample is randomly divided into a training set and a testing set according to the proportion of 7. Then, preprocessing the official document: extracting case facts and applicable legal provisions thereof from the case facts according to a literary structure of a referee document, wherein the case facts are used for generating feature vectors of case samples, the applicable legal provisions are used for expressing class labels of the case samples, an original text data set is converted into a semi-structured multi-label training set and a semi-structured testing set, and the semi-structured sample is in the form of: (case fact description, legal provisions text); correcting errors and format inconsistency in case-applicable legal provisions; the language technology platform LTP of the Hayada is used as a language processing tool (the LTP is a whole set of Chinese language processing system, a language processing result representation based on XML is formulated, and a whole set of bottom-up rich and efficient Chinese language processing modules (including six Chinese processing core technologies such as lexical, syntactic and semantic) are provided on the basis of the language processing result representation, and an application program interface and a visualization tool based on a Dynamic Link Library (DLL) can be used in a network service form) are used for word segmentation and part-of-speech tagging on case fact description.
And 2, because the organization of the legal provision in the legal system is in a tree structure, correspondingly, the label space formed by the category labels in the multi-label training set is in a tree structure. Based on a hierarchical structure of label space formed by category labels in a multi-label training set, expanding legal provisions corresponding to case facts of all samples, and enabling the category label corresponding to each case fact to be a subset of the label space and meet hierarchical limitation;
step 3, performing feature selection on the word segmentation result from the training set in the step 1 (the word segmentation result refers to the case fact part of the semi-structured multi-label training set in the step 1), and selecting feature words capable of fully representing case facts to construct feature vectors; obtaining a structured extended multi-label training set Tr and a test set Te through text representation;
step 4, constructing a prediction model: finding k neighbor sample sets N (x) of unseen examples x from the extended multi-label test set Te in the extended multi-label training set Tr, wherein the unseen examples are the case facts to be classified, setting weight for each neighbor sample, calculating confidence degrees that the unseen examples belong to each category in the label space according to the classification weight of the k neighbor samples to each category in the label space, predicting a category label set h (x) of the unseen examples, and the h (x) meets the hierarchical constraint. And finally, removing the hierarchical restriction (namely the reverse process of label expansion) in the prediction type label set h (x) according to the tree structure of the label space to obtain the specific applicable legal provisions of the unseen examples. .
The step 2 comprises the following steps:
step 2-1, in the hierarchical multi-label classification problem, a d-dimensional instance space is given(A real number set), and a label space Y = { Y) containing q classes 1 ,y 2 ,…,y q }，y i Representing the ith category, the category label space layerThe substructures may be represented by a doublet of Y, if any i ,y j E is Y and Y i ＜y j Then class y i Belong to the category y j ，y i Is y j Descendant class of, y j Is y i The ancestor class of < represents the partial order relationship of the class label, the partial order relationship < can be understood as "belonging to" the relationship, i.e., if there is y i ,y j E is Y and Y i ＜y j Then class y i Belong to the category y j ，y i Is y j Descendant class of, y j Is y i The ancestor class of (1). The partial order relationship < has asymmetry, non-reflexibility and transitivity, and can be described by the following four characteristics:
a) The only root node in the class label hierarchy is represented by a virtual class label R for any y i Is epsilon of Y, has Y i ＜R；
b) For any y i ,y j E.g. Y, if there is Y i ＜y j Then, then
c) Arbitrary y i Is e.g. Y, has
d) Arbitrary y i ,y j ,y k ∈Y，y i ＜y j And y is j ＜y k Then there is y i ＜y k 。
The multi-label classification problem in which the organizational structure of the category labels satisfies the above four features can be regarded as a hierarchical multi-label classification problem. As can be seen from the above formal definitions, in the hierarchical class label space, all other class nodes (excluding the start node) on the unique path formed by tracing back from any class node to the root node are ancestor class nodes of the class node. So if the sample has a category label y i Then the sample implicitly also has y i All ancestor class labels, which requires a classifier pairThe set of prediction classes h (x) for the unseen example also satisfy the hierarchical constraint, i.e.,and y '< y' e h (x). Wherein y ' is a class in h (x) and y ' is an ancestor class of y ';
step 2-2, for any training sample (x) i ,h i ) (i is more than or equal to 1 and less than or equal to m), m is the number of all the obtained referee document samples, x i e.X is a feature vector with d dimension for representing the fact part of the case,is a and x i A corresponding set of class labels, i.e. x i Corresponding legal provisions, the expanded category label set isThen h is i In' contains h i All category labels in (1) and all ancestor category labels thereof. In a formalized way, the method comprises the following steps of,
the label extension process explicitly expresses the hierarchical relationship of the category labels in the category labels of the sample: if a sample is marked as a certain category, then the ancestor categories of the categories are also explicitly assigned to the sample through label expansion; the category label of each sample can be viewed as a subtree of the label space tree, and the top level of each subtree is the root node. It can be seen that if there is y i ,y j E is Y and Y i ＜y j In the k neighbor samples in the extended multi-label training set, the unseen example has a class label y i Must not be less than having a class label y j The number of samples of (2). The label expansion is an important step for ensuring that the prediction result of the learning algorithm meets the level limit.
The step 3 comprises the following steps:
and 3-1, the purpose of feature selection is to reduce dimensions of features, and since a general text feature selection algorithm cannot directly process a multi-label data set, multi-label data needs to be converted into single-label data for processing. The conversion method comprises the following steps: for each multi-label sample (x, h), the number of label categories in the label category set h is represented by | h |, and is replaced by | h | new single-label samples (x, y |) i )(1≤i≤|h|,y i E h), class y for each new sample i That is, a class label in the original multi-label sample class label set h, table 1 gives an example of converting a multi-label sample into a single-label sample according to the above-mentioned strategy.
TABLE 1 Multi-label sample conversion Process
Step 3-2, after the conversion process of step 3-1, the multi-labeled case samples are converted into case samples with multiple single labels, a general feature selection algorithm can be used to perform feature selection on the word segmentation result obtained from the original training set in step 1, a certain number of feature words with distinguishing capability (usually, depending on the original text data set, for example, when the feature selection is performed by using an information gain algorithm, the total information gain amount of the selected feature words is as large as possible and the number of the feature words is not too much, usually, at least 100 feature words are taken) are selected to form a feature space, and the feature words from the feature space are used to represent the case fact part of each case sample. Wherein, the attribute value corresponding to each feature word, namely the feature weight, is calculated by adopting a commonly used TF-IDF algorithm. And considering the case fact part of each case sample as a document with segmented words, and then the case fact parts of all case samples form a document set. Feature weights tf-i of jth dimension features in ith document in document setdf ij The definition is as follows:
wherein, tf ij Representation feature word t j In document d i Frequency of occurrence of idf j Representation feature word t j Inverse document frequency in a document collection, N represents the total number of documents in the document collection, N j Representation feature word t j Frequency of documents in a document set, i.e. occurrence of a characteristic word t in a document set j The denominator is the normalization factor.
And 3-3, performing feature selection on the word segmentation result obtained from the original training set in the step 1, and selecting about 100 feature words with the most distinguishing capability to form a feature vector. Commonly used text feature selection methods are mainly based on Document Frequency (DF), mutual Information (MI), information Gain (IG), chi-square statistics (χ) 2 Statistical, CHI) and the like. The feature selection based on the document frequency is too simple, the feature words with the most classified information cannot be selected frequently, and the mutual information has the defect that the mutual information is easily influenced by the marginal probability of the feature words, so that the hierarchical multi-label classification method selects the information gain or chi-square statistical algorithm to select the features.
Step 3-3 comprises: and (3) selecting features by adopting an information gain algorithm: the information gain IG (t) of the feature word t is defined as follows:
wherein, P r (y i ) Represents a category y i Probability of occurrence, P r (t) represents the probability of occurrence of the feature t, P r (y i T) represents the category y on the premise that the feature t appears i The probability of occurrence of the event is,indicating the probability that the feature t does not occur,indicating class y without the occurrence of feature t i The probability of occurrence. And calculating the information gain of each feature word in the document set, wherein the feature words with the information gain value lower than a set threshold (for example, 0.15 is taken, and the threshold is set so that the total information gain of the selected feature words is as large as possible and the number of the feature words is not too much) are not included in the feature space.
Step 3-3 can also adopt chi-square statistical algorithm to carry out feature selection: it is assumed that the feature words are not related to the class, and if the test value calculated using the CHI distribution deviates more from the threshold, then the original hypothesis is more confident negated, and an alternative hypothesis to the original hypothesis is accepted: i.e. the characteristic words have a high degree of correlation with the categories.
Let A be the number of documents containing feature words t and belonging to category y, B be the number of documents containing feature words t but not belonging to category y, C be the number of documents not containing feature words t but belonging to category y, D be the number of documents not containing feature words t but not belonging to category y, and N be the total number of documents, then chi-square statistic of feature words t and category y 2 (t, y) is defined as:
when the characteristic word t is independent of the category y, the chi-square statistic is 0, the chi-square statistic of each category is calculated for one characteristic word, and then the mean chi is calculated respectively 2 avg (t) and the maximum value χ 2 max (t), comprehensively considering the two ways, and selecting about 100 most distinguishing feature words:
χ 2 avg (t)＝∑ i＝1 P r (y i )χ 2 (t,y i )，
χ 2 max (t)＝max i＝1,...,q χ 2 (t,y i )。
P r (y i ) Represents the category y i The probability of occurrence. The main advantage of the chi-squared statistical feature selection algorithm over mutual information is that it is a normalized value, and therefore can better scale different feature words in the same category.
In step 4, when k is found to be close to the neighbor, no example x and no sample (x) are found i ,h i ) Distance d (x, x) i ) The inverse of the cosine similarity of their feature vectors is used for the measurement. The cosine similarity cos (γ, λ) between the feature vector γ of the unseen example and the feature vector λ of the neighboring sample is calculated as follows:
where S denotes the index of the vector component, i.e. the position of the component in the vector, S denotes the dimension of the vector, γ s Denotes the s-th component of the vector y, λ s Representing the s-th component of the vector lambda.
In step 4, d (x, x) is used i ) Represent example x and sample (x) i ,h i ) The distance of (2) is calculated by adopting a full label distance weight method or an entropy label distance weight method (x) i ,h i ) e.N (x)) for h i Class y of (1) j Is weighted by classification ij ：
Calculation of w by full-label distance weight method ij ：
Calculation of w by entropy label distance weight method ij ：
Examples belong to category y j C (x, y) of j ) The calculation formula is as follows:
where r denotes the r-th class, w ir Denotes h i Of the r-th class y r The classification weight of (2);
the class label set h (x) for the prediction unseen instance x is:
and selecting 0.5 as a decision threshold, and returning the class with the highest confidence as the class to which the unseen instance belongs when the confidence of the unseen instance belonging to each class is less than the decision threshold.
As a hierarchical multi-label classification method, the prediction result thereof needs to satisfy the hierarchical constraint, that is, and y '< y' e h (x). The following is a demonstration: from the confidence calculation formula, if the algorithm predicts that no instance x has a class label y a (y a E.g., Y), then x belongs to category Y a Confidence of (c) (x, y) a ) Greater than the threshold t or maximum in all categories. Investigation class y a Ancestor class y of b (y b ∈Y,y a ＜y b ) If y is b Corresponding to the virtual root node in the category hierarchy, then x has a category label y a Clearly meeting the hierarchical constraint; otherwise, for any neighbor sample of x (x) i ,Y i ) E.n (x), if y a ∈Y i Then y is also present b ∈Y i And otherwise, the result is not necessarily true, and the label extension process of the training set ensures that the conclusion is true. Therefore, with the full tag distance weight method and the entropy tag distance weight method, it can be derived:
on the denominatorRemain unchanged, so x belongs to class y b Confidence of (c) (x, y) b ) Not less than x belonging to class y a C (x, y) of a ) If there is c (x, y) a )>, t, must also have c (x, y) b )&T, so the prediction result meets the hierarchical constraint.
Finally, the performance evaluation index of the learning method adopts a hierarchical evaluation index: hierarchical precision (hP), hierarchical recall (hR) and hierarchical F metric (hF), which are defined as follows:
wherein,is a set of classes to which the test sample i is predicted to belong and its ancestor classes,is the set of classes to which the test sample i actually belongs and its ancestor classes, and the summation operation is to compute the values over all test samples.
In order to make the identification of case applicable laws more practical, the target category predicted by the algorithm is preferably a specific legal provision, not just a broad law, so the method considers the prediction performance of the target category in both cases of a whole legal provision and a specific legal provision. Hereinafter, the hierarchical precision, recall rate and F metric value of the system when the target category is all legal provisions are denoted by hP _ all, hR _ all and hF _ all, respectively, and the hierarchical precision, recall rate and F metric value of the algorithm when the target category is a specific legal provision are denoted by hP _ partial, hR _ partial and hF _ partial.
Besides the hierarchical evaluation index, the precision, the recall rate and the F metric value of each category can be calculated respectively, and the average value of the precision, the recall rate and the F metric value of all the categories is used as the evaluation index of the system performance, namely Macro-averaging (Macro-averaging) of the precision, the recall rate and the F metric value. For each category, let TP denote the number of true positive examples, FP denote the number of false positive examples, TN denote the number of true negative examples, and FN denote the number of false negative examples, the calculation formulas of the Macro-average Macro-P, macro-R, and Macro-F of the precision, recall ratio, and F-value are as follows:
the invention relates to a global hierarchical multi-label classification method, which considers the hierarchical structure of class labels on the whole and ensures that the prediction result also meets the hierarchical limitation. The learning method is an inertia learning algorithm, a clear prediction model is not required to be constructed on a training set, and only the original multi-label sample is subjected to label expansion and then stored, so that incremental learning is supported; in the prediction stage, k adjacent samples of the unseen examples in the training set are firstly found, the confidence coefficient of the examples belonging to each class is determined according to the classification weight of the adjacent samples to each class, and then the class of the unseen examples is predicted. The learning method is simple in model, supports incremental learning, and can be well applied to automatic identification of the problem of multi-level multi-label classification which contains massive data and continuously increases data in case-applicable law.
Has the beneficial effects that: the hierarchical multi-label classification method suitable for legal recognition provided by the invention fully considers the tree-shaped hierarchical structure of the legal provision label space on the whole, so that the prediction result meets the hierarchical limitation, and the prediction result does not need to be additionally corrected. Meanwhile, the method is simple in model, supports incremental learning, and can be well applied to automatic identification of the problem of multi-level multi-label classification which contains massive data and continuously increases data in case-applicable law.
Drawings
The foregoing and other advantages of the invention will become more apparent from the following detailed description of the invention when taken in conjunction with the accompanying drawings.
FIG. 1 is a main flow chart of the present invention.
FIG. 2 is a sample official document.
FIG. 3 legal provisions tag space tree structure.
The legal provisions of fig. 4 combine frequency distributions.
Fig. 5 shows a comparison of hierarchical index performance under different neighbor numbers.
Figure 6 compares macro average indicator performance for different neighbor numbers.
FIG. 7 is a comparison of performance of indexes under different weighting strategies.
Detailed Description
The invention is further explained by the following embodiments in conjunction with the drawings.
The invention discloses a hierarchical multi-label classification method suitable for legal identification, which comprises the following steps:
step 1, crawling a required referee document original text data set from the Internet by using a crawler technology based on a jsup, and randomly dividing the referee document original text data set into a training set and a test set according to the proportion of 7. Then, preprocessing the referee document, and mainly completing the following work:
extracting case facts and applicable legal provisions thereof from a case according to a literary structure of a referee document, wherein the case facts are used for generating feature vectors of case samples, and the applicable legal provisions are used for representing class labels of the case samples, and converting an original text data set into a semi-structured multi-label training set and a semi-structured testing set;
correcting errors and format inconsistency in case-applicable legal provisions;
and utilizing a language technology platform LTP of the Hadamard to perform word segmentation and part of speech tagging on the case fact description.
And 2, because the organization of the legal provision in the legal system is in a tree structure, correspondingly, the label space formed by the category labels in the multi-label training set is in a tree structure. Based on the hierarchical structure of the label space, expanding legal provisions corresponding to case facts of all samples, and enabling a category label set corresponding to each case fact to be a subset of the label space and meet the hierarchical limitation;
step 3, performing feature selection on the word segmentation result obtained from the original training set in the step 1, and selecting feature words capable of sufficiently representing case facts to construct feature vectors; obtaining a structured extended multi-label training set Tr and a test set Te through text representation;
step 4, constructing a prediction model: finding k neighbor sample sets N (x) of unseen examples x from the extended multi-label test set Te in the extended multi-label training set Tr, setting weight for each neighbor sample, calculating confidence of the unseen examples belonging to each category in the label space according to the classification weight of the k neighbor samples to each category in the label space, predicting category label sets h (x) of the unseen examples, and enabling h (x) to meet the hierarchical constraint. And finally, removing the hierarchical restriction in the prediction category set h (x) (namely, the inverse process of label expansion) according to the tree structure of the label space to obtain a specific applicable legal provision without an example.
The step 2 comprises the following steps:
step 2-1, in the hierarchical multi-label classification problem, a d-dimensional instance space is givenAnd label space Y = { Y) containing q classes 1 ,y 2 ,…,y q }，y i Representing the ith class, the class label spatial hierarchy can be represented by a binary (Y, <) < representing a partial ordering relationship for the class labels, which can be understood as "belonging to" the relationship, i.e., if there is Y i ,y j E is Y and Y i ＜y j Then class y i Belong to the category y j ，y i Is y j Descendant class of, y j Is y i The ancestor class of (1). The partial order relationship < has asymmetry, non-reflexibility and transitivity, and can be described by the following four characteristics:
e) The only root node in the class label hierarchy is represented by a virtual class label R for any y i E is Y, has Y i ＜R；
f) For any y i ,y j E.g. Y, if there is Y i ＜y j Then, then
g) Any of y i Is e.g. Y, has
h) Arbitrary y i ,y j ,y k ∈Y，y i ＜y j And y is j ＜y k Then there is y i ＜y k 。
The multi-label classification problem in which the organizational structure of the category labels satisfies the above four features can be regarded as a hierarchical multi-label classification problem. From the above formal definitions, in the hierarchical class label space, all the nodes on the unique path formed by tracing from any class node up to the root nodeThe other category nodes (excluding the start node) are all ancestor category nodes of the category node. Thus if the sample has a class label c i Then the sample implicitly has c i All ancestor class labels of (a), this requires that the classifier also satisfy the hierarchical constraint for the set of prediction classes h (x) for the unseen instance, i.e.,and y ' < y ' is y ' ∈ h (x).
Step 2-2, for any training sample (x) i ,y i ) (i is more than or equal to 1 and less than or equal to m), m is the number of all the obtained referee document samples, x i E X is a feature vector of dimension d,is a and x i A corresponding set of category labels. Let the expanded category label set be y i ', then y i ' therein contains y i All category labels in (1) and all ancestor category labels thereof. In a formalized way, the raw materials are mixed,
the label extension process expresses the hierarchical relationship of the category labels explicitly in the category labels of the sample: if the sample is marked as certain categories, then the ancestor categories of the categories are also explicitly assigned to the sample through label expansion; the category label of each sample can be viewed as a subtree of the label space tree, and the top level of each subtree is the root node. It can be seen that if there is y i ,y j E is Y and Y i ＜y j In the k neighbor samples in the extended multi-label training set, the unseen example has a class label y i Must not be less than having a class label y j The number of samples of (2). The label expansion is an important step for ensuring that the prediction result of the learning algorithm meets the level limit.
The step 3 comprises the following steps:
step 3-1, the purpose of feature selection is to reduce dimensions of features, and since a general text feature selection algorithm cannot directly process a multi-label data set, multi-label data needs to be converted into single-label data for processing. The conversion method comprises the following steps: for each multi-label sample (x, h), the number of label categories in the label category set h is represented by | h |, and is replaced by | h | new single-label samples (x, y) i )(1≤i≤|y|,y i E h), class y for each new sample i That is, a class label in the original multi-label sample class label set h, table 1 gives an example of converting a multi-label sample into a single-label sample according to the above-mentioned strategy.
TABLE 1 Multi-label sample conversion Process
And 3-2, converting the multi-label case sample into a single-label case sample through the conversion process of the step 3-1, performing feature selection on the word segmentation result obtained from the original training set in the step 1 by using a general feature selection algorithm, and selecting about 100 feature words with the highest distinguishing capability to form a feature space. And (3) representing case fact parts of each case sample by using feature words from a feature space, wherein attribute values, namely feature weights, corresponding to each feature word are calculated by adopting a common TF-IDF algorithm. And considering the case fact part of each sample as a document with words segmented, the case fact parts of all samples form a document set. Feature weight tf-idf of jth dimension feature in ith document ij The definition is as follows:
wherein, tf ij Representation feature word t j In document d i Frequency of occurrence of idf j Representation feature word t j Inverse document frequency in a document set, N denotes the document setTotal number of documents in (1), n j Representation feature word t j Document frequency in a document set, i.e. the occurrence of a characteristic word t in a document set j The denominator is the normalization factor.
And 3-3, performing feature selection on the word segmentation result obtained from the original training set in the step 1, and selecting a certain number of feature words with distinguishing capability to form feature vectors. Commonly used text feature selection methods are mainly based on Document Frequency (DF), mutual Information (MI), information Gain (IG), chi-square statistics (χ) 2 Statistical, CHI) and other weighing indexes. The feature selection based on the document frequency is too simple, the feature words with the most classified information cannot be selected, and the mutual information has the defect that the mutual information is easily influenced by the marginal probability of the feature words, so that the hierarchical multi-label classification method selects information gain or chi-square statistical algorithm to select the features.
Step 3-3 comprises: and (3) selecting features by adopting an information gain algorithm: the information gain IG (t) of the feature word t is defined as follows:
wherein, P r (y i ) Represents the category y i Probability of occurrence, P r (t) represents the probability of occurrence of the feature t, P r (y i I t) represents the category y on the premise that the feature t appears i The probability of occurrence of the event is determined,indicating the probability that the feature t does not occur,indicating class y without the occurrence of feature t i The probability of occurrence. And calculating the information gain of each feature word in the document set, wherein the feature words with the information gain value lower than a set threshold value are not included in the feature space.
Step 3-3 can also adopt chi-square statistical algorithm to carry out feature selection on case fact texts in the training set: it is assumed that the feature words are not related to class, and if the test value calculated using the CHI distribution deviates more from the threshold, then there is more confidence in negating the original hypothesis, accepting an alternative hypothesis to the original hypothesis: i.e. the characteristic words have a high degree of correlation with the categories.
Let A be the number of documents containing feature words t and belonging to category y, B be the number of documents containing feature words t but not belonging to category y, C be the number of documents not containing feature words t but belonging to category y, D be the number of documents not containing feature words t but not belonging to category y, and N be the total number of documents, then chi-square statistic of feature words t and category y 2 (t, y) is defined as:
when the characteristic word t is independent of the category y, the chi-square statistic is 0, the chi-square statistic about each category is calculated for one characteristic word, and then the mean chi is calculated respectively 2 avg (t) and the maximum value X 2 max (t), comprehensively considering the two modes, and selecting the most distinguishing characteristic words:
X 2 avg (t)＝∑ i＝1 P r (y i )χ 2 (t,y i )，
χ 2 max (t)＝max i＝1,...,q χ 2 (t,y i )。
P r (y i ) Represents a category y i Probability of occurrence. The main advantage of the chi-squared statistical feature selection algorithm over mutual information is that it is a normalized value, and therefore can better weigh different feature words in the same category.
In step 4, when k is found to be close to the neighbor, no example x and no sample (x) are found i ,h i ) D (x, x) i ) The inverse of the cosine similarity of their feature vectors is used for the measurement. The cosine similarity cos (γ, λ) between the feature vector γ of the unseen example and the feature vector λ of the neighboring sample is calculated as follows:
where S denotes the index of the vector component, i.e. the position of the component in the vector, S denotes the dimension of the vector, γ s Denotes the s-th component of the vector y, λ s Representing the s-th component of the vector lambda.
In step 4, d (x, x) is used i ) Represent example x and sample (x) i ,h i ) The distance of (2) is calculated by using a full label distance weighting method to obtain a sample ((x) i ,h i ) E N (x)) for category y j By classification weight w ij ：
Calculation of w by full-label distance weight method ij ：
Calculation of w by entropy label distance weight method ij ：
Unseen examples belong to category y j Confidence of (c) (x, y) j ) The calculation formula is as follows:
the set of class labels h (x) for the prediction of unseen instance x is:
and selecting 0.5 as a decision threshold, and returning the class with the highest confidence as the class to which the unseen instance belongs when the confidence of the unseen instance belonging to each class is less than the decision threshold.
Examples
As shown in FIG. 1, the method comprises the following steps:
step one, crawling required referee document original text data sets from the Internet by using a joup-based crawler technology, and randomly dividing the data sets into a training set and a test set according to the proportion of 7. Then, preprocessing the referee document, and mainly completing the following work:
extracting case facts and applicable legal provisions thereof from a case according to a literary structure of a referee document, wherein the case facts are used for generating feature vectors of case samples, and the applicable legal provisions are used for representing class labels of the case samples, and converting an original text data set into a semi-structured multi-label training set and a semi-structured testing set;
correcting errors and format inconsistency in case-applicable legal provisions;
and utilizing a language technology platform LTP of the Hadamard to perform word segmentation and part of speech tagging on the case fact description.
Expanding legal provisions corresponding to case facts of all samples based on a hierarchical structure of a label space, so that a category label corresponding to each case fact is a subset of the label space and meets the hierarchical limitation;
step three, performing feature selection on the word segmentation result obtained from the original training set in the step 1, and selecting feature words capable of sufficiently representing case facts to construct feature vectors; obtaining a structured extended multi-label training set Tr and a test set Te through text representation;
step four, constructing a prediction model: firstly, finding k neighbor sample sets N (x) of unseen examples x from an extended multi-label test set Te in an extended multi-label training set Tr, setting weight for each neighbor sample, calculating confidence coefficients of the unseen examples belonging to various categories in a label space according to classification weights of the k neighbor samples on various categories in the label space, predicting category label sets h (x) of the unseen examples, and enabling the h (x) to meet the hierarchical constraint. And finally, removing the hierarchical restriction in the prediction category set h (x) (namely, the inverse process of label expansion) according to the tree structure of the label space to obtain a specific applicable legal provision without an example.
The implementation data is obtained from official documents of people's court at all levels of Zhejiang province published by Zhejiang court.
FIG. 2 is a sample official document in which the straight underline marked portion is the case fact portion and the curved underline marked portion is the applicable legal provisions for the case. And extracting case facts and legal provisions thereof according to the law of the official document. The pretreatment work is mainly to clean and correct the legal part of the case.
In fig. 3, a tree structure of a legal provision tag space is shown. Based on the hierarchical structure, the legal provision corresponding to each case fact is subjected to label expansion.
Fig. 4 is a legal provision combination histogram. According to the frequency of citation of each legal provision, 26 laws such as ' people's republic of China ' litigation law, ' people's republic of China ' and ' 451 specific legal provisions contained in the laws are selected as category labels to form a label space, namely, the dimension of the label space is 477. The set of category labels for each case sample is represented in the form of a label vector, each dimension of which represents a category label in the label space, i.e., a complete legal provision. If a case is applicable to a certain legal provision, the corresponding label entry values of the legal provision and all legal provisions containing the legal provision in the label vector are both 1, otherwise, the corresponding label entry values are 0. Therefore, the label vector of each sample corresponds to one legal provision combination, the frequency of occurrence of each combination is the number of corresponding case samples, and the frequency of occurrence of each legal provision combination can also reflect some properties of the case sample set. By calculating each, and selecting the combination with higher frequency of occurrence and arranging it in order from large to small, fig. 4 can be obtained. As can be seen from the figure, the occurrence frequency of the legal provision combinations is approximately in a long tail distribution, the occurrence frequency of a few legal provision combinations is extremely high, which indicates that a large number of case samples are suitable for the legal provision combinations, and in addition, the occurrence frequency of most legal provision combinations is relatively balanced.
And step three, selecting an information gain algorithm to select characteristics. Through calculating the information gain of each feature word, it can be found that most words with higher information gain are verbs or nouns, and table 2 shows the proportion of verbs and nouns in the feature words with the highest information gain value, so that the nouns and the verbs have higher distinguishing capability in the problem of legal identification compared with words with other properties.
Table 2 ratio of verb nouns in feature words:
| number of feature words | Verb noun number ratio | Verb noun information gain total proportion |
| 100 | 88.0％ | 87.9％ |
| 200 | 80.0％ | 82.3％ |
| 300 | 81.0％ | 82.5％ |
| 400 | 80.5％ | 82.0％ |
| 500 | 76.8％ | 79.7％ |
Table 3 summary of experimental training set and test set:
| number of samples | Sample average class label number | |
| Training set | 102608 | 7.6344 |
| Test set | 44210 | 7.6397 |
Fig. 5 and 6 are comparisons of the performance of the hierarchical index and the macro-average index when different numbers of neighbors are taken.
As can be seen from fig. 5: when the number of the neighbors is even, the precision of the algorithm is high, and the recall rate is low; when the number of the neighbors is odd, the precision of the algorithm is low, and the recall rate is high. This distinction becomes progressively smaller as the number of neighbors increases. This phenomenon can be explained by analyzing the principles of the algorithm: the decision threshold set by the algorithm is 0.5, and when the number of neighbors is even, only the class label with the occurrence frequency exceeding k =2 is predicted as the class label of the unseen instance due to the addition of the smoothing parameter, and the class label with the occurrence frequency just being k =2 is not given to the unseen instance. Therefore, when the number of neighbors is even, the condition that each class label endows the unseen instance is severer, so that the prediction precision of the algorithm is higher, and the recall rate is correspondingly lower. This effect is gradually reduced as the number of neighbors increases, and thus the difference becomes smaller. It can also be seen from the figure that when the target category is all legal provisions, each prediction index of the algorithm is higher than that when the target category is a specific legal provision. This is because broader legal categories contain more case samples, thus making the model more predictive in these categories. In summary, when the k value of the number of neighbors is 5, the comprehensive prediction performance of the algorithm is best.
From fig. 6 it can be found that: as the number of the neighbors increases, the macro average precision, recall ratio and F metric value of the algorithm are all reduced. The reason for this may be that as the number of neighbors increases, it is more difficult for the classes with fewer samples to reach the decision threshold, thus leading to a decrease in the prediction performance of most classes and ultimately to a decrease in the corresponding macro average performance.
Fig. 7 shows the performance of the algorithm on each evaluation index when the number of the fixed neighbors is 5 and the sample weight strategy is a full label distance weight method and an entropy label distance weight method, respectively. In general, whether hierarchical indexes or macro-average indexes are adopted, an entropy label distance weighting strategy can achieve a better effect in precision, and a full label distance weighting strategy can achieve a better effect in recall rate and F metric values. The entropy label weight strategy is biased to samples with fewer class labels, and in the expanded hierarchical multi-label samples, the more specific the class to which the sample belongs, the more class labels, the smaller the classification weight under the entropy label weight strategy, so that the prediction result is more biased to the upper class by adopting the entropy label weight strategy, and the larger the generalization error. Although the algorithm has a decline in performance when the target category is a specific legal provision, there is still a hierarchical accuracy close to 80% and a hierarchical recall rate of more than 65%, indicating that case-applicable legal identification based on the present hierarchical multi-label classification algorithm is valid.
In consideration of the two cases that the target category is all legal provisions and specific legal provisions, the macro average precision, recall rate and F metric value of the algorithm when the target category is all legal provisions are respectively represented by mP _ all, mP _ all and mP _ all, and the macro average precision, recall rate and F metric value of the algorithm when the target category is specific legal provisions are represented by mP _ partial, mP _ partial and mP _ partial.
Two common hierarchical multi-label classification algorithms, namely a TreeBoost.MH local algorithm and a Clus-HMC global algorithm, are selected respectively in the implementation and are compared with the prediction performance of the hierarchical multi-label classification algorithm, the performance comparison of the hierarchical multi-label classification algorithm on each hierarchical index is given in a table 5, and the prediction performance comparison of the hierarchical multi-label classification algorithm on each macro-average index is given in a table 6.
Table 5 comparison of hierarchical index performance of each algorithm:
table 6 macro-average performance comparison of algorithms:
the fact proves that the multi-label classification algorithm of the level can achieve better prediction performance than the existing method. By combining the characteristic that the Lazy-HMC algorithm supports incremental learning, an effective and applicable automatic case law identification system can be constructed by utilizing the Lazy-HMC algorithm.
The present invention provides a hierarchical multi-label classification method suitable for legal identification, and a plurality of methods and ways for implementing the technical scheme, and the above description is only a preferred embodiment of the present invention, it should be noted that, for those skilled in the art, a plurality of improvements and decorations can be made without departing from the principle of the present invention, and these improvements and decorations should also be regarded as the protection scope of the present invention. All the components not specified in the present embodiment can be realized by the prior art.
Claims (8)
1. A hierarchical multi-label classification method suitable for legal identification is characterized by comprising the following steps:
step 1, acquiring an original text data set of a referee document, dividing the original text data set into a training set and a testing set, and preprocessing: extracting case facts and applicable legal provisions thereof from the case facts according to a literary structure of a referee document, wherein the case facts are used for generating feature vectors of case samples, the applicable legal provisions are used for representing category labels of the case samples, and an original text data set is converted into a semi-structured multi-label training set and a semi-structured testing set; performing word segmentation and part-of-speech tagging on case fact description;
step 2, based on a hierarchical structure of a label space formed by category labels in a multi-label training set, expanding legal provisions corresponding to case facts of all samples, and enabling the category label corresponding to each case fact to be a subset of the label space and meet hierarchical restriction;
step 3, performing feature selection on the word segmentation result from the training set in the step 1, and selecting feature words capable of sufficiently representing case facts to construct feature vectors; obtaining a structured extended multi-label training set Tr and a test set Te through text representation;
step 4, constructing a prediction model: finding out a k neighbor sample set N (x) of an unseen example x from an extended multi-label test set Te in an extended multi-label training set Tr, wherein the unseen example is a case fact to be classified, setting weight for each neighbor sample, calculating confidence coefficient of the unseen example belonging to each class according to the classification weight of the k neighbor samples to each class, predicting a class label set h (x) of the unseen example, wherein h (x) meets the hierarchical constraint, and finally removing the hierarchical constraint in the predicted class label set h (x) according to a tree structure of a label space to obtain a specific applicable legal provision of the unseen example.
2. The method according to claim 1, wherein in step 1, the raw text data set of the official document is randomly divided into a training set and a test set in a ratio of 7.
3. The method of claim 2, wherein: the step 2 comprises the following steps:
step 2-1, in the hierarchical multi-label classification problem, a d-dimensional instance space is given Is a real number set, and a label space Y = { Y) containing q classes 1 ,y 2 ,…,y q }，y i Representing the ith category, the spatial hierarchy of category labels is represented by a binary (Y, <) < representing the partial order relationship of the category labels, if Y exists i ,y j E is Y and Y i ＜y j Then class y i Belong to the category y j ，y i Is y j Descendant class of y j Is y i The classifier must satisfy the hierarchical constraint on the set of predicted classes h (x) for unseen instances, i.e.,y 'belongs to h (x), wherein y' is a category in h (x), and y 'is an ancestor category of y';
step 2-2, for arbitrary samples (x) i ,h i ) (i is more than or equal to 1 and less than or equal to m), and m is the sample of all the obtained referee documentsNumber, x i The epsilon X is a characteristic vector with the dimension d and is used for representing the fact part of the case,is equal to x i A corresponding set of class labels, i.e. x i Corresponding legal provision, the expanded category label set is h i ′，Then h is i In which h is included i All class labels in (1) and all ancestor class labels thereof:
4. a method as claimed in claim 3, characterized in that: the step 3 comprises the following steps:
step 3-1, converting the multi-label data into single-label data for processing: for each multi-label sample (x, h), the number of label categories in the label category set h is represented by | h |, and is replaced by | h | new single label samples (x, y) i )(1≤i≤|h|,y i E h), class y for each new sample i The category label is a category label in the original multi-label sample category label set h;
step 3-2, through the conversion process of step 3-1, the case samples with multiple labels are converted into case samples with multiple single labels, the case fact part of each case sample is regarded as a document with words, then the case fact parts of all case samples form a document set, and the feature weights tf-idf of the jth dimension feature in the ith document in the document set ij The definition is as follows:
wherein, tf ij Representation featureWord t j In document d i Frequency of occurrence in, idf j Representation feature word t j Inverse document frequency in a document collection, N represents the total number of documents in the document collection, N j Representation feature word t j Frequency of documents in a document set, i.e. occurrence of a characteristic word t in a document set j The denominator of the number of documents is a normalization factor;
and 3-3, performing feature selection on the word segmentation result obtained from the original training set in the step 1 by using a general feature selection algorithm, and selecting a certain number of feature words with distinguishing capability to form a feature space.
5. The method of claim 4, wherein: step 3-3 comprises: and (3) selecting features by adopting an information gain algorithm: the information gain IG (t) of the feature word t is defined as follows:
wherein, P r (y i ) Represents a category y i Probability of occurrence, P r (t) represents the probability of occurrence of the feature t, P r (y i I t) represents the category y on the premise that the feature t appears i The probability of occurrence of the event is,indicating the probability that the feature t does not occur,indicating class y without the occurrence of feature t i And calculating the information gain of each characteristic word in the document set according to the occurrence probability, wherein the characteristic words with the information gain value lower than a set threshold value are not included in the characteristic space.
6. The method of claim 5, wherein: step 3-3 comprises: and (3) selecting features by adopting a chi-square statistical algorithm:
let A be the number of documents containing a feature word t and belonging to category y, B be the number of documents containing a feature word t but not belonging to category y, C be the number of documents not containing a feature word t but belonging to category y, D be the number of documents not containing a feature word t but not belonging to category y, and N be the total number of documents, then chi-square statistic χ of feature word t and category y 2 (t, y) is defined as:
when the characteristic word t is independent of the category y, the chi-square statistic is 0, the chi-square statistic about each category is calculated for one characteristic word, and then the mean chi is calculated respectively 2 avg (t) and the maximum value χ 2 max (t) selecting a certain number of characteristic words with distinguishing capability by comprehensively considering the two ways, wherein P is r (y i ) Represents the probability of occurrence of a category:
χ 2 avg (t)＝∑ i＝1 P r (y i )χ 2 (t,y i )，
χ 2 max (t)＝max i＝1,...,q χ 2 (t,y i )。
7. the method according to claim 5 or 6, characterized in that: in step 4, when k is found to be close to the neighbor, no example x and no sample (x) are found i ,h i ) Distance d (x, x) i ) The inverse of the cosine similarity of the feature vectors is adopted for measurement, and the cosine similarity cos (gamma, lambda) of the feature vector gamma of the unseen example and the feature vector lambda of the adjacent sample is calculated as follows:
where S denotes the index of the vector component, i.e. the position of the component in the vector, S denotes the vector dimension, γ s Represents the s-th component of the vector y,λ s representing the s-th component of the vector lambda.
8. The method of claim 7, wherein: in step 4, d (x, x) is used i ) Showing that example x and neighboring sample (x) are not seen i ,h i ) The distance of (2) is calculated by adopting a full label distance weight method or an entropy label distance weight method to obtain a neighboring sample ((x) i ,h i ) e.N (x)) for h i Class y of j By classification weight w ij ：
W is calculated by a full-label distance weight method ij ：
Calculation of w by entropy label distance weight method ij ：
Unseen examples belong to category y j Confidence of (c) (x, y) j ) The calculation formula is as follows:
wherein w ir Denotes h i Of the r-th class y r The classification weight of (a);
the class label set h (x) for the prediction unseen instance x is:
and selecting 0.5 as a decision threshold, and returning the class with the highest confidence as the class to which the unseen instance belongs when the confidence of the unseen instance belonging to each class is less than the decision threshold.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710832304.8A CN107577785B (en) | 2017-09-15 | 2017-09-15 | Hierarchical multi-label classification method suitable for legal identification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710832304.8A CN107577785B (en) | 2017-09-15 | 2017-09-15 | Hierarchical multi-label classification method suitable for legal identification |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107577785A true CN107577785A (en) | 2018-01-12 |
| CN107577785B CN107577785B (en) | 2020-02-07 |
Family
ID=61035969
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710832304.8A Active CN107577785B (en) | 2017-09-15 | 2017-09-15 | Hierarchical multi-label classification method suitable for legal identification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107577785B (en) |
Cited By (45)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108304386A (en) * | 2018-03-05 | 2018-07-20 | 上海思贤信息技术股份有限公司 | A kind of logic-based rule infers the method and device of legal documents court verdict |
| CN108334500A (en) * | 2018-03-05 | 2018-07-27 | 上海思贤信息技术股份有限公司 | A kind of judgement document's mask method and device based on machine learning algorithm |
| CN108664924A (en) * | 2018-05-10 | 2018-10-16 | 东南大学 | A kind of multi-tag object identification method based on convolutional neural networks |
| CN108763361A (en) * | 2018-05-17 | 2018-11-06 | 南京大学 | A kind of multi-tag taxonomy model method based on topic model |
| CN109543178A (en) * | 2018-11-01 | 2019-03-29 | 银江股份有限公司 | A kind of judicial style label system construction method and system |
| CN109685158A (en) * | 2019-01-08 | 2019-04-26 | 东北大学 | A kind of cluster result semantic feature extraction and method for visualizing based on strong point collection |
| CN109919368A (en) * | 2019-02-26 | 2019-06-21 | 西安交通大学 | A kind of law article recommendation forecasting system and method based on associated diagram |
| CN109961094A (en) * | 2019-03-07 | 2019-07-02 | 北京达佳互联信息技术有限公司 | Sample acquiring method, device, electronic equipment and readable storage medium storing program for executing |
| CN110046256A (en) * | 2019-04-22 | 2019-07-23 | 成都四方伟业软件股份有限公司 | The prediction technique and device of case differentiation result |
| CN110135592A (en) * | 2019-05-16 | 2019-08-16 | 腾讯科技（深圳）有限公司 | Classifying quality determines method, apparatus, intelligent terminal and storage medium |
| CN110163849A (en) * | 2019-04-28 | 2019-08-23 | 上海鹰瞳医疗科技有限公司 | Training data processing method, disaggregated model training method and equipment |
| CN110245229A (en) * | 2019-04-30 | 2019-09-17 | 中山大学 | A kind of deep learning theme sensibility classification method based on data enhancing |
| CN110245907A (en) * | 2018-03-09 | 2019-09-17 | 北京国双科技有限公司 | The generation method and device of court's trial notes content |
| CN110287287A (en) * | 2019-06-18 | 2019-09-27 | 北京百度网讯科技有限公司 | Case by prediction technique, device and server |
| CN110347839A (en) * | 2019-07-18 | 2019-10-18 | 湖南数定智能科技有限公司 | A kind of file classification method based on production multi-task learning model |
| CN110442722A (en) * | 2019-08-13 | 2019-11-12 | 北京金山数字娱乐科技有限公司 | Method and device for training classification model and method and device for data classification |
| CN110543634A (en) * | 2019-09-02 | 2019-12-06 | 北京邮电大学 | corpus data set processing method and device, electronic equipment and storage medium |
| CN110633365A (en) * | 2019-07-25 | 2019-12-31 | 北京国信利斯特科技有限公司 | Word vector-based hierarchical multi-label text classification method and system |
| CN110751188A (en) * | 2019-09-26 | 2020-02-04 | 华南师范大学 | User label prediction method, system and storage medium based on multi-label learning |
| CN110781650A (en) * | 2020-01-02 | 2020-02-11 | 四川大学 | Method and system for automatically generating referee document based on deep learning |
| CN110825879A (en) * | 2019-09-18 | 2020-02-21 | 平安科技（深圳）有限公司 | Case decision result determination method, device and equipment and computer readable storage medium |
| CN110837735A (en) * | 2019-11-17 | 2020-02-25 | 太原蓝知科技有限公司 | Intelligent data analysis and identification method and system |
| CN110851596A (en) * | 2019-10-11 | 2020-02-28 | 平安科技（深圳）有限公司 | Text classification method and device and computer readable storage medium |
| CN110895703A (en) * | 2018-09-12 | 2020-03-20 | 北京国双科技有限公司 | Legal document routing identification method and device |
| CN110909157A (en) * | 2018-09-18 | 2020-03-24 | 阿里巴巴集团控股有限公司 | Text classification method and device, computing equipment and readable storage medium |
| CN110968693A (en) * | 2019-11-08 | 2020-04-07 | 华北电力大学 | Multi-label text classification calculation method based on ensemble learning |
| CN111126053A (en) * | 2018-10-31 | 2020-05-08 | 北京国双科技有限公司 | Information processing method and related equipment |
| CN111143569A (en) * | 2019-12-31 | 2020-05-12 | 腾讯科技（深圳）有限公司 | Data processing method and device and computer readable storage medium |
| CN111540468A (en) * | 2020-04-21 | 2020-08-14 | 重庆大学 | ICD automatic coding method and system for visualization of diagnosis reason |
| CN111723208A (en) * | 2020-06-28 | 2020-09-29 | 西南财经大学 | Conditional classification tree-based legal decision document multi-classification method and device and terminal |
| CN111738303A (en) * | 2020-05-28 | 2020-10-02 | 华南理工大学 | Long-tail distribution image identification method based on hierarchical learning |
| CN111737479A (en) * | 2020-08-28 | 2020-10-02 | 深圳追一科技有限公司 | Data acquisition method and device, electronic equipment and storage medium |
| CN111930944A (en) * | 2020-08-12 | 2020-11-13 | 中国银行股份有限公司 | File label classification method and device |
| CN112016430A (en) * | 2020-08-24 | 2020-12-01 | 郑州轻工业大学 | Hierarchical action identification method for multi-mobile-phone wearing positions |
| CN112131884A (en) * | 2020-10-15 | 2020-12-25 | 腾讯科技（深圳）有限公司 | Method and device for entity classification and method and device for entity presentation |
| CN112182213A (en) * | 2020-09-27 | 2021-01-05 | 中润普达(十堰)大数据中心有限公司 | Modeling method based on abnormal lacrimation feature cognition |
| CN112232524A (en) * | 2020-12-14 | 2021-01-15 | 北京沃东天骏信息技术有限公司 | Multi-label information identification method and device, electronic equipment and readable storage medium |
| CN112464973A (en) * | 2020-08-13 | 2021-03-09 | 浙江师范大学 | Multi-label classification method based on average distance weight and value calculation |
| CN113407727A (en) * | 2021-03-22 | 2021-09-17 | 天津汇智星源信息技术有限公司 | Qualitative measure and era recommendation method based on legal knowledge graph and related equipment |
| CN114117040A (en) * | 2021-11-08 | 2022-03-01 | 重庆邮电大学 | Text data multi-label classification method based on label specific features and relevance |
| US11379758B2 (en) | 2019-12-06 | 2022-07-05 | International Business Machines Corporation | Automatic multilabel classification using machine learning |
| CN114860892A (en) * | 2022-07-06 | 2022-08-05 | 腾讯科技（深圳）有限公司 | Hierarchical category prediction method, device, equipment and medium |
| CN117216688A (en) * | 2023-11-07 | 2023-12-12 | 西南科技大学 | Enterprise industry identification method and system based on hierarchical label tree and neural network |
| CN118210926A (en) * | 2024-05-21 | 2024-06-18 | 山东云海国创云计算装备产业创新中心有限公司 | Text label prediction method and device, electronic equipment and storage medium |
| CN118313376A (en) * | 2024-06-07 | 2024-07-09 | 腾讯科技（深圳）有限公司 | Text processing method, device, equipment, storage medium and product |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104199857A (en) * | 2014-08-14 | 2014-12-10 | 西安交通大学 | Tax document hierarchical classification method based on multi-tag classification |
| US20150161198A1 (en) * | 2013-12-05 | 2015-06-11 | Sony Corporation | Computer ecosystem with automatically curated content using searchable hierarchical tags |
| CN104881689A (en) * | 2015-06-17 | 2015-09-02 | 苏州大学张家港工业技术研究院 | Method and system for multi-label active learning classification |
| CN105868773A (en) * | 2016-03-23 | 2016-08-17 | 华南理工大学 | Hierarchical random forest based multi-tag classification method |
| CN106126972A (en) * | 2016-06-21 | 2016-11-16 | 哈尔滨工业大学 | A kind of level multi-tag sorting technique for protein function prediction |
-
2017
- 2017-09-15 CN CN201710832304.8A patent/CN107577785B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150161198A1 (en) * | 2013-12-05 | 2015-06-11 | Sony Corporation | Computer ecosystem with automatically curated content using searchable hierarchical tags |
| CN104199857A (en) * | 2014-08-14 | 2014-12-10 | 西安交通大学 | Tax document hierarchical classification method based on multi-tag classification |
| CN104881689A (en) * | 2015-06-17 | 2015-09-02 | 苏州大学张家港工业技术研究院 | Method and system for multi-label active learning classification |
| CN105868773A (en) * | 2016-03-23 | 2016-08-17 | 华南理工大学 | Hierarchical random forest based multi-tag classification method |
| CN106126972A (en) * | 2016-06-21 | 2016-11-16 | 哈尔滨工业大学 | A kind of level multi-tag sorting technique for protein function prediction |
Cited By (76)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108334500A (en) * | 2018-03-05 | 2018-07-27 | 上海思贤信息技术股份有限公司 | A kind of judgement document's mask method and device based on machine learning algorithm |
| CN108304386A (en) * | 2018-03-05 | 2018-07-20 | 上海思贤信息技术股份有限公司 | A kind of logic-based rule infers the method and device of legal documents court verdict |
| CN108334500B (en) * | 2018-03-05 | 2022-02-22 | 上海思贤信息技术股份有限公司 | Referee document labeling method and device based on machine learning algorithm |
| CN110245907A (en) * | 2018-03-09 | 2019-09-17 | 北京国双科技有限公司 | The generation method and device of court's trial notes content |
| CN108664924A (en) * | 2018-05-10 | 2018-10-16 | 东南大学 | A kind of multi-tag object identification method based on convolutional neural networks |
| CN108763361A (en) * | 2018-05-17 | 2018-11-06 | 南京大学 | A kind of multi-tag taxonomy model method based on topic model |
| CN110895703A (en) * | 2018-09-12 | 2020-03-20 | 北京国双科技有限公司 | Legal document routing identification method and device |
| CN110895703B (en) * | 2018-09-12 | 2023-05-23 | 北京国双科技有限公司 | Legal document case recognition method and device |
| CN110909157A (en) * | 2018-09-18 | 2020-03-24 | 阿里巴巴集团控股有限公司 | Text classification method and device, computing equipment and readable storage medium |
| CN110909157B (en) * | 2018-09-18 | 2023-04-11 | 阿里巴巴集团控股有限公司 | Text classification method and device, computing equipment and readable storage medium |
| CN111126053B (en) * | 2018-10-31 | 2023-07-04 | 北京国双科技有限公司 | Information processing method and related equipment |
| CN111126053A (en) * | 2018-10-31 | 2020-05-08 | 北京国双科技有限公司 | Information processing method and related equipment |
| CN109543178A (en) * | 2018-11-01 | 2019-03-29 | 银江股份有限公司 | A kind of judicial style label system construction method and system |
| CN109543178B (en) * | 2018-11-01 | 2023-02-28 | 银江技术股份有限公司 | Method and system for constructing judicial text label system |
| CN109685158A (en) * | 2019-01-08 | 2019-04-26 | 东北大学 | A kind of cluster result semantic feature extraction and method for visualizing based on strong point collection |
| CN109919368B (en) * | 2019-02-26 | 2020-11-17 | 西安交通大学 | Law recommendation prediction system and method based on association graph |
| CN109919368A (en) * | 2019-02-26 | 2019-06-21 | 西安交通大学 | A kind of law article recommendation forecasting system and method based on associated diagram |
| CN109961094B (en) * | 2019-03-07 | 2021-04-30 | 北京达佳互联信息技术有限公司 | Sample acquisition method and device, electronic equipment and readable storage medium |
| CN109961094A (en) * | 2019-03-07 | 2019-07-02 | 北京达佳互联信息技术有限公司 | Sample acquiring method, device, electronic equipment and readable storage medium storing program for executing |
| CN110046256A (en) * | 2019-04-22 | 2019-07-23 | 成都四方伟业软件股份有限公司 | The prediction technique and device of case differentiation result |
| CN110163849A (en) * | 2019-04-28 | 2019-08-23 | 上海鹰瞳医疗科技有限公司 | Training data processing method, disaggregated model training method and equipment |
| CN110245229A (en) * | 2019-04-30 | 2019-09-17 | 中山大学 | A kind of deep learning theme sensibility classification method based on data enhancing |
| CN110245229B (en) * | 2019-04-30 | 2023-03-28 | 中山大学 | Deep learning theme emotion classification method based on data enhancement |
| CN110135592B (en) * | 2019-05-16 | 2023-09-19 | 腾讯科技（深圳）有限公司 | Classification effect determining method and device, intelligent terminal and storage medium |
| CN110135592A (en) * | 2019-05-16 | 2019-08-16 | 腾讯科技（深圳）有限公司 | Classifying quality determines method, apparatus, intelligent terminal and storage medium |
| CN110287287A (en) * | 2019-06-18 | 2019-09-27 | 北京百度网讯科技有限公司 | Case by prediction technique, device and server |
| CN110287287B (en) * | 2019-06-18 | 2021-11-23 | 北京百度网讯科技有限公司 | Case prediction method and device and server |
| CN110347839A (en) * | 2019-07-18 | 2019-10-18 | 湖南数定智能科技有限公司 | A kind of file classification method based on production multi-task learning model |
| CN110347839B (en) * | 2019-07-18 | 2021-07-16 | 湖南数定智能科技有限公司 | Text classification method based on generative multi-task learning model |
| CN110633365A (en) * | 2019-07-25 | 2019-12-31 | 北京国信利斯特科技有限公司 | Word vector-based hierarchical multi-label text classification method and system |
| CN110442722A (en) * | 2019-08-13 | 2019-11-12 | 北京金山数字娱乐科技有限公司 | Method and device for training classification model and method and device for data classification |
| CN110442722B (en) * | 2019-08-13 | 2022-05-13 | 北京金山数字娱乐科技有限公司 | Method and device for training classification model and method and device for data classification |
| CN110543634A (en) * | 2019-09-02 | 2019-12-06 | 北京邮电大学 | corpus data set processing method and device, electronic equipment and storage medium |
| CN110825879A (en) * | 2019-09-18 | 2020-02-21 | 平安科技（深圳）有限公司 | Case decision result determination method, device and equipment and computer readable storage medium |
| CN110825879B (en) * | 2019-09-18 | 2024-05-07 | 平安科技（深圳）有限公司 | Decide a case result determination method, device, equipment and computer readable storage medium |
| CN110751188A (en) * | 2019-09-26 | 2020-02-04 | 华南师范大学 | User label prediction method, system and storage medium based on multi-label learning |
| CN110851596A (en) * | 2019-10-11 | 2020-02-28 | 平安科技（深圳）有限公司 | Text classification method and device and computer readable storage medium |
| CN110851596B (en) * | 2019-10-11 | 2023-06-27 | 平安科技（深圳）有限公司 | Text classification method, apparatus and computer readable storage medium |
| CN110968693A (en) * | 2019-11-08 | 2020-04-07 | 华北电力大学 | Multi-label text classification calculation method based on ensemble learning |
| CN110837735B (en) * | 2019-11-17 | 2023-11-03 | 内蒙古中媒互动科技有限公司 | Intelligent data analysis and identification method and system |
| CN110837735A (en) * | 2019-11-17 | 2020-02-25 | 太原蓝知科技有限公司 | Intelligent data analysis and identification method and system |
| US11379758B2 (en) | 2019-12-06 | 2022-07-05 | International Business Machines Corporation | Automatic multilabel classification using machine learning |
| CN111143569A (en) * | 2019-12-31 | 2020-05-12 | 腾讯科技（深圳）有限公司 | Data processing method and device and computer readable storage medium |
| CN111143569B (en) * | 2019-12-31 | 2023-05-02 | 腾讯科技（深圳）有限公司 | Data processing method, device and computer readable storage medium |
| CN110781650A (en) * | 2020-01-02 | 2020-02-11 | 四川大学 | Method and system for automatically generating referee document based on deep learning |
| CN110781650B (en) * | 2020-01-02 | 2020-04-14 | 四川大学 | Method and system for automatically generating referee document based on deep learning |
| CN111540468A (en) * | 2020-04-21 | 2020-08-14 | 重庆大学 | ICD automatic coding method and system for visualization of diagnosis reason |
| CN111540468B (en) * | 2020-04-21 | 2023-05-16 | 重庆大学 | ICD automatic coding method and system for visualizing diagnostic reasons |
| CN111738303A (en) * | 2020-05-28 | 2020-10-02 | 华南理工大学 | Long-tail distribution image identification method based on hierarchical learning |
| CN111738303B (en) * | 2020-05-28 | 2023-05-23 | 华南理工大学 | Long-tail distribution image recognition method based on hierarchical learning |
| CN111723208A (en) * | 2020-06-28 | 2020-09-29 | 西南财经大学 | Conditional classification tree-based legal decision document multi-classification method and device and terminal |
| CN111723208B (en) * | 2020-06-28 | 2023-04-18 | 西南财经大学 | Conditional classification tree-based legal decision document multi-classification method and device and terminal |
| CN111930944A (en) * | 2020-08-12 | 2020-11-13 | 中国银行股份有限公司 | File label classification method and device |
| CN111930944B (en) * | 2020-08-12 | 2023-08-22 | 中国银行股份有限公司 | File label classification method and device |
| CN112464973B (en) * | 2020-08-13 | 2024-02-02 | 浙江师范大学 | Multi-label classification method based on average distance weight and value calculation |
| CN112464973A (en) * | 2020-08-13 | 2021-03-09 | 浙江师范大学 | Multi-label classification method based on average distance weight and value calculation |
| CN112016430A (en) * | 2020-08-24 | 2020-12-01 | 郑州轻工业大学 | Hierarchical action identification method for multi-mobile-phone wearing positions |
| CN112016430B (en) * | 2020-08-24 | 2022-10-11 | 郑州轻工业大学 | Hierarchical action identification method for multi-mobile-phone wearing positions |
| CN111737479A (en) * | 2020-08-28 | 2020-10-02 | 深圳追一科技有限公司 | Data acquisition method and device, electronic equipment and storage medium |
| CN111737479B (en) * | 2020-08-28 | 2020-11-17 | 深圳追一科技有限公司 | Data acquisition method and device, electronic equipment and storage medium |
| CN112182213A (en) * | 2020-09-27 | 2021-01-05 | 中润普达(十堰)大数据中心有限公司 | Modeling method based on abnormal lacrimation feature cognition |
| CN112182213B (en) * | 2020-09-27 | 2022-07-05 | 中润普达(十堰)大数据中心有限公司 | Modeling method based on abnormal lacrimation feature cognition |
| CN112131884A (en) * | 2020-10-15 | 2020-12-25 | 腾讯科技（深圳）有限公司 | Method and device for entity classification and method and device for entity presentation |
| CN112131884B (en) * | 2020-10-15 | 2024-03-15 | 腾讯科技（深圳）有限公司 | Method and device for entity classification, method and device for entity presentation |
| CN112232524A (en) * | 2020-12-14 | 2021-01-15 | 北京沃东天骏信息技术有限公司 | Multi-label information identification method and device, electronic equipment and readable storage medium |
| CN112232524B (en) * | 2020-12-14 | 2021-06-29 | 北京沃东天骏信息技术有限公司 | Multi-label information identification method and device, electronic equipment and readable storage medium |
| CN113407727A (en) * | 2021-03-22 | 2021-09-17 | 天津汇智星源信息技术有限公司 | Qualitative measure and era recommendation method based on legal knowledge graph and related equipment |
| CN113407727B (en) * | 2021-03-22 | 2023-01-13 | 天津汇智星源信息技术有限公司 | Qualitative measure and era recommendation method based on legal knowledge graph and related equipment |
| CN114117040A (en) * | 2021-11-08 | 2022-03-01 | 重庆邮电大学 | Text data multi-label classification method based on label specific features and relevance |
| CN114860892A (en) * | 2022-07-06 | 2022-08-05 | 腾讯科技（深圳）有限公司 | Hierarchical category prediction method, device, equipment and medium |
| CN114860892B (en) * | 2022-07-06 | 2022-09-06 | 腾讯科技（深圳）有限公司 | Hierarchical category prediction method, device, equipment and medium |
| CN117216688B (en) * | 2023-11-07 | 2024-01-23 | 西南科技大学 | Enterprise industry identification method and system based on hierarchical label tree and neural network |
| CN117216688A (en) * | 2023-11-07 | 2023-12-12 | 西南科技大学 | Enterprise industry identification method and system based on hierarchical label tree and neural network |
| CN118210926A (en) * | 2024-05-21 | 2024-06-18 | 山东云海国创云计算装备产业创新中心有限公司 | Text label prediction method and device, electronic equipment and storage medium |
| CN118313376A (en) * | 2024-06-07 | 2024-07-09 | 腾讯科技（深圳）有限公司 | Text processing method, device, equipment, storage medium and product |
| CN118313376B (en) * | 2024-06-07 | 2024-08-27 | 腾讯科技（深圳）有限公司 | Text processing method, device, equipment, storage medium and product |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107577785B (en) | 2020-02-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107577785B (en) | Hierarchical multi-label classification method suitable for legal identification | |
| CN110825877A (en) | Semantic similarity analysis method based on text clustering | |
| CN112732934B (en) | Power grid equipment word segmentation dictionary and fault case library construction method | |
| CN107798033B (en) | Case text classification method in public security field | |
| CN107992633A (en) | Electronic document automatic classification method and system based on keyword feature | |
| CN110807084A (en) | Attention mechanism-based patent term relationship extraction method for Bi-LSTM and keyword strategy | |
| WO2020063071A1 (en) | Sentence vector calculation method based on chi-square test, and text classification method and system | |
| CN112836029A (en) | Graph-based document retrieval method, system and related components thereof | |
| CN112579783B (en) | Short text clustering method based on Laplace atlas | |
| CN115952292B (en) | Multi-label classification method, apparatus and computer readable medium | |
| CN114265935A (en) | Science and technology project establishment management auxiliary decision-making method and system based on text mining | |
| Gao et al. | A maximal figure-of-merit (MFoM)-learning approach to robust classifier design for text categorization | |
| CN114611491A (en) | Intelligent government affair public opinion analysis research method based on text mining technology | |
| CN115186069A (en) | CNN-BiGRU-based academic text abstract automatic classification method | |
| CN116128544A (en) | Active auditing method and system for electric power marketing abnormal business data | |
| CN115292490A (en) | Analysis algorithm for policy interpretation semantics | |
| Thielmann et al. | Coherence based document clustering | |
| CN110968693A (en) | Multi-label text classification calculation method based on ensemble learning | |
| Hamdi et al. | Machine learning vs deterministic rule-based system for document stream segmentation | |
| CN117057346A (en) | Domain keyword extraction method based on weighted textRank and K-means | |
| CN117077680A (en) | Question and answer intention recognition method and device | |
| Zhang et al. | Extending associative classifier to detect helpful online reviews with uncertain classes | |
| GHAREB et al. | HYBRID STATISTICAL RULE-BASED CLASSIFIER FOR ARABIC TEXT MINING. | |
| Gao et al. | Library Similar Literature Screening System Research Based on LDA Topic Model | |
| Geng | Media bias detecting based on word embedding |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |