CN103412878B - Document theme partitioning method based on domain knowledge map community structure - Google Patents

Abstract

The invention discloses a document subject division method based on domain knowledge map community structure, which mainly solves the problem of division of document resources related to subject or domain knowledge, so as to store subject-related documents in similar logical positions and improve learning efficiency . It is characterized in that: a hierarchical community discovery algorithm based on the Fast Geedy algorithm and the GN algorithm is proposed to construct a topic structure tree; the feature extraction process uses the knowledge unit directly as a feature vector, because the knowledge unit has semantic integrity, compared with the traditional method based on word segmentation It can better reflect the topic characteristics of feature vectors; the process of calculating feature vector values proposes a method of combining degree centrality and knowledge unit document frequency, where the concept of degree centrality reflects the status of knowledge units in the overall knowledge map. Through the above method, the accuracy of document subject division is effectively improved, and it is suitable for document subject division based on knowledge map community structure in general scenarios.

Description

Document subject division method based on domain knowledge map community structure

technical field

The invention relates to the division of document topics based on the domain knowledge map community structure, and mainly solves the problem of division of document resources related to subject or domain knowledge, so as to store subject-related documents in similar logical positions and improve storage and access efficiency.

Background technique

With the expansion of the network course platform, the scale of the various subject documents of the network course continues to expand. Documents with similar topics are stored in similar logical locations. When learners learn a resource, they can prefetch other resources associated with the topic. , reduce the time overhead of reading files, and improve storage and access efficiency.

For the subject division method of documents, the following three patent documents provide different technical solutions:

1. Text classification feature selection and weight calculation method based on domain knowledge (CN101290626)

2. Based on the modified K-nearest neighbor text classification method (CN102033949A)

3. A new method and device for text classification-oriented feature vector weights (CN1719436A)

The method in Document 1 includes: (1) collecting domain text and non-domain text as training corpus and test corpus; (2) text preprocessing, including word segmentation processing and counting word frequency and document frequency; (3) selecting classification feature space and using improved (4) Select the feature space based on step (3) and expand the domain terms to the feature space; (5) Select the classification feature space, and use the improved TF-IDF algorithm to calculate the feature weight Perform calculation and adjustment; (6) Use SVM machine learning method to train text divider, construct domain text division model, and conduct experimental verification on domain text.

The method in Document 2 includes (1) text preprocessing: first, segment each document in the training text set, remove stop words, and represent the text as an item; (2) text feature selection: then reduce the dimensionality of the text vector , construct a feature function to score feature words, and select document features that are as few as possible and closely related to the topic concept of the document; (3) Text classification: Finally, use the deviation-based K-nearest neighbor text classification algorithm to construct a classifier for classification and obtain classification results .

The method in Document 3 includes: (1) collecting training corpus and test corpus by domain; (2) removing “garbage”, word segmentation, and part-of-speech tagging from webpage text; (3) extracting the vocabulary of each domain from the training corpus, and Extract the general vocabulary; (4) establish an information vocabulary with different numbers of keywords for classification according to the general vocabulary and domain vocabulary; (5) use the TF-IWF-DBV algorithm to classify the test text, and optimize to obtain the most (6) Determine the optimal number of keywords according to the classification results. Since the TF-IDF and TF-IWF methods both rely too much on word frequency, and at the same time cannot express the unbalanced distribution of vector elements between categories, Document 3 proposes a new weight calculation method (TF-IWF-DBV), In the TF-IWF method, the nth root of DBV and TF is introduced to make up for the deficiency of the method.

The methods described in the above literature mainly focus on the optimization of feature extraction methods for text classification. However, terms are still selected as feature items based on traditional word segmentation methods, and the subject characteristics of feature items are not fully considered, resulting in poor classification accuracy.

Contents of the invention

In order to solve the subject division problem of various subject documents in the existing large-scale network courses, the present invention provides a division method combining domain knowledge map community structure and document subject division, so as to divide documents with similar subjects.

To achieve the above object, the present invention is achieved by taking the following technical solutions:

A method for dividing the subject of documents based on the domain knowledge map community structure, characterized in that it comprises the following steps:

1. Domain knowledge map community structure tree construction:

(1) The domain knowledge map preprocessing process converts the domain knowledge map into a simple undirected graph, and takes the converted domain knowledge map as the root community node of the community structure tree, and adds it to the node queue CAQ to be analyzed; the community The formal representation of the node is as follows:

CNode(V _C ,Children,Parent) (1)

Among them, V _C represents the knowledge unit set contained in the community node, Children represents the child node set of the community node, and Parent represents the parent node of the community node;

(2) The domain knowledge map hierarchical community division process, the team head node CH is taken out from the CAQ, and the Fast Greedy and GN algorithms are used to divide the community of the domain knowledge map corresponding to CH or its sub-graph, and the modularity threshold is introduced If the modularity values corresponding to the community division results obtained by the above two algorithms are less than If the division is invalid, go to step (3); otherwise, compare the modularity values corresponding to the division results of the above two algorithms, select the community division result corresponding to the larger modularity value, and create a community node corresponding to each community as CH The sub-community node of , and add it to the CAQ queue;

(3) Perform step (2) on all nodes in the CAQ until the CAQ queue is empty, so as to obtain the community structure tree C-Tree corresponding to the domain knowledge map, and its formal representation is as follows:

C-Tree(CNodeSet,croot,n) (2)

Among them, CNodeSet represents the community node set of the community structure tree, croot represents the root community node of the community structure tree, and n represents the number of community nodes, that is, the number of communities existing in the network;

2. Through the community theme identification of the community structure tree corresponding to the domain knowledge map obtained in step 1, construct the domain theme structure tree, and realize the mapping from the community structure to the theme structure;

3. Document feature vector extraction:

(1) Construct a feature space, and use all knowledge units in the domain knowledge map as feature items to form a multi-dimensional feature space;

(2) The preprocessing process of the document, converting the document into a plain text form, extracting the text segment of each document, and using the TF-IDF algorithm based on the vector space model to correspond the text segment of the document to the knowledge unit ku of the domain knowledge map library Similarity matching is performed on the content of the text segment. If the similarity reaches the threshold μ, the document is considered to contain ku, and all knowledge units contained in the document are extracted accordingly;

(3) Use the formula (3) to calculate the degree centrality of the knowledge unit in the feature space in the domain knowledge map, and combine the frequency of occurrence of the knowledge unit in the document to abstract the document into the following form:

X _j ={W ₁ ,W ₂ ,...,W _i ,...,W _n }, where n represents the dimension of the feature vector, W _i represents the weight of the i-th feature item, and its formal expression is as follows:

W _i =C _deg (ku _i )*kuf(ku _i ,d) (7)

Among them, kuf(ku _i ,d) represents the frequency of knowledge unit appearing in document d, and C _deg (ku _i ) represents the degree centrality of knowledge unit ku _i ;

Fourth, the construction of the document subject division model:

(1) Construct the training data set. For each document in the given training data set D, use the method described in step 3 to extract its feature vector, combine the domain knowledge map community structure tree C-Tree in step 1 and step 2 In the domain topic structure tree T-Tree, the training data set is abstracted into the following form:

D={(X ₁ ,Y ₁ ),(X ₂ ,Y ₂ ),...,(X _j ,Y _j ),...,(X _m ,Y _m )} (8)

Among them, X _j (j=1,2,...,m) represents the feature vector of the j-th document, and Y _j (j=1,2,...,m) represents the hashtag set of the j-th document , which is formalized as follows:

Y _j ={L ₁ ,L ₂ ,...,L _i ...,L _k } (9)

Among them, m is the number of documents in the training set, and k is the number of community topics;

(2) In the training process, the BR-SVM algorithm is selected, and the cross-validation method is adopted. Based on the training document set D, the document topic division model M is obtained through training;

5. Document subject division: For the document to be divided, extract the knowledge units contained in the document, use the method of step 3 to obtain the document feature vector representation, and use the document subject division model obtained in step 4 to realize document subject division.

In the above method, the specific steps of constructing the domain subject structure tree are as follows:

(1) Community central point analysis, calculate the degree centrality of the knowledge units contained in each community node in the C-Tree in the domain knowledge map subgraph corresponding to the community, and select the node set with a large centrality as the community central node group CCNS; the calculation method of the degree centrality of knowledge units in the domain knowledge map subgraph corresponding to the community is as follows:

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; deg</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; ku</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; deg</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; ku</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; deg</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; ku</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; ku</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; KU</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

Among them, deg(ku _i ) represents the degree of the knowledge unit ku _i community, and KU represents the set of knowledge units contained in the domain knowledge map or its subgraph;

(2) For the knowledge units in CCNS, search the domain knowledge map database to obtain the core term set contained in CCNS, combine the degree centrality of the knowledge unit and the frequency of occurrence of the core term in the knowledge unit in CCNS, and calculate the centrality weight of the core term W _Central , its formal representation is as follows:

${\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; Central</span>}}^{\mathrm{<span\; class="notranslate">\; term</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; ku</span>}}^{\mathrm{<span\; class="notranslate">\; CCNS</span>}}\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; ku</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; \&delta;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; ku</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

Among them, C(ku) represents the centrality of knowledge units in CCNS, δ(term, ku) represents the frequency of terms appearing in ku, and the core term with the largest centrality weight is selected as the topic of the community;

(3) Step (2) is carried out for each community node of C-Tree, so as to construct the domain topic structure tree T-Tree, and realize the mapping from community structure to topic structure. The formal expression of T-Tree is as follows:

T-Tree(CTopicSet,troot,n) （5）

Among them, CTopicSet represents the collection of community topic nodes, troot represents the root node of the topic structure tree, and n represents the number of topics; the formal representation of community topic nodes is as follows:

CTopic(Y _C , SubTopics, PTopic) (6)

Among them, Y _C represents the label of the community topic, SubTopics represents the set of sub-nodes of the topic node, and PTopic represents the parent node of the topic node.

Compared with the prior art, the method of the present invention has the advantages that: in the process of constructing the topic structure tree, a hierarchical community discovery algorithm based on the Fast Geedy algorithm and the GN algorithm is proposed to construct the community structure tree; the feature extraction process uses the knowledge unit directly as a feature Vector, because the knowledge unit has semantic integrity, compared with the traditional method based on word segmentation, it can better reflect the topic characteristics of the feature vector; the process of calculating the value of the feature vector proposes a method that combines degree centrality and knowledge unit document frequency. The concept of degree reflects the position of knowledge unit in the overall knowledge map. Through the above improvements, compared with traditional methods, the accuracy of document subject division is effectively improved.

Description of drawings

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Fig. 1 is a flowchart of subject division of documents based on knowledge map community structure in the present invention.

Fig. 2 is a flow chart of building the subject system of domain knowledge map in Fig. 1.

Fig. 3 is a flowchart of feature vector extraction in Fig. 1 .

Detailed ways

The domain knowledge map is a complex network that describes the knowledge in a certain field (course or discipline) and the association between these knowledge; the knowledge unit refers to the basic knowledge fragment with complete expression ability in the knowledge map; the domain knowledge map library is a storage The database of knowledge units in the field records the detailed information of knowledge units, such as the name of the knowledge unit, the corresponding text segment of the knowledge unit, the core terms contained in the knowledge unit, and the relationship between the knowledge units. Usually, the knowledge map of a subject is constructed from the document resources of the subject, expressed as a network of knowledge units and their associated relationships; after using complex network community discovery algorithms to divide the domain knowledge map into community structures, each community has a relative separate subject. Therefore, the knowledge unit community structure can be used as the basis for document topic division.

The implementation process of document subject division based on the knowledge map community structure is shown in Figure 1, which can be divided into two parts: the construction of the document subject division model and the subject division of the document to be divided.

The construction of the document topic classification model is divided into three steps:

1. Construction of domain knowledge map topic system: First, the Fast Greedy algorithm is proposed (Fast Greedy algorithm is a cohesive community discovery algorithm proposed by Newman et al., each node is a community at the beginning, and then any two nodes in the network are calculated. The increment of community modularity after community aggregation, select the two communities with the largest increment to merge; the process is recursive until the modularity no longer increases) and GN algorithm (GN algorithm is a proposed by Girvan and Newman) A split community discovery algorithm, which continuously calculates the edge betweenness in the network during the execution process; each time the edge with the largest edge betweenness is selected and deleted from the network until the modularity no longer increases), the hierarchical community discovery algorithm for the domain The knowledge map is divided into communities, and the community structure tree of the domain knowledge map is obtained; each node of the community structure tree represents a community of the domain knowledge map, and the knowledge units of the same community show theme consistency; secondly, by analyzing the community center nodes (ie Some important nodes described by the degree centrality of knowledge units in the community) determine the community theme, thereby constructing the domain theme structure tree, and realizing the mapping from the community structure to the theme structure;

2. Construct the feature space and calculate the feature vector values of each dimension: use all the knowledge units of the domain knowledge map as feature items to construct the feature space; extract the knowledge units contained in the document, and combine the degree centrality of the knowledge units to calculate the features of each dimension vector value;

3. Construct the training data set and train the topic division model: construct the training data set, select the BR-SVM multi-label classification algorithm, train the training data set, and obtain the document topic division model.

The specific steps of document subject division for the document to be divided are as follows:

1. Document feature vector representation: For the document d to be divided, apply the method described in step 2 in the construction part of the document subject classification model, extract the document knowledge unit, and obtain the feature vector X _d of the document to be divided;

2. Document subject division: The feature vector X _d of the document to be divided is used as the input of the domain document subject division model M, and the output of the model is the subject label Y _d of the document. According to the relationship between Y _d and the domain topic structure tree T-Tree Corresponding relationship, get the subject division of document d.

As shown in Figure 2, the specific implementation steps of the domain knowledge map topic system construction process are as follows:

(1) The domain knowledge map preprocessing process converts the domain knowledge map into a simple undirected graph, and takes the converted domain knowledge map as the root community node of the community structure tree, and adds it to the node queue CAQ to be analyzed. The formal representation of community nodes is as follows:

CNode(V _C ,Children,Parent) (1)

Among them, V _C represents the knowledge unit set contained in the community node, Children represents the child node set of the community node, and Parent represents the parent node of the community node;

(2) The domain knowledge map hierarchical community division process, the team head node CH is taken out from the CAQ, and the Fast Greedy and GN algorithms are used to divide the community of the domain knowledge map corresponding to CH or its sub-graph, and the modularity threshold is introduced (The default value is 0.35); if the modularity values corresponding to the community division results obtained by the above two algorithms are both less than 0.35, the division is invalid, and step (3) is performed; otherwise, compare the modularity values corresponding to the division results of the above two algorithms , select the community division result corresponding to the larger modularity value, create a community node corresponding to each community, as a sub-community node of CH, and add it to the CAQ queue;

(3) Perform step (2) on all nodes in the CAQ until the CAQ queue is empty, so as to obtain the community structure tree C-Tree corresponding to the domain knowledge map, and its formal representation is as follows:

C-Tree(CNodeSet,croot,n) （2）

Among them, CNodeSet represents the community node set of the community structure tree, croot represents the root community node of the community structure tree, and n represents the number of community nodes, that is, the number of communities existing in the network;

(4) Community central point analysis, calculate the degree centrality of the knowledge units contained in each community node in the C-Tree in the domain knowledge map subgraph corresponding to the community, and select the node set with a large centrality as the community central node group CCNS; the calculation method of the degree centrality of knowledge units in the domain knowledge map subgraph corresponding to the community is as follows:

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; deg</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; ku</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; deg</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; ku</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; deg</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; ku</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; ku</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; KU</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

Among them, deg(ku _i ) represents the degree of the knowledge unit ku _i community, and KU represents the set of knowledge units contained in the domain knowledge map or its subgraph;

(5) For the knowledge units in CCNS, search the domain knowledge map library, get the core term set contained in CCNS, combine the degree centrality of knowledge units and the frequency of occurrence of core terms in CCNS knowledge units, and calculate the centrality weight of core terms W _Central , its formal representation is as follows:

${\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; Central</span>}}^{\mathrm{<span\; class="notranslate">\; term</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; ku</span>}}^{\mathrm{<span\; class="notranslate">\; CCNS</span>}}\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; ku</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; \&delta;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; ku</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

Among them, C(ku) represents the centrality of knowledge units in CCNS, and δ(term, ku) represents the frequency of term appearing in ku. Select the core term with the largest centrality weight as the topic of the community;

(6) Step (2) is performed for each community node of C-Tree, so as to construct the domain topic structure tree T-Tree, and realize the mapping from community structure to topic structure. The formal expression of T-Tree is as follows:

T-Tree(CTopicSet,troot,n) （5）

Among them, CTopicSet represents the collection of community topic nodes, troot represents the root node of the topic structure tree, and n represents the number of topics. The formal representation of community topic nodes is as follows:

CTopic(Y _C , SubTopics, PTopic) (6)

Among them, Y _C represents the label of the community topic, SubTopics represents the set of sub-nodes of the topic node, and PTopic represents the parent node of the topic node.

As shown in Figure 3, the specific implementation steps of constructing the feature space and calculating the feature vector values of each dimension are as follows:

(1) Construct a feature space, using all knowledge units in the domain knowledge map as feature items to form a multi-dimensional (each knowledge unit is a dimension) feature space;

(2) The preprocessing process of the document, converting the document into a plain text form (ie txt file), extracting the text segment of each document, using the TF-IDF algorithm based on the vector space model (TF-IDF algorithm based on the vector space model Use the TF-IDF algorithm to represent the text as a feature vector form with terms as feature items, and use the cosine of the angle between the vectors to represent the similarity between documents) The text segment of the document corresponds to the knowledge unit ku of the domain knowledge map library The content of the text segment is matched by similarity. If the similarity reaches the threshold μ (the default value is 0.8), the document is considered to contain ku, and all knowledge units contained in the document are extracted accordingly;

(3) Calculate the degree centrality of the knowledge unit in the feature space in the domain knowledge map (see formula (3) for the calculation method), combine the frequency of occurrence of the knowledge unit in the document, and abstract the document into the following form: X _j ={W ₁ ,W ₂ ,...,W _i ,...,W _n }, where n represents the dimension of the feature vector, W _i represents the weight of the i-th feature item, and its formal expression is as follows:

W _i =C _deg (ku _i )*kuf(ku _i ,d) (7)

Among them, kuf(ku _i ,d) represents the frequency of the knowledge unit appearing in document d, and C _deg (ku _i ) represents the degree centrality of the knowledge unit ku _i .

The specific steps for constructing a training data set and training a topic segmentation model include:

(1) Construct the training data set. For each document in the given training data set D, use the method described in step 4 to extract its feature vector, combine the community structure tree C-Tree of the domain knowledge map and the domain topic structure tree T- Tree, which abstracts the training data set into the following form:

D={(X ₁ ,Y ₁ ),(X ₂ ,Y ₂ ),...,(X _j ,Y _j ),...,(X _m ,Y _m )} (8)

Among them, X _j (j=1,2,...,m) represents the feature vector of the j-th document, and Y _j (j=1,2,...,m) represents the hashtag set of the j-th document , which is formalized as follows:

Y _j ={L ₁ ,L ₂ ,...,L _i ...,L _k } (9)

Among them, m is the number of documents in the training set, and k is the number of community topics;

(2) Select the BR-SVM algorithm in the training process (the BR-SVM method adopts the "one-to-many" strategy to convert multi-label problems into multiple binary classification problems, and uses the mature binary classification problem training method SVM to solve this series of binary classification problems. training), using the cross-validation method, based on the training document set D, the document topic division model M is trained.

Claims (1)

1. A method for dividing the subject of documents based on domain knowledge map community structure, characterized in that it comprises the following steps: 1. Domain knowledge map community structure tree construction: (1) The domain knowledge map preprocessing process, transforming the domain knowledge map into a simple undirected graph, and taking the converted domain knowledge map as the root community node of the community structure tree, adding it to the node queue CAQ to be analyzed; community The formal representation of the node is as follows: CNode(V _C ,Children,Parent) (1) Among them, V _C represents the knowledge unit set contained in the community node, Children represents the child node set of the community node, and Parent represents the parent node of the community node; (2) The domain knowledge map hierarchical community division process, take out the team leader node CH from the CAQ, use the Fast Greedy and GN algorithms to divide the community of the domain knowledge map or its subgraphs corresponding to CH, and introduce the modularity threshold ; If the modularity values corresponding to the community division results obtained by the above two algorithms are less than , then the division is invalid, and go to step (3); otherwise, compare the modularity values corresponding to the division results of the above two algorithms, select the community division result corresponding to the larger modularity value, and create a community node corresponding to each community, as CH's sub-community nodes, and add them to the CAQ queue; (3) Perform step (2) on all nodes in the CAQ until the CAQ queue is empty, so as to obtain the community structure tree C-Tree corresponding to the domain knowledge map, and its formal expression is as follows: C-Tree(CNodeSet,croot,n) (2) Among them, CNodeSet represents the community node set of the community structure tree, croot represents the root community node of the community structure tree, and n represents the number of community nodes, that is, the number of communities existing in the network; 2. Through the community theme identification of the community structure tree corresponding to the domain knowledge map obtained in step 1, construct the domain theme structure tree, and realize the mapping from the community structure to the theme structure; 3. Document feature vector extraction: (1) Construct a feature space, and use all knowledge units in the domain knowledge map as feature items to form a multi-dimensional feature space; (2) The preprocessing process of the document, converting the document into a plain text form, extracting the text segment of each document, and using the TF-IDF algorithm based on the vector space model to correspond the text segment of the document to the knowledge unit ku of the domain knowledge map library Similarity matching is performed on the content of the text segment. If the similarity reaches the threshold μ, the document is considered to contain ku, and all knowledge units contained in the document are extracted accordingly; (3) Use the formula (3) to calculate the degree centrality of the knowledge unit in the feature space in the domain knowledge map, and combine the frequency of occurrence of the knowledge unit in the document to abstract the document into the following form: X _j ＝{W ₁ ,W ₂ ,...,W _i ,...,W _n }, where n represents the dimension of the feature vector, W _i represents the weight of the i-th feature item, and its formal expression is as follows: W _i ＝C _deg (ku _i ) ^* kuf(ku _i ,d) (7) Among them, kuf(ku _i ,d) represents the frequency of knowledge unit appearing in document d, and C _deg (ku _i ) represents the degree centrality of knowledge unit ku _i ; The expression of formula (3) is: ${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; deg</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; ku</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; deg</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; ku</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; deg</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; ku</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; ku</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; KU</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$ Among them, deg(ku _i ) represents the degree of the knowledge unit ku _i community, and KU represents the set of knowledge units contained in the domain knowledge map or its subgraph; Fourth, the construction of the document subject division model: (1) Construct a training data set. For each document in a given training data set D, use the method described in step 3 to extract its feature vector, combine the domain knowledge map community structure tree C-Tree in step 1 and step 2 In the domain topic structure tree T-Tree, the training data set is abstracted into the following form: D＝{(X ₁ ,Y ₁ ),(X ₂ ,Y ₂ ),...,(X _j ,Y _j ),...,(X _m ,Y _m )} (8) Among them, X _j (j=1,2,...,m) represents the feature vector of the jth document, and Y _j (j=1,2,...,m) represents the hashtag set of the jth document , which is formalized as follows: Y _j ＝{L ₁ ,L ₂ ,...,L _i ...,L _k } (9) Among them, m is the number of documents in the training set, and k is the number of community topics; (2) The BR-SVM algorithm is selected for the training process, and the cross-validation method is used to obtain the document topic division model M based on the training document set D; 5. Document subject division: For the document to be divided, extract the knowledge units contained in the document, use the method of step 3 to obtain the document feature vector representation, and use the document subject division model obtained in step 4 to realize document subject division; In the above steps, the specific method of constructing the domain subject structure tree in step 2 is as follows: (1) Community central point analysis, calculate the degree centrality of the knowledge units contained in each community node in the C-Tree in the domain knowledge map subgraph corresponding to the community, and select the node set with a large centrality as the community central node group CCNS; the calculation of the degree centrality of the knowledge unit in the domain knowledge map subgraph corresponding to the community is carried out according to formula (3); (2) For the knowledge units in CCNS, search the domain knowledge map library, get the core term set contained in CCNS, combine the degree centrality of the knowledge unit and the frequency of the core term in the knowledge unit in CCNS, and calculate the centrality of the core term term The weight W _Central , its formal expression is as follows: ${\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; Central</span>}}^{\mathrm{<span\; class="notranslate">\; term</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; ku</span>}}^{\mathrm{<span\; class="notranslate">\; CCNS</span>}}\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; ku</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; \&delta;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; term</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; ku</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$ Among them, C(ku) represents the centrality of knowledge units in CCNS, δ(term, ku) represents the frequency of terms appearing in ku, and the core term with the largest centrality weight is selected as the topic of the community; (3) Step (2) is performed for each community node of the C-Tree, thereby constructing a domain topic structure tree T-Tree, and realizing the mapping from the community structure to the topic structure. The T-Tree is formally expressed as follows: T-Tree(CTopicSet,troot,n) (5) Among them, CTopicSet represents the collection of community topic nodes, troot represents the root node of the topic structure tree, and n represents the number of topics; the formal representation of community topic nodes is as follows: CTopic(Y _C ,SubTopics,PTopic) (6) Among them, Y _C represents the label of the community topic, SubTopics represents the set of sub-nodes of the topic node, and PTopic represents the parent node of the topic node.