CN110704638A - A Construction Method of Electric Power Text Dictionary Based on Clustering Algorithm - Google Patents

Abstract

The invention relates to a clustering algorithm-based electric power text dictionary construction method, which mainly comprises four parts: the system comprises a data classification preprocessor, a data word segmentation processor, a clustering processor and a data processing operation core. The invention provides a clustering algorithm-based electric power text dictionary construction method, which is a strategic method and is mainly used for constructing a dictionary in the electric power field text classification process. Through the model in the invention, the key phrases which can represent the text types in the text in the power field can be more accurately found, and the construction of the dictionary is carried out by utilizing the key phrases.

Description

A Construction Method of Electric Power Text Dictionary Based on Clustering Algorithm

technical field

The invention relates to the field of power system data processing, in particular to a method for constructing a power text dictionary based on a clustering algorithm, which is mainly used for text data processing in the power field.

Background technique

Power grid enterprises are asset-intensive enterprises, and the management of the health status of power equipment is its core task, and the use of big data for scientific management is an inevitable trend. However, it is generally believed that power grid data is difficult to use due to its large volume, many types, low value density and rapid changes. Among them, the data value density is low, which means that most of the data is the normal data of the power grid, and there is only a very small amount of abnormal data. The serious skewness of the data affects the mining effect based on artificial intelligence methods such as machine learning and deep learning. Fortunately, there are many types of power data, among which text data has the characteristics of high value density because "important things are often recorded", and the mining prospect is good. Therefore, power text mining is one of the key technologies that focus on the health management of power equipment. . Some of the existing data mining for the power grid is all about the research and application of the structured data in the power grid, while the research on the text direction of the unstructured data in the power grid is basically seldom studied. There are almost zero research reports on text processing. For the time being, there are no technical ways and solutions to obtain power text information, and it is impossible to build a detailed power corpus. Therefore, it is necessary to construct a dictionary of power grid related fields.

In the process of equipment operation and maintenance management, power grid enterprises will record information such as enemy obstacles, defects, maintenance, and elimination of equipment in Chinese. This information will be stored in the information management system in the form of text, which not only reflects the past history of the individual health status of the power equipment, but also contains a wealth of technology for the reliability of similar equipment. Chinese text classification has always been regarded as an important and difficult technology, especially when it is applied to various professional fields, it needs to be closely combined with professional domain knowledge, which is even more difficult. Nowadays, various fields are developing rapidly, and new words, new concepts, and new relationships are constantly emerging. If we still stay in traditional lexical analysis, it is far from meeting our needs. However, the emergence of domain dictionaries can largely To solve this problem, by building a dictionary, collecting the latest concepts and interrelationships, research in a specific field can be done with ease.

The dictionary construction mainly considers two aspects: (1) How to solve the problem that the words in the power grid data text are highly specialized, which leads to the difficulty of dictionary construction. (2) There are many texts in the electric power field that do not strictly conform to Chinese grammar, and there are many irregular formats in the texts, which bring difficulties to text processing and semantic parsing in the electric power field.

SUMMARY OF THE INVENTION

In order to solve the above technical problems, the present invention provides a power text dictionary construction method based on a clustering algorithm to solve the problem of power system text dictionary construction. This mechanism is a strategic method. By using this method, the power system can be The text dictionary is more complete and correct, improving the effect of subsequent text processing.

The method for constructing a power text dictionary based on a clustering algorithm according to the present invention adopts the technical scheme as follows: the equipment used in the power text dictionary construction method includes a data classification preprocessor, a data word segmentation processor, and a clustering processor. processor, data processing operation core;

The steps of constructing the power text dictionary are:

Step 1: Use the relevant documents in the electric power field to create the electric power field corpus that needs to be processed, prepare to process the text in the electric power field corpus, and go to step 2;

Step 2: Preprocess the text to be processed, delete some words that do not affect the semantics of the text according to the word list to remove stop words, and go to Step 3;

Step 3: Use a general dictionary for word segmentation on the preprocessed text in Step 2 to obtain a batch of good words, and go to Step 4;

Step 4: Use the tf-idf algorithm to find some keywords that can represent the text for the text after the word segmentation in step 3, and go to step 5;

Step 5: Use the word2vec model to construct the word vector for the keywords obtained in step 4, and go to step 6;

Step 6: Use the k-meas clustering algorithm to cluster the constructed word vectors, and go to Step 7;

Step 7: Select k word vectors constructed by the word2vec model in the text as the clustering centers (μ ₁ , μ ₂ ,...μ _k-1 , μ _k ), and go to step 8;

Step 8: Calculate the cosine distance from each word vector to k word vectors constructed using the word2vec model, and go to Step 9;

Step 9: The word vector is classified into the k clusters with the smallest cosine distance, and the mean value of the data points in each cluster after division is calculated, and this value is used as the new cluster center;

Step 10: If the cluster center no longer changes or the maximum number of iterations is reached, the algorithm stops and goes to step 11;

Step 11: Check whether the keywords obtained by the clustering reach a predetermined threshold, and use the words that reach the threshold as keywords, and discard the words that do not reach the threshold, and go to step 12;

Step 12: Use step 4 and step 11 to obtain relevant keywords to construct a dictionary, and enter step 13;

Step 13: End.

Further, the data classification preprocessor performs text preprocessing on the test text to be classified according to the electric power field corpus and the removal stop word list, and removes some meaningless words and numerical symbols in the text.

Further, the stop word list includes words, numbers and symbols that often appear in the text without actual meaning.

Further, the method for establishing the stop word list is to establish a data statistics knowledge rule base, whether to fill in a certain number or symbol into the stop word list to set a threshold, and compare with this threshold to confirm whether to use the text in the text. Some numbers and symbols are added to the stop word list.

Further, the data word segmentation processor performs word segmentation on the preprocessed text as follows:

(1) Use a general dictionary to perform word segmentation on the preprocessed text, and after word segmentation, perform vectorized representation for each word;

其中a为该词在文本中出现的次数，b文本的总词数，c为电力领域语料库的文档总数，e为包含该词的文档数，分母加1是为了避免分母为0的情况出现，计算该词tf×idf的值，选择计算结果最大的一些词语作为关键词；(2) Feature selection for a large number of word vectors, using the tf-idf algorithm,

where a is the number of times the word appears in the text, b is the total number of words in the text, c is the total number of documents in the power field corpus, e is the number of documents containing the word, and the denominator is increased by 1 to avoid the denominator being 0. Calculate the value of the word tf × idf, and select some words with the largest calculation results as keywords;

(3) Calculate the word vector of the keyword obtained in (2) using the word2vec model.

Further, in step (3), the word2vec model is used as the skip-grim model.

Further, the clustering processor uses the k-meas algorithm to cluster the word vectors obtained by the word2vec algorithm to obtain a batch of new keywords, and uses a preset threshold to remove unreasonable ones obtained by clustering. Keywords, a dictionary is constructed using the keywords clustered above the threshold and the keywords originally derived using the tf-idf algorithm.

Further, the data processing operation core includes all specific operations required for data processing after feature selection of the data.

The beneficial effects of the present invention are as follows: the present invention proposes a method for constructing a power text dictionary based on a clustering algorithm, which is a strategic method, and is mainly used for the construction process of the dictionary in the text classification process in the power field. The method based on clustering is used to solve the problem of difficult word segmentation and difficult word segmentation to construct a good power domain dictionary due to its strong text specialization. Through the model in the present invention, the keyword group that can represent the text category in the text in the electric power field can be found more accurately, and the dictionary can be constructed by using it.

Description of drawings

In order to make the content of the present invention easier to understand clearly, the present invention will be described in further detail below according to specific embodiments and in conjunction with the accompanying drawings.

Figure 1 is a schematic diagram of the system structure.

Figure 2 is a schematic flow chart of the method of the present invention.

Detailed ways

As shown in Figures 1 and 2, a method for constructing a power text dictionary based on a clustering algorithm according to the present invention is characterized in that the equipment used in the method for constructing a power text dictionary includes a data classification preprocessor, a data word segmentation process processor, clustering processor, data processing operation core;

The steps of constructing the power text dictionary are:

Step 1: Use the relevant documents in the electric power field to create the electric power field corpus that needs to be processed, prepare to process the text in the electric power field corpus, and go to step 2;

Step 2: Preprocess the text to be processed, delete some words that do not affect the semantics of the text according to the word list to remove stop words, and go to Step 3;

Step 3: Use a general dictionary for word segmentation on the preprocessed text in Step 2 to obtain a batch of good words, and go to Step 4;

Step 4: Use the tf-idf algorithm to find some keywords that can represent the text for the text after the word segmentation in step 3, and go to step 5;

Step 5: Use the word2vec model to construct the word vector for the keywords obtained in step 4, and go to step 6;

Step 6: Use the k-meas clustering algorithm to cluster the constructed word vectors, and go to Step 7;

Step 7: Select k word vectors constructed by the word2vec model in the text as the clustering centers (μ ₁ , μ ₂ ,...μ _k-1 , μ _k ), and go to step 8;

Step 8: Calculate the cosine distance from each word vector to k word vectors constructed using the word2vec model, and go to Step 9;

Step 9: The word vector is classified into the k clusters with the smallest cosine distance, and the mean value of the data points in each cluster after division is calculated, and this value is used as the new cluster center;

Step 10: If the cluster center no longer changes or the maximum number of iterations is reached, the algorithm stops and goes to step 11;

Step 11: Check whether the keywords obtained by the clustering reach a predetermined threshold, and use the words that reach the threshold as keywords, and discard the words that do not reach the threshold, and go to step 12;

Step 12: Use step 4 and step 11 to obtain relevant keywords to construct a dictionary, and enter step 13;

Step 13: End.

Data classification preprocessor, mainly used in the preprocessing of data and training datasets in the process of text classification, text preprocessing is a necessary stage to convert semi-structured or unstructured text into an appropriate text representation . Usually, the special characters, punctuation marks, numbers and other characters that do not contain any information that appear in the text are deleted first. However, due to the particularity of the electric field, the text generally contains a large number of numbers and symbols. In the process of preprocessing, special treatment should be performed for this part, and the valid numbers and symbols in its text should be preserved.

In the text classification, it is necessary to remove those common words in the text, where the common words refer to those words that often appear in the text, such as 'a', 'the' in English, '的', 'ah' in Chinese ', as well as some numbers and symbols, these words can not bring any help to the classification, are collected into a set called "stop word list", in the process of text preprocessing, the words contained in the text should be stopped Words are deleted, but due to the particularity of the field of electricity, its text must contain a large number of numbers and symbols. However, with the different application environments of text classification, stop words are often not limited to the words in the stop word list, because this method is to solve the text related to the power field, so in this method, a data statistical knowledge rule is established. Library, whether to fill a certain number or symbol into the stop word list, set a threshold, and confirm whether to add some numbers and symbols in the text to the "stop word list" by comparing with this threshold. Removing stop words can greatly increase the performance of text classification.

Since most of the documents in the electric power field are documents such as equipment status and equipment maintenance, they are mostly short documents. The preprocessed text needs to be processed by text segmentation, and the particularity of the electric power field determines that the text in this field will inevitably have For many highly specialized texts, we need to use the data segmentation processor to segment these texts to solve the problem of highly specialized texts.

Word segmentation is a very important part in the process of text classification. The function of word segmentation is to segment the text through the existing word segmentation tools in the existing text, so that a series of good words will be obtained, which we call them. participle set.

The present application first uses a data word segmentation processor to perform word segmentation processing on the preprocessed short text. After the word segmentation processing, a series of a large number of words will be obtained. We also use the data word segmentation processor to first use the statistical model (ie the tf-idf algorithm) to select a feature. At this time, we will get some words that can represent the text, that is, keywords. However, due to the particularity of the power field, it may be Some words with the same semantic meaning as the keywords will be missed. This application uses the word2vec algorithm to calculate the word vectors for the above keywords.

The method for the word segmentation processor to perform word segmentation on the preprocessed text is as follows:

(1) Use a general dictionary to perform word segmentation on the preprocessed text, and after word segmentation, perform vectorized representation for each word;

(2) Feature selection for a large number of word vectors, using the tf-idf algorithm, where a is the number of times the word appears in the text, b is the total number of words in the text, c is the total number of documents in the power field corpus, e is the number of documents containing the word, and the denominator is increased by 1 to avoid the denominator being 0. Calculate the value of the word tf × idf, and select some words with the largest calculation results as keywords;

(3) Use the word2vec model to calculate the word vector of the keywords obtained in (2); word2vec is an algorithm that converts words into vector form and calculates the similarity in vector space to represent the semantic similarity of text. In the embodiment of this application, we use the skip-grim model in the word2vec algorithm, which uses a word as an input to predict the context around it. The essence of this model is to find u _x ^T v _c (that is, the similarity of two words), we use v _c to represent the word vector of the target word, u _x to represent the word vector of the xth word except the target word, where v _c =Ww _c , W represents the matrix of the target word, W is a d×V matrix, where V represents the number of all words, d represents the dimension of the target word, and w _c represents the one-hot vector of the target word.

The professionalism of the words separated by the data word segmentation processor may be guaranteed, but because the results of word segmentation processing are limited after all, clustering is used to process the word vectors obtained by clustering. , in order to get more professional vocabulary and make corresponding preparations for the following construction dictionary.

After a series of keywords are obtained by the data word segmentation processor, and their word vectors are obtained by the word2vec algorithm, we use the word vectors to cluster the words, and use the k-meas clustering algorithm to cluster the word vectors. Obtain a series of new keywords, use a preset threshold to remove the unreasonable keywords obtained by clustering, use the keywords obtained by clustering above the threshold and the keywords originally obtained by the tf-idf algorithm to construct a dictionary.

The core of the data processing operation includes all the specific operations required for data processing after the feature selection of the data. The addition of other parts in the present invention does not affect the processing of the data itself, but only ensures that the data processing can be more smoothly and effectively. conduct.

For the convenience of description, the following application example is taken as an example:

Now a power company hopes to conduct a data analysis on a series of texts about customer complaints and customer repairs recorded in the previous company, to mine users' needs and improve users' evaluation of the company, and at the same time hope to improve the user's experience. At this time, use The effect of traditional classification on data mining of complaints and maintenance texts in the power field is definitely not satisfactory.

At this time, we can first use the method proposed in this patent to construct a dictionary for the company's electric power enterprise complaint text and maintenance text, and then use the dictionary to mine the text data.

Its specific implementation is:

(1) Perform text preprocessing through the text to be processed, that is, stop word removal processing of the text, and then perform word segmentation processing on the text.

(2) For the above words after preprocessing and word segmentation, use tf-idf to perform feature selection to select keywords of their texts.

(3) The word2vec algorithm is used to construct the word vector for the keywords in (2).

(4) Using the word vector constructed in (3), the k-means algorithm is used to cluster it, and a series of new keywords are obtained by clustering

(5) Use the keywords obtained in (2) and (4) as root words to construct a related dictionary.

The above descriptions are only the preferred solutions of the present invention, and are not intended to further limit the present invention, and all equivalent changes made by using the contents of the description and drawings of the present invention are within the protection scope of the present invention.

Claims (8)

1. a power text dictionary construction method based on clustering algorithm, is characterized in that, the equipment that described power text dictionary construction method uses comprises data classification preprocessor, data word segmentation processor, clustering processor, data processing operation core ; The steps of constructing the power text dictionary are: Step 1: Use the relevant documents in the electric power field to create the electric power field corpus that needs to be processed, prepare to process the text in the electric power field corpus, and go to step 2; Step 2: Preprocess the text to be processed, delete some words that do not affect the semantics of the text according to the word list to remove stop words, and go to Step 3; Step 3: Use a general dictionary for word segmentation on the preprocessed text in Step 2 to obtain a batch of good words, and go to Step 4; Step 4: Use the tf-idf algorithm to find some keywords that can represent the text for the text after the word segmentation in step 3, and go to step 5; Step 5: Use the word2vec model to construct the word vector for the keywords obtained in step 4, and go to step 6; Step 6: Use the k-meas clustering algorithm to cluster the constructed word vectors, and go to Step 7; Step 7: Select k word vectors constructed by the word2vec model in the text as the clustering centers (μ ₁ , μ ₂ ,...μ _k-1 , μ _k ), and go to step 8; Step 8: Calculate the cosine distance from each word vector to k word vectors constructed using the word2vec model, and go to Step 9; Step 9: The word vector is classified into the k clusters with the smallest cosine distance, and the mean value of the data points in each cluster after division is calculated, and this value is used as the new cluster center; Step 10: If the cluster center no longer changes or the maximum number of iterations is reached, the algorithm stops and goes to step 11; Step 11: Check whether the keywords obtained by the clustering reach a predetermined threshold, and use the words that reach the threshold as keywords, and discard the words that do not reach the threshold, and go to step 12; Step 12: Use step 4 and step 11 to obtain relevant keywords to construct a dictionary, and enter step 13; Step 13: End. 2. a kind of power text dictionary construction method based on clustering algorithm according to claim 1, is characterized in that, described data classification preprocessor, according to electric power domain corpus and removing stop word list, the test to be classified The text is preprocessed to remove some meaningless words and numerical symbols from the text. 3 . The method for constructing a power text dictionary based on a clustering algorithm according to claim 1 , wherein the stop word list contains meaningless words, numbers and symbols that often appear in the text. 4 . 4. The method for constructing a power text dictionary based on a clustering algorithm according to claim 1, wherein the method for establishing the stop word list is to establish a data statistical knowledge rule base, whether to use a certain number or Fill in the stop word list with symbols to set a threshold, and compare with this threshold to confirm whether to add some numbers and symbols in the text to the stop word list. 5. a kind of power text dictionary construction method based on clustering algorithm according to claim 1, is characterized in that, described data word segmentation processor, the method that word segmentation has been carried out on the preprocessed text is: (1) Use a general dictionary to perform word segmentation on the preprocessed text, and after word segmentation, perform vectorized representation for each word; (2) Feature selection for a large number of word vectors, using the tf-idf algorithm,

where a is the number of times the word appears in the text, b is the total number of words in the text, c is the total number of documents in the power field corpus, e is the number of documents containing the word, and the denominator is increased by 1 to avoid the denominator being 0. Calculate the value of the word tf × idf, and select some words with the largest calculation results as keywords; (3) Calculate the word vector of the keyword obtained in (2) using the word2vec model. 6 . The method for constructing a power text dictionary based on a clustering algorithm according to claim 5 , wherein the word2vec model used in (3) is the skip-grim model. 7 . 7 . The method for constructing a power text dictionary based on a clustering algorithm according to claim 1 , wherein the clustering processor performs a clustering process on the word vector obtained by the word2vec algorithm by using the k-meas algorithm. 8 . , get a batch of new keywords, use the preset threshold to remove the unreasonable keywords obtained by clustering, use the keywords obtained by clustering above the threshold and the keywords originally obtained by the tf-idf algorithm to construct a dictionary. 8 . The method for constructing a power text dictionary based on a clustering algorithm according to claim 1 , wherein the data processing operation core includes all specific operations required for data processing after feature selection of the data. 9 . .

Images