TW201516713A - File classification method based on group characteristic values - Google Patents

Abstract

The present invention is a file classification method based on group characteristic values. The method performs a file pretreatment for text normalization and stopwords filtration, calculates the number of each text appeared in the file by means of a pre-built Chinese character library, Chinese thesaurus library and English character library, separately calculates the number of each text not appeared in the pre-built character or thesaurus libraries, and retrieves the greater number of occurrences of the text as a key word. After taking the characteristics from two distinct sources and giving them different weights in accordance with their importance, the degree of similarity compared to group characteristic values according to the preset similarity formula is calculated. If it exceeds a threshold value, it means that a similar file is detected and attributes to the same classification for assisting file classification.

Description

File classification method based on group eigenvalue

The invention relates to a file classification method; in particular, it relates to a file classification method based on group feature values, which is used for classifying files, and can be matched to a data protection system, a paper plagiarism system, and the like. In today's increasingly important data security situation, the use of content analysis for file classification, the detection of the existence of confidential documents, to avoid the outflow of confidential documents, and make the data protection system's confidential protection become one of the main technical areas.

In the present invention, the technical field mentioned mainly includes file classification and its extended confidentiality protection. In terms of file classification, in order to complete the above system and solve related problems, the commonly used method is cluster analysis, which is mainly divided into top-down divisive clustering and bottom-up. Agglomerative clustering.

The top-down segmentation method classifies all files as a whole and then divides them down. However, the problem is that the segmentation method must decide in advance that all documents should be divided into several segments. This is not in line with our needs when we cannot know in advance how many types and quantities of documents.

As for the part of the cohesion method, the distance between any two files must be calculated first, and then merged into a larger group according to the related data, until all the files are in the same group, or the group and The similarity between groups is lower than a specified threshold. However, to complete this complete process, it takes a lot of time and memory, which is insufficient in the practical part.

In data clustering, the key step is to choose the way to calculate the distance. The current common methods can be roughly divided into the following types:

Manhattan Distance

Euclidean Distance

Hamming Distance

Cosine Similarity

The Manhattan distance represents the sum of the absolute wheelbases on the standard coordinate system at two points; the Euclidean distance represents the linear distance between the two points in the Euclidean space; the Hamming distance refers to the two words. The metastring corresponds to the number of different position characters. Although the above method can obtain the corresponding distance between files, since the feature values of the file are mainly multi-dimensional, and the numerical values vary greatly, the above distance calculation method is not suitable for file classification.

Cosine similarity refers to measuring the similarity between the cosines of the angles of the product spaces in two vectors. This cosine similarity can be used not only in the vector comparison of any dimension, but also in the high-dimensional positive space. . Therefore, in the preferred embodiment, we use cosine similarity as the way of distance calculation to achieve the purpose of file classification.

In the step of calculating the distance, the process of taking words and calculating the frequency of words is generally used. There are two common ways of this process, one is that there is no In the case of pre-built materials, the content of the document is used as the basis for word-taking, and the other is based on the pre-built word bank as a reference for word-taking.

In the absence of pre-built materials and the content of the documents as the basis for the wording, although the text keywords can be obtained, the time complexity is high, and the time-consuming and long-term situation is easy to occur. How the word should be applied to the similarity calculation stage is also an inconsistible problem.

Nowadays, if the pre-built word library is used as the basis for word retrieval, although the problem of high time complexity is solved, relatively speaking, the frequency of the word can only be calculated according to the content of the pre-built word bank, and the article may not be obtained. Keyword words lead to misjudgments and other conditions.

It can be seen that there are still many shortcomings in the above-mentioned methods of use, which is not a good design, but needs to be improved.

In view of the shortcomings derived from the above-mentioned conventional methods, the inventor of the present invention has improved and innovated, and after years of painstaking research, he finally succeeded in researching and developing this invention.

In order to solve the above situation, the patent proposes a method of combining the advantages of the two methods, so that the content keyword and the pre-built word library content can be obtained as a match under low-time condition, thereby generating higher accuracy. The result of the operation.

In this patent, when the similarity exceeds a certain threshold, it is regarded as similar files belonging to the same kind. In this way, the computational complexity between a large number of files is saved, so that we can improve the system practicability and complete the file classification in a more efficient manner. Work.

The purpose of the patent of the present invention is to establish a file classification method based on group feature values. Through the document classification method proposed in this patent, the accuracy and efficiency of the current related document classification system are strengthened, the computational complexity between a large number of files is saved, the system practicability can be improved, and the file classification work can be completed in a more efficient manner, and then Use this patent to develop a data protection system to prevent the flow of confidential files.

A document classification method based on group eigenvalues for achieving the above object is used to calculate the similarity between a document and a group classified document, and use this as a basis for document classification. The content analysis method firstly processes the word normalization for a file, so that the upper and lower case of the file is unified, and deletes various punctuation marks, and then filters the disabled words, and the virtual word, the connected word, and the meaningless word are deleted. Or eliminate the need for comparison vocabulary. At this point, the behavior of word-breaking words can be used. The pre-built Chinese character library, Chinese lexicon and English font are used to calculate the number of occurrences of each word in the file, while those not appearing in the pre-built word database are keyword words. In the name of the other, the number of occurrences is calculated separately: it takes too long to prevent the key Chinese words from being taken out, and the pre-built Chinese character library is also used as an auxiliary for screening. After obtaining the frequency of occurrence of the pre-built word bank and the keyword word, a vector information is established with its frequency, and the weight of the keyword word is increased according to the importance, and the cosine similarity calculation file is used to calculate the similarity of the feature values of each classification group. Degree, if its value is greater than 1, it is regarded as 1, and if the value is higher than a threshold, it shall be deemed to be classified into the category.

A method for classifying a file based on a population feature value according to the present invention may comprise at least (1) a step of extracting a feature of a file, performing pre-processing on the file, sequentially normalizing the word, breaking the word, and establishing a stop word. The library filters the action of stopping words; (2) according to the strategy of acquiring words, obtains the number of occurrences of pre-built Chinese thesaurus, pre-built English fonts, key Chinese words, key English words; (3) extracting file feature vectors a step of creating a file vector information according to the frequency of occurrence of each word, There are pre-built Chinese thesaurus, pre-built English fonts, high-frequency key Chinese words, high-frequency key English words; (4) file classification steps based on the file vector, calculate the similarity between the files to be classified and the group feature values And (5) after repeating the foregoing process to obtain similarity with all the classifications, classify the documents to be classified into the categories to be attributed.

Further, the process of extracting file features may further include: (1) pre-processing the file, first performing word normalization, wherein the normalizing action includes unifying the English capitalization of the file to be analyzed, And deleting various punctuation marks; (2) secondly performing word breaks, and establishing a stop word library to filter the action of deactivating words, wherein the stop words include virtual words, connected words, meaningless words or no comparison The vocabulary; and (3) the number of occurrences of the pre-built Chinese thesaurus, pre-built English fonts, key Chinese words, and key English words. The process of extracting the file feature vector is to use the pre-built Chinese character library and Chinese vocabulary. And the English font calculates the number of occurrences of each word in the document, while those who do not appear in the pre-built word library are referred to as key Chinese words and key English words.

In addition, the file classification process may be: (1) the extraction of the group feature value includes the keyword word frequencies of all the files belonging to the category, and the average of the file is obtained by averaging a specific classification file vector mean; Secondly, the similarity calculation is performed by the file vector, and the calculation method can be the cosine similarity method; (3) after obtaining the similarity with all the classifications, if all the values do not reach a specified threshold, then the file to be classified is returned. For the new classification; otherwise, classify it into the category with the highest similarity and complete the file classification work.

The invention provides a wireless and its backhaul network equipment cooperative operation energy-saving system and method, and when compared with other conventional technologies, the following advantages are obtained:

1. A file classification method based on group eigenvalues of the present invention adopts a statistical analysis method for research and development, and can process unstructured texts, which are not affected by word order.

2. A file classification method based on group eigenvalues of the present invention proposes the concept of using a pre-built word library and a semantic keyword word to perform word-breaking words, combining the advantages of the two methods, so that it can be used at low time consumption. In the case of the situation, the content keyword of the text is matched with the content of the pre-built word library, and different weights are given according to the importance thereof, thereby generating a calculation result with higher accuracy.

3. The document classification method based on the group feature value of the present invention cooperates with the concept of the keyword word of the file to be classified, thereby obtaining a representative feature, not only expanding the range of words of the known classification after the classification is completed, but also It also improves the accuracy of document classification.

110‧‧‧ formalization of words

120‧‧‧words

130‧‧‧Disable word filtering

140‧‧‧Pre-built Chinese fonts

150‧‧‧High frequency Chinese characters

160‧‧‧Pre-built Chinese lexicon appearances

170‧‧‧Pre-built English fonts

180‧‧‧Key Chinese word appearances

190‧‧‧Key English word appearances

210‧‧‧Pre-built Chinese lexicon appearances

220‧‧‧Key Chinese word appearances

230‧‧‧Pre-built English fonts

240‧‧‧Key English word occurrences

250‧‧‧High frequency key Chinese word appearances

260‧‧‧High frequency key English word appearances

270‧‧‧ file vector

310‧‧‧Uncategorized file vector

320‧‧‧Specific classification file vector mean

330‧‧‧Classification similarity

340‧‧‧ classification results

350‧‧‧Average file vector mean

Please refer to the detailed description of the present invention and the accompanying drawings, which can further understand the technical content of the present invention and its effect. The related drawings are: FIG. 1 is a file for extracting a file classification method based on group feature values of the present invention. Feature flow chart.

2 is a flow chart of the file feature vector of the file classification method based on the group feature value of the present invention.

FIG. 3 is a flow chart of file classification of a file classification method based on group feature values according to the present invention.

FIG. 4 is a file classification method based on group feature values according to the present invention.

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Hereinafter, the present invention will be further described with reference to the accompanying drawings: Referring to FIG. 1, FIG. 1 is a flow chart of extracting file features of a file classification method based on group feature values according to the present invention. As shown in Figure 1, after the file to be analyzed enters the system, the text in the word normalization processing 110 is not processed, and the English part is unified to lowercase, and the full-length punctuation of each type of unrelated meaning is deleted. And placed in the space between the Chinese and English words, in order to provide a blank word 120, and cut out all Chinese and English words. After all the Chinese and English words are separated, the stop word filtering 130 is performed, and the virtual words, the connected words, the meaningless words or the vocabulary without comparison are eliminated, and the file features are prevented from being affected by the unrelated words.

At this point, the action of taking words and calculating the frequency of occurrence can be started. The method of the present invention is to integrate two common methods: the situation without the pre-built word library, and the advantages of the pre-built word library, and The shortcomings of both are removed. To preserve the advantages of time-consuming aspects.

Therefore, the present invention is based on the method of pre-built word bank, and in order to solve the problem that the keyword word cannot be obtained, the present invention supplements the method of not having the pre-built word bank, that is, after calculating the pre-built word. After the frequency of occurrence of the lexicon, the words that do not appear in the vocabulary are taken and the frequency is calculated.

The overall concept is to do this, followed by detailed setting and explanation: the invention supports the analysis of Chinese and English. When taking the file feature, we select the Chinese part. Words are units. The first reason is that Chinese generally uses words as a unit to express meaning. The second reason is that if the length of the word is not fixed, the time complexity will increase significantly. Under these two preconditions, Chinese takes two words as the unit; There is no such problem in the English part, so the English part is in words.

Under this setting, when taking key Chinese words, it may be because the number of unrepeated two words is too large, resulting in too high time complexity of the comparison, slowing down the overall performance, thus losing the use of the pre-built words of the present invention. The purpose of the library. Therefore, in taking the key Chinese words, the present invention utilizes the characteristics that the final keyword words have a high frequency of occurrence, and pre-built the Chinese character library to extract high-frequency Chinese characters in advance, if any two words do not appear in the pre-built Chinese thesaurus. And the first word is high-frequency Chinese characters, which will be recorded as key Chinese words to improve performance.

After understanding the algorithm design of the present invention, proceeding to the subsequent process, after completing the stop word filtering 130, according to the design method mentioned in the foregoing, calculating the occurrence status of each Chinese character in the pre-built Chinese character library, and obtaining the pre-preparation The number of occurrences of the Chinese character library is 140, and a certain number N is set. The Chinese characters with more than N before the number of occurrences are set to the high frequency Chinese characters 150 to prepare for the subsequent removal of the key Chinese words. The other end calculates the number of occurrences of the pre-built Chinese lexicon 160, the number of pre-built English fonts 170, and then takes the key Chinese words and key English words, the Chinese two-word part, based on the high-frequency Chinese characters, The condition check of the number of occurrences of key Chinese words 180.

If any two words do not appear in the pre-built Chinese thesaurus, and the first word is a high-frequency Chinese character, it will be recorded as a key Chinese word and its number of occurrences will be accumulated; the part of the English word is not made. Special treatment, only the condition check for calculating the number of occurrences of key English words: 190: If any English word does not appear in the pre-built English font, it is recorded as a key English word and the number of occurrences is accumulated. So far, the document feature extraction process of the present invention has been completed.

Please refer to FIG. 2. FIG. 2 is a flow chart of extracting file feature vectors of the file classification method based on the group feature value according to the present invention. As shown in Figure 2, it includes the number of occurrences of the pre-built Chinese lexicon 210, the number of occurrences of key Chinese words 220, the number of occurrences of pre-built English fonts 230, and the number of occurrences of key English words 240. The part of the word does not take all the keyword words to do the similarity operation. The keyword words that appear few times will cause the similarity to shift.

Therefore, in the present invention, the high frequency keyword word is taken out as a parameter of the similarity operation in a manner similar to the foregoing, and a certain number M is set, and the number of occurrences of the pre-built Chinese lexicon number M is used as a reference, and the number of occurrences exceeds this value. The key Chinese words are recorded as the number of occurrences of high-frequency key Chinese words 250, and the number of occurrences of the number of pre-built English fonts is M. The key English words appearing more than this value are recorded as the number of high-frequency key English words. 260. Finally, the pre-built Chinese lexicon appearance number 210, the pre-built English font occurrence number 230, the high frequency key Chinese word appearance frequency 250, and the high frequency key English word appearance frequency 260 constitute a file vector 270.

Please refer to FIG. 3. FIG. 3 is a flowchart of file classification of a file classification method based on group feature values according to the present invention. As shown in FIG. 3, and according to the flow in FIG. 2, the unclassified file vector 310 is obtained for the file to be classified, and the specific classification file vector mean 320 is taken, which is obtained by averaging the file vectors of the specific classification; In conjunction with the concept of extracting keyword words proposed in this patent, the scope of words of known classifications is expanded after the classification is completed.

Therefore, the vector mean of the specific classification file also includes the keyword words of each file to improve the accuracy of the file classification. After the collection of the vector mean values of the various classification files is completed, the unclassified file vector 310 is used to perform the similarity calculation with the specific classification file vector mean 320, and one point is obtained for each specific classification. The similarity degree 330, if all the classification similarities do not reach a specified threshold, classify the file to be classified into a new classification; otherwise, classify it into the classification with the highest similarity to obtain the classification result 340. After the classification is completed, the average vector value 350 of the classified file is recalculated for the classification of the file to be used for the next classification.

The similarity part is mainly calculated by the vector cosine angle comparison method. The similarity formula of the file D and the classification E is: Among them, F _db represents the number of occurrences of the pre-built Chinese lexicon of the file 210, the number of occurrences of the pre-built English fonts 230, F _kw represents the number of occurrences of the high-frequency key Chinese words of the file 250, and the number of occurrences of the high-frequency key English words 260, and α indicates the importance of the keyword word to the pre-built word bank in this algorithm. Since the keyword word is a high-frequency word that does not appear in the pre-built word bank, it expresses the file in The characteristics of a particular domain have an indicator meaning, so α is generally set to a value greater than 1, to highlight the importance of keyword words. Then, according to a similarity formula, the classification similarity 340 is calculated, and the value must fall between 0 and 1. If it is greater than 1, it is taken as 1, and the similarity is used to assist the file classification work.

At this point, with the above process, the similarity between the document and each category can be finally obtained, and the classification result of the file can be known, thereby developing a data protection system, a file classification system, etc., to help prevent the outflow of the confidential file.

Please refer to FIG. 4. FIG. 4 is another operational flowchart of the wireless system and method for cooperating with the backhaul network device of the present invention. As shown in Figure 4, The detailed description of the present invention is intended to be illustrative of a preferred embodiment of the invention, and is not intended to limit the scope of the invention. The patent scope of this case.

To sum up, this case is not only innovative in terms of technical thinking, but also has many of the above-mentioned functions that are not in the traditional methods of the past. It has fully complied with the statutory invention patent requirements of novelty and progressiveness, and applied for it according to law. Approved this invention patent application, in order to invent invention, to the sense of virtue.

Claims (7)

A file classification method based on group feature values, the steps of which at least include: A. Steps of extracting file features are pre-processed for files, word normalization, word breaking, and deactivation of word library filtering are disabled. The action of words; B. According to the strategy of taking words, obtain the number of occurrences of pre-built Chinese thesaurus, pre-built English fonts, key Chinese words, key English words; C. Steps of extracting file feature vectors, according to each word The frequency of word occurrence establishes a file vector information, with pre-built Chinese lexicon, pre-built English font, high-frequency key Chinese words, high frequency key English words; D. file classification steps based on file vector, calculate to be classified The similarity between the document and the group feature value; and E. After repeating the foregoing process to obtain the similarity with all the classifications, classify the file to be classified into the category to be attributed. The method for classifying a file based on the group feature value according to the first aspect of the patent application, wherein the step of extracting the feature of the file further comprises: A1. pre-processing the file, first performing word normalization, wherein The normalization action, including unifying the English case of the file to be analyzed, and deleting various punctuation marks; A2. Secondly, the word breaking, and the action of creating a stop word library to filter the stop words, wherein the stop words Words include virtual words, connected words, meaningless words or vocabulary without comparison; and A3. Finally, the pre-built Chinese thesaurus, pre-built English fonts, key Chinese words, The number of occurrences of key English words, wherein the process of extracting the file feature vector is to use the pre-built Chinese character library, Chinese lexicon and English font to calculate the number of occurrences of each word in the file, but does not appear in the pre-built word library. They are called with key Chinese words and key English words. For example, the document classification method based on the group eigenvalues mentioned in the second paragraph of the patent application, in which the process of taking out the key Chinese words is taken in order to avoid taking too long, the word is taken as a unit, and the pre-match is used. The Chinese character library is used as an auxiliary for screening. For example, the document classification method based on the group feature value described in the first paragraph of the patent application, wherein the process of extracting the file feature vector is for the part of the key Chinese word and the key English word, and the number of occurrences is calculated separately, and the frequency of occurrence is taken. The frequency of occurrence of the collaborative pre-built word library constitutes vector information. For example, in the document classification method based on the group feature value described in Item 4 of the patent application, in order to avoid the occurrence of the offset of the similarity of the keyword words which are not frequently generated, the process of taking out the high frequency keyword words is matched with Pre-built word library as an aid to screening. The document classification method based on the group feature value according to claim 1, wherein the file classification process is: D1. The extraction of the group feature value includes the keyword word frequency of each file belonging to the category, The average obtains a specific classification file vector mean; D2. Secondly, the file vector is used for the similarity calculation, and the calculation method can be the cosine similarity method; D3. After obtaining the similarity with all the classifications, if all the values are less than one specified door 槛, the file to be classified is classified into a new category; otherwise, it is classified into the category with the highest similarity to complete the file classification work. For example, the document classification method based on the group feature value described in claim 6 of the patent application, wherein the process of calculating the similarity further includes the step of increasing the weight of the keyword word according to the importance.