CN102521402A - Text filtering system and method - Google Patents

Abstract

The invention discloses a text filtering system and method. The system at least includes: an ontology library building module, which is used to build an ontology library according to the user's filtering requirements; an adaptive learning module, which trains and learns a group of filtering samples to The ontology library established by the ontology library building module is dynamically adjusted to make it gradually approach the user's filtering requirements; and the text filtering module obtains the The degree of correlation between the text to be filtered and the ontology, and the text to be filtered is filtered according to the degree of correlation. Through the present invention, not only can the user's filtering model be accurately expressed, but also self-learning can be performed during filtering, and the users who use the ontology expression can be adjusted. Filtering model, and can dynamically adjust the filtering threshold to achieve better filtering effect.

Description

Text filtering system and method

technical field

The invention relates to a text filtering system and method, in particular to an ontology-based adaptive text filtering system and method.

Background technique

In the field of information retrieval and filtering, text filtering has always been a research hotspot. At present, many domestic and foreign literatures have adopted different methods to realize text filtering.

The current text filtering methods mainly include the fuzzy clustering text filtering method based on genetic algorithm, the text filtering method using improved classification algorithm, the text filtering method using adaptive learning filtering algorithm and the text filtering method only using ontology. Among them, the fuzzy clustering method based on genetic algorithm is used to directly cluster each individual in the population with a fuzzy similarity matrix, and then use the proposed fitness function to evaluate the fitness of the population according to the clustering results. However, this The filtering accuracy of this text filtering method depends on the effect of clustering, and it cannot express the user's filtering needs well; the text filtering method using an improved classification algorithm filters bad text information, and improves the traditional one from the perspective of the data layer. The disadvantage of the KNN algorithm is also that it is not accurate enough to express the user's needs; the text filtering method using the adaptive learning filtering algorithm can perform adaptive learning by training the sample set, and can adjust the filtering model, but its filtering effect on the user The expression of requirements is also not precise enough; only the text filtering method of ontology is used, and the filtering accuracy depends on the establishment of ontology. If the ontology database is not created accurately, it will greatly affect the accuracy of text filtering.

To sum up, it can be seen that in the text filtering method of the prior art, there is a problem that the expression of the user's needs is not accurate enough or the creation of the ontology database is not precise enough to affect the accuracy of the text filtering. Therefore, it is necessary to propose an improved technical means to solve this problem.

Contents of the invention

In order to overcome the deficiencies in the above-mentioned prior art, the main purpose of the present invention is to provide a text filtering system and method, which can not only accurately express the user's filtering model, but also can carry out independent learning during filtering, and adjust the user's text using ontology expression. Filtering model, and can dynamically adjust the filtering threshold to achieve better filtering effect.

To achieve the above and other purposes, the present invention provides a text filtering system, at least including:

Ontology library building module, used to build ontology library according to user's filtering requirements;

The self-adaptive learning module dynamically adjusts the ontology library established by the ontology library building module by training and learning a group of filtering samples, making it gradually approach the user's filtering requirements; and

The text filtering module obtains the degree of correlation between the text to be filtered and the ontology after preprocessing the text to be filtered, extracting the feature word set and matching the similarity, and filters the text to be filtered according to the degree of correlation.

Further, the ontology library building module at least includes:

The domain determination module is used to clarify the domain and scope covered by the ontology to be built according to the user's filtering requirements, so as to determine the domain and scope of the ontology;

The collection and analysis module is used to collect and analyze information within the scope of the ontology, clarify key concepts and the relationship between concepts, and express them in precise terms; and

The ontology frame building module is used to build the ontology frame according to the collection and analysis results.

Further, the ontology is represented by a triplet Topic(C, P, S), where C represents a collection of concept classes with the same attribute and behavior structure abstracted from noun concepts in the filtering field; P describes concepts and The attribute of the relationship; S represents the structural relationship between classes, such as parent class, subclass, etc.

Further, the self-adaptive learning module adopts an incremental iterative method to train and learn a group of filtered samples to dynamically adjust the ontology library built by the ontology library building module.

Further, the text filtering module includes at least

A preprocessing module for removing stop words from the text to be filtered;

The feature word set extraction module is used to extract the feature words that express the text content from the text to be filtered, assign corresponding weights according to the different positions and frequencies of the feature words, and add the same feature word weight values to form text features vocabulary;

The similarity calculation module calculates the correlation between the text to be filtered and the ontology according to the vector space model; and

The filtering module filters the text to be filtered according to the correlation degree and a set threshold.

Further, the filtering module filters the texts that are lower than the threshold in the filtered texts.

In order to achieve the above and other purposes, the present invention provides a text filtering method, which at least includes the following steps:

Build an ontology library according to the user's filtering requirements;

Carry out training and learning on a set of filtered samples to dynamically adjust the established ontology library, making it gradually approach the user's filtering requirements; and

After the text to be filtered is preprocessed, feature word set extracted and similarity matching processed, the correlation between the text to be filtered and the ontology is obtained, and the text to be filtered is filtered according to the correlation.

Further, the step of establishing an ontology library according to the user's filtering requirements at least includes the following steps:

According to the user's filtering requirements, specify the field and scope covered by the ontology to be constructed to determine the field and scope of the ontology;

Collect and analyze information within the scope of the ontology, clarify the key concepts and the relationship between concepts, and express them in precise terms; and

Based on the collected and analyzed results, an ontology framework is established.

Further, the dynamic adjustment of the ontology database is realized by an incremental iterative method.

Further, the step of filtering the text to be filtered further includes at least the following steps:

Remove stop words from the text to be filtered;

Extract the feature words expressing the text content in the text to be filtered, assign corresponding weights according to the different positions and frequencies of the feature words, and add the same feature word weight values to form a text feature word set;

Calculate the correlation between the text to be filtered and the ontology according to the vector space model; and

The text to be filtered is filtered according to the relationship between a set threshold and the correlation degree.

Compared with the prior art, a text filtering system and method of the present invention can more accurately express the user's filtering requirements by establishing an ontology database, and at the same time, in order to further ensure that the ontology database is closer to the user's filtering requirements, the present invention adopts adaptive learning By training and learning a group of samples, part of the ontology library is dynamically adjusted, which overcomes the shortcomings of the traditional feature vector method and the general method of establishing an ontology library, which cause inaccurate expression of user needs and cause low filtering accuracy. In addition, this In the filtering stage, the invention uses a space vector model to calculate the similarity between the text to be filtered and the ontology library, filters out texts below the threshold, and can dynamically adjust the filtering threshold to achieve a better filtering effect. Practice has proved that the present invention An adaptive text filtering method based on ontology can obtain higher filtering accuracy.

Description of drawings

Fig. 1 is a system architecture diagram of a text filtering system of the present invention;

FIG. 2 is a flow chart of the steps of a text filtering method in the present invention.

Detailed ways

The implementation of the present invention is described below through specific examples and in conjunction with the accompanying drawings, and those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific examples, and various modifications and changes can be made to the details in this specification based on different viewpoints and applications without departing from the spirit of the present invention.

FIG. 1 is a system architecture diagram of a text filtering system of the present invention. As shown in FIG. 1 , a text filtering system of the present invention includes at least: an ontology database building module 10 , an adaptive learning module 11 and a text filtering module 12 .

The ontology library building module 10 is used to build an ontology library according to the user's filtering requirements, and at least includes a field determination module 101 , a collection and analysis module 102 and an ontology framework building module 103 . The domain determination module 101 first defines the domain and scope covered by the ontology to be constructed according to the user's filtering requirements to determine the domain and scope of the ontology; the collection and analysis module 102 is used to collect information within the scope of the domain involved in the ontology and analysis, clarify key concepts and the relationship between concepts, and express them in precise terms, for example, in a preferred embodiment of the present invention, the ontology is represented by a triple Topic(C, P, S), where: C represents the collection of concept classes with the same attribute and behavior structure abstracted from noun concepts in the filtering field; P describes the attributes of concepts and relationships; S represents the structural relationship between classes, such as parent class, subclass, etc. C adopts vector space model (VSM) to represent, and uses two-tuple C _i (Key _i , Weight _i ), wherein Key _i represents a keyword, and Weight _i represents the weight of a keyword; The collection and analysis results of the analysis module 102 establish an ontology framework.

The self-adaptive learning module 11 dynamically adjusts the ontology library built by the ontology library building module 10 by training and learning a group of filtering samples, so that it gradually approaches the user's filtering requirements. In a preferred embodiment of the present invention, the self-adaptive learning module 11 uses an incremental iterative method to train a group of filtered samples, and sets a fixed value m as the window size for observing the number of new documents that need to be filtered out. Set flexibly according to the parameter n of the evaluation index, and set the number of training iterations to 5. In the incremental iterative training process, it is necessary to determine the number of feature items added each time to avoid generating more noise. According to the increased effective feature value , select a certain number to add to the existing ontology library, and enrich the user's filtering demand model. Therefore, with continuous learning, the ontology library is getting closer to the user's filtering requirements, and the necessary features of the ontology library are gradually reduced.

The text filtering module 12 preprocesses the text to be filtered, extracts the feature word set and matches the similarity, and then filters the text to be filtered according to the correlation between the text to be filtered and the ontology. It includes at least a preprocessing module 121 , a feature word set extraction module 122 , a similarity calculation module 123 and a filtering module 124 . Among them, the preprocessing module 121 is used to remove stop words and other preprocessing operations on the text to be filtered, and the feature word set extraction module 122 is used to extract the feature words expressing the content of the text, and assign corresponding words according to different positions and frequencies of the feature words. Weight, the same feature word weight value is added, forms text feature word set Ti={(Word1k, Weight1k)}, and the text to be filtered has adopted feature vector to represent like this; Similarity calculation module 123 is according to vector space model, two The cosine value of the angle between the eigenvectors can represent their degree of correlation, and thus the degree of correlation Sim _j between a text to be filtered and the ontology can _be calculated; the filtering module 124 treats Filter the text to filter, that is, to filter the text below the threshold.

FIG. 2 is a flow chart of the steps of a text filtering method in the present invention. As shown in Figure 2, a kind of text filtering method of the present invention comprises the following steps at least:

In step 201, an ontology library is established according to user's filtering requirements. In this step, firstly, according to the user's filtering requirements, specify the domain and scope covered by the ontology to be constructed; The relationship between concepts is expressed in precise terms; finally, the ontology framework is established. In a preferred embodiment of the present invention, the ontology is represented by a triplet Topic (C, P, S), wherein: C represents a set of concept classes that are abstracted from the noun concepts in the filtering field and have the same attributes and behavioral structures ;P describes the attributes of concepts and relationships; S represents the structural relationship between classes, such as parent class, subclass, etc., C is represented by the vector space model (VSM), using the binary group C _i (Key _i , Weight _i ) , where Key _i represents the keyword, and Weight _i represents the weight of the keyword.

Step 202, training and learning a group of filtering samples to dynamically adjust the established ontology database so that it gradually approaches the user's filtering requirements. In a preferred embodiment of the present invention, an incremental iterative method is used to train a group of filtered samples, and a fixed value m is set as the window size for observing the number of new documents that need to be filtered out. According to the parameter n of the evaluation index To set it flexibly, and set the number of training iterations to 5. In the incremental iteration training process, it is necessary to determine the number of feature items added each time to avoid generating more noise. According to the increased effective feature value, select a certain number of increased Into the existing ontology library, enrich the user's filtering requirement model, so with continuous learning, the ontology library is getting closer to the user's filtering requirement, and the necessary features of the ontology library are gradually reduced.

Step 203: After preprocessing the text to be filtered, extracting feature word sets and matching similarity, the text to be filtered is filtered according to the correlation between the text to be filtered and the ontology. The specific process is as follows: firstly, the filtered text is subjected to preprocessing operations such as removing stop words; then the feature words expressing the content of the text are extracted, and corresponding weights are given according to the different positions and frequencies of the feature words. Add to form the text feature word set Ti={(Word1k, Weight1k)}, so that the text to be filtered is represented by a feature vector; then according to the vector space model, the cosine value of the angle between the two feature vectors can represent their degree of correlation. Thus, a correlation degree Sim _j between the text to be filtered and the ontology can be calculated; finally, the text to be filtered is filtered according to the relationship between the set threshold and the correlation degree Sim _j , that is, texts below the threshold are filtered.

It can be seen that since ontology can clearly define domain concepts and concepts, a text filtering system and method of the present invention can more accurately express the user's filtering needs by establishing an ontology library, and at the same time, in order to further ensure that the ontology library is closer to the user's To filter requirements, the present invention adopts an adaptive learning method, through training and learning on a group of samples, and partially dynamically adjusting the ontology library, which overcomes the traditional feature vector method and the general method of establishing an ontology library, which cause inaccurate expression of user needs and cause filtering. In addition, the present invention uses a space vector model to calculate the similarity between the text to be filtered and the ontology library in the filtering stage, filters out texts below the threshold, and can dynamically adjust the filtering threshold to achieve better The filtering effect is proved by practice that the ontology-based adaptive text filtering method of the present invention can obtain higher filtering precision.

The above-mentioned embodiments only illustrate the principles and effects of the present invention, but are not intended to limit the present invention. Any person skilled in the art can modify and change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention should be listed in the claims.

Claims (10)

1. A text filtering system comprising at least: Ontology library building module, used to build ontology library according to user's filtering requirements; The self-adaptive learning module dynamically adjusts the ontology library established by the ontology library building module by training and learning a group of filtering samples, making it gradually approach the user's filtering requirements; and The text filtering module obtains the degree of correlation between the text to be filtered and the ontology after preprocessing the text to be filtered, extracting the feature word set and matching the similarity, and filters the text to be filtered according to the degree of correlation. 2. text filter system as claimed in claim 1, is characterized in that, this ontology storehouse builds module to comprise at least: The domain determination module is used to clarify the domain and scope covered by the ontology to be built according to the user's filtering requirements, so as to determine the domain and scope of the ontology; The collection and analysis module is used to collect and analyze information within the scope of the ontology, clarify key concepts and the relationship between concepts, and express them in precise terms; and The ontology frame building module is used to build the ontology frame according to the collection and analysis results. 3. The text filtering system as claimed in claim 2, characterized in that: the ontology is represented by a triplet Topic (C, P, S), wherein C represents an abstraction from a noun concept in the filtering field, and has the same A collection of conceptual classes with attributes and behavioral structures; P describes the attributes of concepts and relationships; S represents the structural relationships between classes, such as parent classes, subclasses, etc. 4. text filtering system as claimed in claim 1, is characterized in that: this self-adaptive learning module adopts the incremental iterative method to carry out training and learning to a group of filtering samples to build up the ontology library dynamics of module establishment to this ontology library. Adjustment. 5. the text filter system as claimed in claim 1, is characterized in that, this text filter module comprises at least: A preprocessing module for removing stop words from the text to be filtered; The feature word set extraction module is used to extract the feature words that express the text content from the text to be filtered, assign corresponding weights according to the different positions and frequencies of the feature words, and add the same feature word weight values to form text features vocabulary; The similarity calculation module calculates the correlation between the text to be filtered and the ontology according to the vector space model; and The filtering module filters the text to be filtered according to the correlation degree and a set threshold. 6. The text filtering system according to claim 5, characterized in that: the filtering module filters texts below the threshold in the filtered texts. 7. A text filtering method, comprising at least the following steps: Build an ontology library according to the user's filtering requirements; Carry out training and learning on a set of filtered samples to dynamically adjust the established ontology library, making it gradually approach the user's filtering requirements; and After the text to be filtered is preprocessed, feature word set extracted and similarity matching processed, the correlation between the text to be filtered and the ontology is obtained, and the text to be filtered is filtered according to the correlation. 8. A kind of text filtering method as claimed in claim 7, it is characterized in that, the step of setting up the ontology library according to user's filtering requirement also comprises the following steps at least: According to the user's filtering requirements, specify the field and scope covered by the ontology to be constructed to determine the field and scope of the ontology; Collect and analyze information within the scope of the ontology, clarify the key concepts and the relationship between concepts, and express them in precise terms; and Based on the collected and analyzed results, an ontology framework is established. 9. A text filtering method as claimed in claim 7, characterized in that: the dynamic adjustment of the ontology database is realized by an incremental iterative method. 10. A kind of text filtering method as claimed in claim 7, is characterized in that, the step of filtering the text to be filtered further comprises the following steps at least: Remove stop words from the text to be filtered; Extract the feature words expressing the text content in the text to be filtered, assign corresponding weights according to the different positions and frequencies of the feature words, and add the same feature word weight values to form a text feature word set; Calculate the correlation between the text to be filtered and the ontology according to the vector space model; and filter the text to be filtered according to the relationship between a set threshold and the correlation.

Images