JP5056337B2 - Information retrieval system - Google Patents

Description

  The present invention relates to an information search system that analyzes text written in a document and classifies the document according to the content of the text.

  Conventionally, methods relating to automatic document classification have been proposed. For document classification using machine learning, for example, a regular expression is described by a specific term group or regular expression determined in advance as in the method disclosed in Patent Document 1, and this regular expression is matched. By replacing character strings with tokens (labels expressed in lexical terms), the number of feature elements required for learning to classify a document having a large vocabulary has been reduced.

JP-T-2003-535407, pages 76-85

  In the method of Patent Document 1, which is an example of document classification based on conventional machine learning, a sentence that is divided into words such as English is assumed, and a sentence that does not have a word break such as Japanese is used. There was a problem that it could not be applied as it was. As a technique for solving this problem, a technique of morpheme analysis in which a sentence without a word break is divided for each word is known. However, morpheme analysis has a problem that it takes time because of a large amount of processing.

  In addition, when the method of Patent Document 1 is applied to detection of confidential information, for example, when personal information that is one of confidential information is tokenized, there is a problem that erroneous detection of personal names increases. However, there was no countermeasure against this, and there was a problem that classification accuracy was lowered due to erroneous detection of tokens.

  The present invention has been made to solve the above-described problems, and is characterized by high-speed text tokenization by character string matching, which is targeted for texts such as Japanese that have no word breaks. The purpose is to provide a search system. It is another object of the present invention to provide an information search system characterized by reducing the erroneous detection of tokens obtained by character string matching and improving the classification accuracy.

  In order to solve the above-described problem, an information search system according to the present invention includes a matching condition storage unit that stores a matching condition between a character string and a keyword and a feature token that identifies the matching condition, and Based on the collation condition stored in the collation condition storage means and the feature token, the character string of the learning document classified in advance by category is collated with the keyword, and the matching condition corresponding to the collation condition is matched. One characteristic token is extracted in association with the category, and based on the matching condition and the feature token stored in the matching condition storage unit, A feature token extracting means for matching the keyword and extracting a second feature token corresponding to the matched matching condition; and the first feature token The first non-feature token obtained by dividing the character string of the learning document that has not been extracted in character units is extracted in association with the category, and the classification target document from which the second feature token has not been extracted A non-characteristic token extracting means for extracting a second non-characteristic token obtained by dividing the character string into character units, and a first token string composed of the first characteristic token and the first non-characteristic token. Learning means for calculating an appearance frequency as a learning frequency in association with the category, an appearance frequency of a second token string composed of the second feature token and the second non-feature token, and the learning means A classification probability indicating similarity to the learning frequency calculated by the above is calculated for each category, and the classification target document is classified into the category in which the classification probability exceeds a predetermined threshold. It is obtained by a further comprising a classification unit that.

  According to the present invention, using a matching condition that defines a feature token to be extracted, a sequence of feature tokens and non-feature tokens is extracted from an input document, and prioritization of matching conditions and token chain probability are used. Therefore, even if a document containing sentences without word breaks is input, tokenization can be performed using character string matching that is faster than morphological analysis, so high-speed processing is possible. In addition, there is an effect that the classification token can be improved by preventing erroneous detection of the feature token.

  In the following description, confidential information retrieval is taken as an example as an embodiment, but the present invention is not limited to confidential information retrieval and can be widely used for document classification in general. In the following description, a search for a Japanese document is taken as an example, but the application of the present invention is not limited to Japanese, and any character code can be applied.

Embodiment 1 FIG.
FIG. 1 is a configuration diagram illustrating an example of an information search system according to the first embodiment.
This information retrieval system includes a preprocessing unit 100, a learning unit 200, a classification unit 300, and a collation condition storage unit 400 that stores collation conditions input to the preprocessing unit 100.

  The preprocessing unit 100 further includes a text extraction unit 101, a feature token extraction unit 102, and a non-feature token extraction unit 103. The learning unit 200 includes a learning frequency calculation unit 201 that calculates the token frequency for learning, and a learning frequency storage unit 202 that accumulates the calculated token frequency for each category of classification and stores it as a learning frequency. . The classification means 300 calculates the input frequency based on the classification frequency calculation means 301 for calculating the token frequency for classification, the token frequency for classification, and the token learning frequency stored in the learning frequency storage means 202. Classification probability calculation means 302 for calculating the classification probability of the input document, and category determination means 303 for finally determining the category of the input document.

  The learning document 501 is a set of a plurality of documents previously classified into a plurality of categories. Specifically, the category can be expressed by two classifications such as (“non-confidential document”, “confidential document”). Or, according to the confidentiality level of the confidential document, it may be expressed by three or more classifications such as (“non-confidential document”, “confidential level 1 document”, “confidential level 2 document”,...). good.

  On the other hand, the classification target document 502 is a document whose category is not known, and is a set of documents to be classified. The category to which the classification target document 502 belongs is determined by the information search system of the first embodiment.

  The collation condition storage unit 400 stores the collation conditions input to the preprocessing unit 100. This collation condition is set prior to learning and classification, and can be divided into a built-in collation condition 4001 and a user-defined collation condition 4002. The built-in collation condition 4001 is a collation condition incorporated in advance at the time of shipping of the information search system. By providing this as a basic collation condition, the user can immediately start using the information search system. Is. The user-defined matching condition 4002 allows each user to customize a matching condition by adding a unique term or the like.

  Said collation conditions are represented by the set of the combination of a keyword and collation condition ID for identifying collation conditions. The keyword here means an expression format for designating a word or character string belonging to each term class with respect to a plurality of term classes indicating a general concept of the word or character string. The keyword expression format may be a list of a plurality of fixed terms, or may be a regular expression.

Next, the operation of the information search system according to Embodiment 1 of the present invention will be described in detail with reference to FIGS. 2 to 6 as appropriate.

First, the operation in the learning stage will be described with reference to FIG.
FIG. 2 is a flowchart showing the operation of the information search system in the first embodiment.
In step S <b> 11, learning documents 501 categorized in advance are input to the preprocessing unit 100. Next, in step S <b> 12, the preprocessing unit 100 extracts a token string from the learning document 501. Next, in step S13, the extracted token string is analyzed by the learning frequency calculation means 201, and the appearance frequency of a sequence composed of consecutive N tokens is calculated. Here, the number of consecutive tokens may be one or more and N or less. Next, in step S14, the learning frequency storage means 202 stores the learned token frequency as a learning frequency for each category. In step S15, the learning unit 200 determines whether learning of all the documents of the learning document 501 has been completed. As a result of the determination, if the learning document 501 includes a plurality of documents, the process proceeds to a NO branch, and the operations after step S11 are repeated. On the other hand, if all the documents are completed as a result of the determination, the process proceeds to a Yes branch, and the learning stage operation is terminated.

Next, the operation in the classification stage will be described with reference to FIG.
FIG. 3 is a flowchart showing the operation of the classification stage in the first embodiment.
In step S <b> 21, the classification target document 502 is input to the preprocessing unit 100. Next, in step S <b> 22, the pre-processing unit 100 extracts a token string from the classification target document 502. Next, in step S23, the classification frequency calculation unit 301 analyzes the extracted token string and calculates the appearance frequency of a sequence composed of consecutive N tokens. Here, the number of consecutive tokens may be one or more and N or less. Next, in step S24, the classification probability calculation unit 302 calculates the probability that the classification target document 502 is classified into each category based on the learning result. Next, in step S25, the category determination unit 303 determines to which category the classification target document 502 is classified based on the probability calculated in step S24. Finally, in step S26, the classification unit 303 outputs the classification destination category determined in step S25 as a classification result (S26).

  In order to realize the learning and classification operations described above, it is possible to use text classification software shown as CRM 114 in Non-Patent Document 1.

"Sparse Binary Polynomial Hashing and the CRM114 Discriminator" by William S. Yerazunis, MIT Spam Conference 2003, January 17, 2003

  It should be noted that the learning operation may be executed collectively in the initial stage of system initialization. Furthermore, when a classification error occurs during the classification operation, the learning data may be updated by re-learning the document in which the classification error has occurred for learning.

Next, the operation of the preprocessing unit 100 will be described in more detail with reference to FIG.
FIG. 4 is a flowchart showing the operation of the preprocessing unit 100 in the first embodiment.
In step S31, the learning document 501 or the classification target document 502 classified in advance is input to the preprocessing unit 100 as an input document. Next, in step S32, text that is a description expressed in a natural language is extracted from the input document by the text extraction unit 101 (S32).

  The text extracting unit 101 extracts text from a document of an arbitrary format. Examples of arbitrary format documents include texts such as various types of documents generated by commercially available document editing software and documents used on the Web such as e-mail and HTML (HyperText Markup Language). Any document may be included. The text extraction means 101 can also use commercially available text extraction software. Further, the output of the text extraction means 101 may be a character string in the document with spaces, tabs, line feeds, etc. removed. As a result, it is possible to reduce detection omissions caused by a space, tab or line feed in the middle of a term.

  Next, in step S33, the feature token is extracted from the text extracted in step S32 by the feature token extraction means 102. The feature token here means a character string representing a term class set corresponding to each matching condition in the matching conditions stored in the matching condition storage unit 400. This feature token may be the term class name itself or the collation condition ID itself, but for example, the collation condition ID is not confused with a non-feature token extracted as a character string by the non-feature token extraction means 103 described later. Corresponding to = 1, a special character string such as “TOKEN_1” composed of alphanumeric characters can be used.

  The feature token extraction unit 102 refers to the collation condition stored in the collation condition storage unit 400, collates the text extracted in step S32 with the keyword set in the collation condition, and matches the keyword. The inside character string is replaced with a feature token corresponding to the matched matching condition ID. Such collation processing and replacement processing with feature tokens are as follows when the keyword in the collation condition is given by a plurality of fixed terms. The feature token extraction unit 102 compares all the matching conditions by comparing the character strings of all the terms belonging to each matching condition with the text, and replaces the character string with the feature token if they match. This process is executed while shifting the text collation position one character at a time.

  When the keywords set in the collation conditions are given by regular expressions, for example, SED of a text processing program used in a general OS (Operating System), PERL of a programming language, and other regular expression libraries As described above, it can be realized by using software having a regular expression replacement function.

  In this way, by replacing the term in the text with the feature token, the vocabulary size used as the feature amount of the document classification can be reduced, and the learning amount necessary for the learning process by the learning means 200 at the subsequent stage can be reduced. For example, a person name used as a feature quantity related to personal information requires a term of about several thousand words even if it is limited to a frequent person name. In order to learn this frequent person name, a larger amount of learning data than a few thousand words is required. Therefore, there is a problem that it is difficult for the user to collect learning data. However, if the vocabulary size is reduced by extracting feature tokens as described above, highly accurate classification can be realized with less learning data.

  Next, in step S34, the non-feature token extraction unit 103 extracts a non-feature token from a character string in the text that has not been extracted as a feature token. The non-character token here is a single character. That is, a character string other than the feature token is extracted character by character as a non-feature token.

  More preferably, taking a Japanese document as an example, Japanese characters are extracted in units of one character, and alphanumeric characters and symbols are extracted in units of switching character types such as Japanese characters and alphanumeric characters. Do it. In this way, if the feature token is described by alphanumeric characters and symbols as in the above-mentioned example of “TOKEN_1”, the feature token already extracted by the feature token extracting means 102 can be easily distinguished from other character strings. Thus, in order to prevent the feature token from being divided by the extraction processing by the non-feature token extraction means 103, it is possible to omit the processing step of checking whether or not it is a feature token. Of course, non-characteristic tokens can be extracted similarly for documents other than Japanese.

  Next, in step S35, the preprocessing unit 100 outputs a token string composed of the characteristic tokens extracted above and the non-characteristic tokens. As a data output method of this token string, for example, a token string is expressed by using a space delimiter, a line feed delimiter, etc., and a text file storing this token string is passed to the subsequent learning means 200 and classification means 300. To. Alternatively, the token string may be stored in a memory as an array of character strings and passed to the learning means 200 or the classification means 300 at the subsequent stage.

  Through the above preprocessing, a token string (a sequence composed of characteristic tokens and non-characteristic tokens) is extracted from the input document. As described above, even when a document including a sentence without a word break is input, high-speed processing can be performed by performing tokenization using character string matching that is faster than morphological analysis.

  Here, the matching condition will be described in more detail with reference to the example of the matching condition shown in FIG. In the following explanation, a keyword expressed by a regular expression (regular expression keyword) is taken as an example, but a keyword using a fixed term can be similarly applied.

FIG. 5 is a diagram illustrating an example of collation conditions in the first embodiment.
As a matching condition, a set of a regular expression keyword and a matching condition ID can be prepared for each term class, and a plurality of sets can be provided. As shown in FIG. 5, for a term class such as a person name (last name) or a prefecture name, for example, a regular expression keyword that lists terms (proper nouns) belonging to the class can be created. Further, for term classes such as e-mail addresses and telephone numbers, regular expression keywords based on expression patterns unique to them can be created.

  Here, when classified information is classified, personal name and address, telephone number, e-mail address, etc. are characteristic term classes for personal information that is one of classified information. Furthermore, name terms that are often used for names are also effective in detecting personal information files. In addition, in order to detect general confidential information, a term class indicating a confidential level such as “confidential”, a document type (such as a specification) that often contains confidential information, and a term such as a supplier name A class can be defined. Although FIG. 5 shows some examples of term classes, the term classes and regular expression keywords belonging to the term classes are not limited to these.

  Further, in the built-in matching condition 4001 and the user-defined matching condition 4002 in FIG. 5, the matching condition ID may overlap. That is, when it is desired to add a term to a term class already defined in the built-in collation condition 4001, the term can be added by using the same collation condition ID in the user-defined collation condition 4002. In the example of FIG. 5, it is shown that the user name desired by the user is added in the matching condition ID = 1.

  By using the same collation condition ID in this way, even when a new term is added during system operation, it does not change as a term class, and thus there is an advantage that already learned learning data can be used as it is. Of course, by learning the learning document 501 again, the learning result including the difference information regarding the added term can be updated and held.

  In the user-defined collation condition 4002, a new collation condition ID can also be assigned. In the example of FIG. 5, in the term class “document type”, a collation condition ID that is not included in the built-in collation condition 4001 of collation condition ID = 100 is used. This allows the user to define new term classes and add terms.

  Furthermore, priorities can be assigned to the matching conditions. In this case, collation with the character string in the text is executed in order from the highest priority. As a method of assigning priorities, for example, it is possible to separately manage priorities such as (ID, regular expression keyword, priority) as collation conditions, or to describe collation conditions Depending on the order, the priority of the collation conditions described earlier can be increased.

  As a more desirable mode, the priority order can be determined based on the size of the collation condition ID. In this case, according to the example of FIG. 5, the collation condition ID = 1 is the lowest priority, and the collation condition ID = 101 is the highest priority. For example, when collation is performed on the character string “Tokyo” in the text, “Tokyo” at ID = 10 and “East” at ID = 1 are hit, but ID = 10. Since the priority is higher, the character string “Tokyo” is converted into a feature token as TOKEN_10. When detecting personal information by personal name, there is a problem of degradation of classification accuracy due to erroneous detection of a single character name such as “East”. By using the priority of this matching condition, False detection when Kanji is used in place names or company names can be reduced.

A specific example in the case where such erroneous detection occurs is shown in FIG.
FIG. 6 is a diagram illustrating an example of a token string generated by the preprocessing unit 100 using the matching condition of FIG. 5 in the first embodiment.
The tokenized character string is shown in a display format of one token per line, “Suzuki” is TOKEN_1, “Tokyo” is TOKEN_10, “East” of “East Nihonbashi” is TOKEN_1, “(03) "1111-2222" is converted into a feature token as TOKEN_12. In addition, since the other characters are Japanese in this example, one character is converted into a non-character token as one token. In the case of the example in FIG. 6, “East Nihonbashi” “East” is converted into a characteristic token as TOKEN_1, which is a false detection. In order to deal with such a false detection, for example, a new term class “place name” can be defined to include “East Nihonbashi” as a term, but all false detections can be avoided. is not.

  In the present embodiment, in addition to the prioritization of the matching conditions described above, the feature tokens are associated with the feature tokens before and after that or the non-feature tokens to prevent false detection and The influence of false detection can be reduced. For example, in FIG. 6, regarding a sequence of three tokens “TOKEN_1, day, book”, the characteristic token TOKEN_1 is more likely to be a place name “East Japan” than a person name. As described above, the learning unit 200 cuts the sequence of 1 to N tokens from the token string and calculates the appearance frequency of the sequence by calculating the chain probability of the three tokens as described above. The token chain probability can be learned. By learning or classifying using the token chain probability, it is possible to prevent erroneous detection of feature tokens and improve classification accuracy.

  Furthermore, in this embodiment, the time required for learning and classification is reduced by not using a sequence of 1 to N tokens that do not include any feature tokens for learning or classification, and further, learning data It is possible to reduce the storage capacity required for storing the. When a sequence of 1 to N tokens does not include a feature token, that is, all are non-feature tokens, the sequence is an enumeration of characters (non-feature tokens) that do not contribute to learning or classification. In particular, when the number of documents included in the learning document 501 is relatively small, each sequence pattern has a low appearance frequency and is unlikely to be significant for classification. The result is not much different. As a specific implementation method for this, the learning means 200 and the classification means 300 may ignore a sequence of 1 to N tokens that do not include a feature token, or the preprocessing means 100 Only a sequence of 1 to N tokens including feature tokens may be output.

  As described above, according to the first embodiment, using a matching condition that defines a feature token to be extracted, a sequence of feature tokens and non-feature tokens is extracted from an input document, and prioritization of matching conditions is performed. Learning or classification using token chain probability, tokenization can be performed using string matching, which is faster than morphological analysis even when a document containing sentences without word breaks is input. Therefore, it is possible to perform high-speed processing, and further, it is possible to prevent erroneous detection of feature tokens and improve classification accuracy.

Embodiment 2. FIG.
In the first embodiment described above, feature token extraction is realized by a character string replacement function. Next, an embodiment in which feature token extraction is realized by a regular expression matching function will be described. .

  As a character string matching method using a regular expression as a matching condition, a method using NFA (Non Deterministic Finite Automaton) or a method using DFA (Deterministic Finite Automaton) is known. In the case of DFA, a regular expression is compiled to generate a state transition table, and collation is performed by applying the state transition table to an input character string. Therefore, it is known that collation can be performed faster than NFA. . Although any method can be used in the present invention, an example using DFA is shown in the following description.

FIG. 7 is a diagram illustrating an example of a configuration related to the feature token extracting unit 102 in the case where the DFA is used in the second embodiment.
In the second embodiment, the state transition table generation unit 104 that generates the state transition table 105 from the collation condition stored in the collation condition storage unit 400 is provided. The feature token extraction unit 102 refers to the state transition table 105. It comprises collation means 1021 for collating with the input character string 1020 and replacement means 1022 for replacing the character string from the collation result.

  In FIG. 7, the state transition table generation unit 104 analyzes all character strings included in the collation condition, and expresses the analysis process until the character string is received by the DFA, and generates the state transition table 105. . The method for generating the state transition table 105 is a generally well-known method, and such an existing method may be used in the state transition table generating unit 104.

  The collating unit 1021 receives the input character string 1020 (text extracted by the text extracting unit 101) one character at a time and performs collation using the state transition table 105 to determine whether the collation condition is met. If they match, the end position of the matched character string (the position from the beginning of the input character string 1020, referred to here as the hit position) and the matched matching condition ID are output. Further, the matching unit 1021 calculates the start position of the matched character string (position from the beginning of the input character string 1020), and the replacement unit 1022 calculates the start position and end position (hit position) of the matched character string. Can be replaced with feature tokens.

  Alternatively, the collating unit 1021 can calculate only the end position (hit position) without calculating the start position. By doing so, processing such as tracing back the state transition table 105 in order to calculate the start position or managing the start position in each state of the state transition table 105 becomes unnecessary, and collation can be speeded up.

In this case, the replacement unit 1022 operates as follows.
The matching conditions include a term class having a proper noun as a term such as a person name, a place name, and a company name, and a term class having a regular expression keyword that matches a variable-length term such as an email address. For collation conditions that form a list of proper nouns, the collation condition storage means 400 can divide the collation conditions for each term length and uniquely determine the length for each collation condition ID. Therefore, the character string having the corresponding length can be replaced with the corresponding feature token from the matching condition ID matched by the matching means. For a term class having a variable-length term, a feature token can be inserted after the end position (hit position) without replacement.

  As a result, even when the starting position is unknown, the character string can be replaced, and the replacement process can be speeded up.

FIG. 8 is a diagram illustrating an example of collation conditions in which term lengths are divided in the second embodiment.
The matching condition ID is configured to include a term class field for identifying a term class and a character number field for identifying the number of characters. In FIG. 8, ID = 101, 102, 103 indicates that the term class field is 1, and the number of characters is 1, 2, 3, respectively. The replacement unit 1022 generates a feature token using the term class field (in this case, TOKEN_1), removes the character indicated in the character number field from the input character string, and inserts the feature token. For ID = 1100, the term class field is 11 and the character count field is 0. The number-of-characters field 0 indicates that it has a variable length. In this case, the replacement unit 1022 inserts a feature token (in this case, TOKEN_11) without deleting the matching character string from the input character string. In the example of FIG. 8, the 1st place and 10th place of the ID are the number-of-characters field, and the higher place is the term class field. However, field assignment is not limited to this.

  In this way, by providing character number information as part of the collation condition ID, it is possible to eliminate the complexity of managing the number of characters separately.

As described above, when the collation condition ID is provided with the number-of-characters information, it takes time and effort for the user to create the collation condition while paying attention to the number of characters. Therefore, the collation condition can be automatically configured.
FIG. 9 is a diagram showing an example of a configuration for automatically configuring the collation conditions in the second embodiment.
In the example of FIG. 9, a matching condition synthesis unit 106 is added. The matching condition synthesizing means 106 receives the matching condition as shown in FIG. 5 that is not divided for each number of characters, analyzes the regular expression keyword, and generates a matching condition divided for each number of characters. When the regular expression keyword is an enumeration of fixed-length characters such as a person name, a regular expression keyword divided for each number of characters is generated, and a new matching condition ID (from the number of characters of each regular expression keyword and the original matching condition ID ( With term class field and number of characters field). If the regular expression keyword has a variable length, 0 is entered in the number of characters field of the matching condition ID.

  As described above, by automatically generating the collation condition having the character number information, it is possible to save the user from creating the collation condition in consideration of the number of characters.

Embodiment 3 FIG.
In the second embodiment described above, even when the matching unit 1021 that outputs only the position in the input text of the character string that matches the matching condition is used, the feature token can be replaced. An embodiment for tokenizing the positional relationship between characteristic terms is shown below.

  In detecting personal information, a document having a structure in which a person's name and address, a telephone number, a mail address, and the like are arranged like a name list is often set as a detection target. The classification accuracy can be improved by extracting and learning the positional relationship between such characteristic terms.

  The preprocessing means 100 in the third embodiment extracts and stores the hit position in the input document of the characteristic term specified by the collation condition ID, and predetermines it in a sequence of two or more characteristic terms. When a sequence that matches the rule appears, the distance between the features is calculated from the stored hit position, and the feature distance token is generated by combining the rule ID and distance assigned to the corresponding rule. . The learning means 200 and the classification means 300 can learn and classify a document having a structure like a name list efficiently by learning the distance token between features in the same manner as other tokens.

FIG. 10 is a configuration diagram illustrating an example of an information search system according to the third embodiment.
In FIG. 10, in the preprocessing unit 100, a feature distance token generation unit 107 is added, and a rule 600 is further added.

FIG. 11 is a diagram illustrating an example of the rule 600 in the third embodiment.
One rule is composed of a set of a rule ID and a collation condition sequence. ID = 1 in FIG. 11 is a rule for detecting the case where the collation condition ID appears in the order of 1 and 11, that is, the case where the collation condition ID appears in the order of person name → mail address. When this is detected, it is possible to identify that rule ID = 1 as the inter-feature distance token, and the inter-feature distance token including this ID and the detected distance is used as the inter-feature distance token generation means 105. Generate with. For example, if the distance is 20, a token RULE_1_20 is generated.

  The collation condition sequence indicates the order of two or more collation conditions. In the example of ID = 3, three collation condition sequences are described. In this case, the distance portion may be, for example, RULE_3_5_6 if the distance between the collation condition IDs 1 and 10 is 5 and the distance between 10 and 12 is 6. The same applies to four or more sequences.

  An upper limit can be set for the notation of the distance. Usually, even in a directory or the like, characteristic terms that are more than a certain distance are likely to be irrelevant, and related characteristic terms are likely to be structurally adjacent. “Structural” means the structure of a document. When text is extracted, it is not necessarily adjacent as a character string, but is likely to be within a certain distance. Therefore, if the distance exceeds the condition, the influence is small even if ignored.

  Further, the distance is not necessarily an accurate distance, and the distance may be divided into several ranges and labeled. For example, the distance can be A if the distance is 5 or less, B if the distance is 6 or more and 10 or less, C if the distance is 11 or more and 20 or less, and otherwise. If the detected distance is 6, the token is represented as RULE_1_B. In this way, by collecting distances in a range, it is possible to efficiently learn with relatively few learning documents.

  When the collation condition stored in the collation condition storage unit 400 is composed of the built-in collation condition 4001 and the user-defined collation condition 4002, the rule 600 is associated with the built-in rule and the user-defined rule. This makes it possible to create a rule including a term class defined by the user.

  By the way, as described in the first embodiment, in the present invention, a sequence of 1 to N tokens including a feature token is learned or classified, but such a sequence is learned for an inter-feature distance token. Is meaningless. As one method for avoiding this, when the learning unit 200 and the classifying unit 300 detect the feature distance token, learning and classification are performed not by the sequence but by a single unit (N = 1).

Alternatively, as another method, when the preprocessing unit 100 outputs a distance token between features, N-1 dummy tokens are inserted before and after and N sequences are learned as they are in learning and classification. Can also be configured. For example, if N = 3, the token string is DUMMY, DUMMY, RULE_X_X, DUMMY, DUMMY, and the combination of N consecutive tokens is as follows:
(DUMMY, DUMMY, RULE_X_X)
(DUMMY, RULE_X_X, DUMMY)
(RULE_X_X, DUMMY, DUMMY)
It becomes. In this way, it is possible to avoid learning the relationship between adjacent feature distance tokens with respect to the feature distance token.

Embodiment 4 FIG.
In the third embodiment described above, the positional relationship between characteristic terms is converted into a distance token between features and added to the token string for use in learning / classification. Next, learning using only the distance token between features is performed. An embodiment in which the learning process is speeded up by performing the above will be described.

FIG. 12 is a configuration diagram illustrating an example of an information search system according to the fourth embodiment.
In FIG. 12, a token output control means 106 is added to the preprocessing means 100. The token output control means 108 has two operation modes (mode 1 and mode 2). When mode 1 is set, as in the third embodiment, an inter-feature distance token and, if necessary, a dummy token are output at the end of the token string composed of characteristic tokens and non-characteristic tokens. When mode 2 is set, output of feature tokens and non-feature tokens is suppressed, and only feature distance tokens are output.

  The operation mode is set in an initial setting file or registry, or given as an execution command parameter when the system is executed. The token output control means 108 is set in the operation mode by these setting methods at the time of activation, and thereafter operates in the operation mode.

  When the mode 1 is set, the token output control unit 108 outputs the output from the non-feature token extraction unit 103 and the output from the feature distance token generation unit 107 as they are.

  When the mode 2 is set, the token output control unit 108 does not output the token sequence output from the non-feature token extraction unit 103 (sequence including the feature token and the non-feature token), and generates an inter-feature distance token. The token sequence generated by the means 107 (sequence consisting of distance tokens between features) is output. This token string may include a dummy token. In the case of operating in mode 2, since it is set that learning of the relationship between adjacent tokens is not necessary, the inter-feature distance token generating unit 107, the learning unit 200, and the classification unit 300 perform N By operating at = 1, it is possible to operate more efficiently without unnecessary learning.

  As described above, by making it possible to select the operation mode of the learning process, it is possible to provide a flexible system capable of selecting the trade-off between the classification accuracy and the classification speed according to the application type.

Embodiment 5 FIG.
In the third embodiment, all tokens are handled equally. Next, an embodiment in which the token weighting can be set will be described.

FIG. 13 is a diagram illustrating an example of a token weighting setting method according to the fifth embodiment.
In FIG. 13, the type indicates whether it is a feature token (TOKEN) or an inter-feature distance token (RULE). The classification means 300 has this weight setting information, and uses the classification probability for each category of the input document calculated from the feature token and the non-feature token and the classification probability calculated from the inter-feature distance token as the weight. According to the allocation, the final classification probability is calculated and the category is determined.

In the classifying means 300 according to the fifth embodiment, different classification probabilities Pct (probability that the input document is classified into category c by the feature token) and Pcr (input document by the inter-feature distance token) depending on the token type. Is probable to be classified into category c). At this time, the probability Pc that the input document is classified into the category c is calculated by the following formula, and the category is determined by this.
Pc = Pct · Wt + Pcr · Wr (Wt + Wr = 1)

  This makes it possible to finely tune the classification accuracy according to the type of application.

1 is a configuration diagram illustrating an example of an information search system in Embodiment 1. FIG. 3 is a flowchart showing an operation of the information search system in the first embodiment. 5 is a flowchart illustrating an operation in a classification stage in the first embodiment. 4 is a flowchart showing the operation of preprocessing means 100 in the first embodiment. In Embodiment 1, it is a figure which shows the example of collation conditions. In Embodiment 1, it is a figure which shows the example of the token row | line | column produced | generated by the pre-processing means 100 using the collation conditions of FIG. In Embodiment 2, it is a figure which shows an example of the structure in connection with the feature token extraction means 102 at the time of using DFA. In Embodiment 2, it is a figure which shows the example of the collation conditions which divided | segmented the length of the term. In Embodiment 2, it is a figure which shows the example of the structure which automatically comprises collation conditions. FIG. 10 is a configuration diagram illustrating an example of an information search system in a third embodiment. In Embodiment 3, it is a figure which shows the example of the rule 600. FIG. FIG. 10 is a configuration diagram illustrating an example of an information search system in a fourth embodiment. In Embodiment 5, it is a figure which shows the example of the weight setting method of a token.

Explanation of symbols

100 Pre-processing means, 101 Text extraction means, 102 Feature token extraction means, 1020 Input character string, 1021 Collation means, 1022 Replacement means, 1023 Output character string, 103 Non-characteristic token extraction means, 104 State transition table generation means, 105 State Transition table, 106 collation condition generation means, 107 feature distance token generation means, 108 token output control means, 200 learning means, 201 learning frequency calculation means, 202 learning frequency storage means, 300 classification means, 301 classification frequency calculation means 302 classification probability calculation means, 303 category determination means, 400 collation condition storage means, 4001 built-in collation conditions, 4002 user-defined collation conditions, 501 learning document, 502 classification target document, 600 rules.

Claims (15)

A matching condition storage means for storing a matching condition between a character string and a keyword and a feature token for identifying the matching condition;
Based on the matching condition and the feature token stored in the matching condition storage unit, the character string of the learning document classified in advance by category is matched with the keyword to correspond to the matched matching condition. The first feature token is extracted in association with the category, and the character string of the classification target document classified by the category based on the matching condition and the feature token stored in the matching condition storage unit And a feature token extracting means for extracting a second feature token corresponding to the matched matching condition;
A first non-feature token obtained by dividing a character string of the learning document from which the first feature token has not been extracted in character units is extracted in association with the category, and the second feature token is extracted. Non-characteristic token extracting means for extracting a second non-characteristic token obtained by dividing the character string of the classification target document that has not been divided into character units;
Learning means for calculating an appearance frequency of a first token string composed of the first characteristic token and the first non-characteristic token in association with the category as a learning frequency;
The classification probability indicating the similarity between the appearance frequency of the second token sequence composed of the second characteristic token and the second non-characteristic token and the learning frequency calculated by the learning means is the category. Classifying means for separately calculating and classifying the classification target document into the category in which the classification probability exceeds a predetermined threshold;
Information retrieval system with The learning means performs the first feature token only when the first feature token is included in a first token chain composed of consecutive n (n is a natural number) tokens in the first token sequence. Calculating the occurrence frequency of the token chain as the learning frequency,
The classifying means performs the second feature token only when the second feature token is included in a second token chain composed of consecutive n (n is a natural number) tokens in the second token sequence. The information search system according to claim 1, wherein the classification probability indicating a similarity between a token chain appearance frequency and the learning frequency calculated by the learning unit is calculated for each category. The collation condition storage means stores a priority setting collation condition in which a priority is set for the collation condition,
The information search system according to claim 1 or 2, wherein the feature token extraction unit extracts the first or second feature token based on the priority setting collation condition and the feature token. The collation condition storage means stores a built-in collation condition in which the collation condition is defined in advance and a user-defined collation condition defined by the user,
The information according to any one of claims 1 to 3, wherein the feature token extraction unit extracts the first or second feature token based on the built-in matching condition, the user-defined matching condition, and the feature token. Search system. The matching condition storage means stores a regular expression matching condition in which the matching condition is defined by a regular expression,
The information search system according to any one of claims 1 to 4, wherein the feature token extracting unit extracts the first or second feature token based on the regular expression matching condition and the feature token. The matching condition storage means stores, in the matching condition, a term class matching condition in which a plurality of the keywords are defined for each term class indicating the classification of the keyword,
The information search system according to claim 1, wherein the feature token extraction unit extracts the first or second feature token based on the term class matching condition and the feature token. The collation condition storage means stores an ID addition collation condition in which a collation condition ID that is an identification number of the regular expression collation condition is given to the regular expression collation condition,
The feature token extracting means collates the character string in the document with the keyword based on the ID addition collation condition, and matches the collation condition ID of the matched ID provision collation condition and the end position of the matched character string. Matching means for outputting a hit position indicating
Replacement that replaces the character string that matches the matching condition with the first or second feature token corresponding to the ID addition matching condition based on the matching condition ID and the hit position output by the matching unit The information search system according to claim 5, further comprising: means. 8. The information retrieval system according to claim 7, wherein the collation means performs character string collation using a deterministic finite automaton. 8. The information retrieval system according to claim 7, wherein the collation means performs character string collation using a nondeterministic finite automaton. The collation condition storage means includes a term class field that holds the identification number of the term class in the collation condition ID and a character information field that holds the number of characters of the keyword in the ID addition collation condition. Remember
The replacement means removes a character string corresponding to the number of characters held in the character number field of the collation condition ID of the field information addition collation condition from the front of the hit position, and the first or second corresponding to the collation condition ID. The information search system according to claim 7, wherein two feature tokens are inserted. The field information addition collation condition includes a plurality of fixed-length keywords,
The field information addition collation condition is divided for each fixed-length keyword, and the field information addition collation conditions in which the number of characters of the fixed-length keyword is the same among the divided field information addition collation conditions are collectively added as new field information. The information search system according to claim 10, further comprising collation condition synthesis means for synthesizing the collation conditions. The collation condition storage means stores a variable length information setting collation condition in which variable length information indicating variable length is set in the character number field corresponding to the variable length keyword when the keyword has a variable length. And
When the variable length information is set in the number-of-characters field of the variable length information setting collation condition, the replacement means does not remove the character string matched with the variable length keyword from before the hit position, and The information search system according to claim 10, wherein the first or second feature token is inserted immediately after a position. A rule that stores a matching condition sequence that defines an order relationship among a plurality of matching condition IDs and a rule ID that is an identifier of the matching condition sequence;
Analyzing the collation condition ID and the hit position output by the collation means, detecting a chain of the collation condition IDs that appear in an order relationship that matches the collation condition sequence stored in the rule, An inter-feature distance token generating means for generating an inter-feature distance token that is an identifier combining the distance between the hit positions in the chain of matching condition IDs and the rule ID of the rule that matches the matching condition sequence;
The learning means learns the learning frequency that is the appearance frequency for each category based on the appearance frequency of the feature distance token,
10. The classification means calculates a classification probability into the category based on an appearance frequency of the feature distance token, and classifies the classification target document into the category in which the classification probability exceeds a predetermined threshold. The information search system described in any of the above. The feature distance token generated by the feature distance token generation unit, the feature token extracted by the feature token extraction unit, and the non-feature token extracted by the non-feature token extraction unit are selectively selected. If the operation mode condition for outputting to the feature token is set, and the operation mode condition is set to suppress the output of the feature token and the non-feature token, control is performed so that only the distance token between features is output. The information retrieval system according to claim 13, further comprising a token output control unit. The classification means includes a first classification probability for the category calculated based on the feature token and the non-feature token, and a second classification for the category calculated based on the inter-feature distance token. The information search system according to claim 13, wherein the classification target document is classified into the category by using a third classification probability obtained by weighting and combining the probabilities.

Images