JP3609252B2 - Automatic character string classification apparatus and method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a method and apparatus for classifying an arbitrary character string extracted from a text into a specialized expression used in a specific field and a general expression used regardless of the field.
[0002]
[Prior art]
In machine translation systems that automatically translate documents that have been converted into data that can be used by information processing devices, and systems that search for documents that use relevant keywords by searching for documents, there is a fixed amount that exists in the document. It is necessary to extract a character string having the meaning of.
Reference 1 “Information Processing Society of Japan Vol.93, No.61 (93-NL-96-1)”
In the technique disclosed in Document 1, a character string of all lengths starting with all characters included in a text document (if the text length is 1 character i (1 ≦ i .Ltoreq.L) leading character string of length n (1.ltoreq.n.ltoreq.Li) (hereinafter referred to as "n-gram character string") is extracted, and the number of appearances is counted to be processed. A character string is extracted from a text document. This method is characterized in that it is not necessary to perform morphological analysis or use a dictionary, and a character string can be extracted only by statistical processing. However, since character strings appearing in text are exhaustively extracted based on the number of characters and the number of appearances, there is a problem that character strings that do not make sense as a language (hereinafter referred to as fragment character strings) are mixed.
[0003]
By the way, a continuous character string (hereinafter referred to as “expression”) that should be recognized as a group like a word or phrase has a high appearance frequency in the text, and various words appear before and after that. There is.
Reference 2 Information Processing Society of Japan Research Report Vol. 95, no. 110 (95-NL-110-11)
The technique disclosed in Document 2 uses the above feature to extract an appropriate continuous character string by calculating a variance value of a character immediately before an arbitrary character string and a variance value of a character immediately after the arbitrary character string. It is about. In this method, only a meaningful character string, that is, “expression” is extracted by removing a character string having a low variance value as a fragmented character string from the character string extracted by the technique of Document 1.
[0004]
[Problems to be solved by the invention]
In general, there are various levels of “expression” (for example, technical expressions such as technical terms and proper nouns, general expressions such as general terms and idiomatic phrases).
However, in the conventional character string extraction method, fragmentary character strings included in the character string extracted from the document are excluded and a meaningful character string “expression” is extracted, but the character strings of various levels of expression are extracted. Will be extracted in a mixed state.
Therefore, when actually using the extraction result, there is a problem that it must be further classified according to the application.
[0005]
From this point, the present invention can arbitrarily classify an arbitrary character string extracted from a text document into a specialized expression used in a specific field and a general expression used regardless of the field. The purpose is to provide an automatic classification device.
[0006]
[Means for Solving the Problems]
In order to solve the problem, the present invention provides a document storage unit that stores a plurality of documents described in a natural language, a character string extraction unit that extracts a character string from an arbitrary document among a plurality of documents, and a character The document importance calculation means for calculating the importance of the character string extracted by the string extraction means in the extracted document as the importance in the document, and the entire character string extracted by the character string extraction means. Inter-document importance calculation means for calculating the importance of documents as inter-document importance, and string importance calculation for calculating the importance of character strings extracted based on intra-document importance and inter-document importance Means, and character string classification means for classifying the extracted character strings based on the character string importance obtained by the character string importance calculation means, the character string classification means for each of the extracted character strings, Characters added to the string A first classification unit that compares the importance level with a predetermined threshold and classifies all the extracted character strings into a general expression used regardless of a specific field or a specialized expression used in a specific field, and a first classification unit When the character string classified into the specialized expression is divided and each component of the divided character string exists in the first classification result, the character string before the division is divided by the combination of the components of the divided character string. There is provided an automatic character string classification device having a second classification unit for reclassifying a classification into a general technical expression or a general expression that is a technical expression used with the general expression from the technical expression.
[0009]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an automatic character string classification device according to an embodiment of the present invention will be described in detail with reference to the drawings.
(A) First embodiment
FIG. 1 is a block diagram showing a first embodiment of the automatic character string classification apparatus of the present invention, which is realized on an information processing apparatus such as a workstation or a personal computer.
[0010]
In FIG. 1, the automatic character string classification device includes an input / output device 1, a processing device 2, and a storage device 3. The input / output device 1 has functions for inputting text, displaying extraction results, and the like. The processing device 2 has a function of extracting a continuous character string and executing various processes for classifying the result. The storage device 3 has a function of storing the input text and the processing result of each stage.
[0011]
Further, the input / output device 1 includes an input unit 11 and an output unit 12. The input unit 11 has a function of inputting a text document serving as data, and is composed of, for example, a keyboard. The output unit 12 has a function of displaying the extracted character string or the like, and is configured by, for example, a display or a printer.
[0012]
The processing device 2 includes a character string extraction unit 21, an importance calculation unit 22, and a character string classification unit 23. The character string extraction unit 21 reads a document file, which will be described later, and extracts an arbitrary n-gram character string included in the document file. As this extraction method, for example, the character string extraction method shown in Document 1 is used. Moreover, you may perform the process which removes a fragmentary character string like literature 2 with respect to an extraction result.
It should be noted that any method that can extract a character string such as a word, a phrase, or a clause may be used regardless of the method described in the prior art.
[0013]
The importance calculation unit 22 calculates the importance of the n-gram character string extracted by the character string extraction unit 21 in the document and between the documents, and the final character string obtained by weighting the character string from the two importances. Is obtained (hereinafter referred to as character string importance). The character string classification unit 23 is used in specialized expressions used in a specific field or in a normal document regardless of the field based on the importance assigned to each character string extracted by the importance calculation unit 22 Are classified into general expressions. Details of the importance calculation unit 22 and the character string classification unit 23 will be described later.
[0014]
The storage device 3 includes a document file 31, an intra-document importance table 32, an inter-document importance table 33, a keyword table 34, and a buffer 35.
The document file 31 stores a text document serving as data input from the input unit 12 as a document file. In the present embodiment, there are a plurality of document files 31, and the field (content) of the document file 31 is not limited, and the document file 31 may be of a different field for each document file.
[0015]
FIG. 2 shows an example of the document importance level table 32. The in-document importance level table 32 includes a character string storage unit that stores the n-gram character string generated from the document file 31 by the character string extraction unit 21, and an in-document importance level that stores the importance level of the character string in the document. Character string importance storage that stores the weight of importance in the storage unit and the importance of the character string in the document and the importance of the character string in the document in addition to the importance of the character string between documents Part.
[0016]
The in-document importance storage table 32 corresponds to each of the plurality of document files 31.
FIG. 3 shows an example of the inter-document importance table 33. The inter-document importance table 33 is extracted in a character string storage unit that stores an n-gram character string generated from the document file 31 by the character string extraction unit 21 and one document file 31 among the plurality of document files 31. The generated n-gram character string has an appearance document number storage unit that stores the number of appearances in other document files 31 and an inter-document importance storage unit that stores the importance of character strings in the plurality of document files 31. . In this embodiment, it is composed of one table.
[0017]
The inter-document importance storage table 33 is created from the character strings stored in the plurality of intra-document importance storage tables 32 generated for each of the plurality of document files 31.
[0018]
The keyword table 34 stores character strings (keywords) that are determined to be important in the document file 31 by the character string classification unit 23 and extracted.
FIG. 4 shows an example of the keyword table 34. The keyword table 34 is important in the document file 31 by the character string storage unit that stores the extracted character string and the character string classification unit 23 that is stored in the character string importance storage unit of the in-document importance degree table 32. An importance level storage unit for storing the character string (keyword) determined and extracted is provided. The keyword table 34 corresponds to each of the plurality of document files 31.
[0019]
The buffer 35 stores values obtained in the course of each process, intermediate results, and the like.
[0020]
Here, the importance calculator 22 will be described in detail.
FIG. 5 is a configuration diagram illustrating the function of the importance degree calculation unit 22.
The importance calculation unit 22 further includes an in-document importance calculation unit 221, an inter-document importance calculation unit 222, and a character string importance calculation unit 223. Processing is performed in association with the intra-document importance table 32 and the inter-document importance table 33, respectively.
[0021]
The in-document importance calculation unit 221 reads one document file 31 from a plurality of document files 31 stored in the storage device 2, and stores the n-gram character string extracted from the document file 31 in the document file 31. Calculate importance. As the importance in the document, for example, an appearance frequency at which a certain character string appears in the document file 31 is used. The calculated in-document importance is stored in the in-document importance storage section of the in-document importance table 32. This in-document importance calculation process is repeated for all document files 31.
[0022]
The appearance frequency can be obtained at the same time when the n-gram character string is extracted by the methods of Documents 1 and 2.
[0023]
The inter-document importance calculation unit 222 calculates the importance of the extracted n-gram character string in the plurality of document files 31 as a whole.
First, from the in-document importance level table 32 corresponding to the document file 31, one record is read and the number of document strings 31 in which the n-gram character string appears is counted. This count is performed for each document importance table 32 corresponding to a plurality of document files 31, and the total number of document files 31 in which a certain character string appears is defined as the number of appearing documents. When the number of appearing documents is large, it can be seen that the character string is a character string that appears frequently regardless of the specific document file 31, and conversely, when the number of appearing documents is small, the character string that appears only in the specific document file. It can be seen that it is.
[0024]
The obtained number of appearing documents is stored in the appearing document number storage section of the inter-document importance table 33.
Furthermore, the importance between documents of each character string is calculated. Here is a string _k The smaller the number of document files 31 including, the greater the importance between documents. As this value, for example, an inverse document frequency is used. Inverse document frequency is obtained by multiplying the reciprocal of the number of documents including a certain character string by the total number of documents. String _k Inverse document frequency of idf _k N is the number of document files, and n is the number of document files 31 including the extracted character strings (the number of appearing documents). _k Then, it is obtained by the following formula.
[0025]
(Formula 1) idf _k = Log (N / n _k ) Where idf _k Is n _k = 1 when the maximum value is taken, n _k When N = N, take the minimum value (= 0) _k The number varies depending on the number of document files 31 including. The degree of importance between documents takes a smaller value as the number of document files 31 including the character string k increases. On the other hand, when the document file 31 includes only a small number of character strings k, the value is large.
[0026]
This inter-document importance calculation process is repeated for all n-gram character strings stored in the character string storage section of the inter-document importance table 33.
The obtained inter-document importance is stored in the inter-document importance storage section of the inter-document importance table 33.
[0027]
The character string importance calculation unit 223 reads the document importance from the document importance table 32 and the document importance from the document importance table 33, and calculates the importance of the character string in the document between documents. The importance of the character string weighted by the importance is calculated. Here, a character string that appears frequently in a specific document file 31 but hardly appears in other document files 31, that is, a character string related to a specific field is set so as to increase in importance. It is set so that the value of the character string appearing in many document files 31 regardless of the specific document file 31, that is, appearing regardless of the specific field, becomes low. As a calculation method, for example, a product of the importance level in a document and the importance level between documents is used.
[0028]
Where the document file _i 31 (1 ≦ _i ≦ N) Character string extracted from _k String importance of W _ik And String _k Document files _i The importance level in the document representing the appearance frequency at 31 is expressed as tf. _ik And the string _k The inter-document importance level indicating the importance level between the documents is idf _k Then, it is obtained by the following formula 2. (Formula 2) W _ik = Tf _ik × idf _k String importance is string _k Idf because it only appears in document files with few _k Is high and the string _k Tf appears in the extracted document file, so tf _ik A high value is obtained when the value of is high.
[0029]
If a value with high importance is obtained, the string _k Are important keywords in the document file. The obtained character string importance is stored in the character string importance storage section of the in-document importance table 33. The calculation processing of the importance level of the character string is repeatedly performed for all the character strings in the in-document importance level table 32.
[0030]
Next, the character string classification unit 23 will be described in detail.
The character string classification unit 23 reads one record at a time from the in-document importance level table 32 and converts the character string into two types of specialized expression and general expression based on the importance of the character string stored in the character string importance storage unit. Classify and extract only specialized expressions.
[0031]
The classification method is performed, for example, by setting a threshold value in advance and comparing it with the threshold value. Here, if the threshold is T, and the importance of the character string is greater than the threshold T, it is determined that the character string is highly important. The character string determined to be important after the comparison with the threshold value is stored in the keyword table 34 together with the character string and the importance of the character string. This classification process is repeated for all the in-document importance level tables 32 to classify the character strings as important in the document file 31.
[0032]
FIG. 6 is a flowchart showing the operation of the automatic character string extraction apparatus of the present invention.
Here, it is assumed that a text document as data is input from the input unit 11 as a document file to the document file 31 of the storage unit 3 and a plurality of document files 31 are stored. It is assumed that the total number of document files 31 is known in advance.
[0033]
First, the character string extraction unit 21 reads one document file from a plurality of document files from the document file 31 (step 1), extracts an n-gram character string from the read document file, and stores the importance level table in the document. It is stored in 32 character string storage units. (Step 2).
[0034]
Frequency at which the extracted n-gram character string appears in the document file The It is obtained and stored in the in-document importance storage section of the in-document importance table 32. (Step 3).
Here, it is determined whether there is a document file 31 that has not been subjected to character string extraction. If there is an unprocessed document file 31, the process returns to step 1, and if it is the last document file 31, the process proceeds to the next step 4 (step 4).
[0035]
Next, using the in-document importance obtained for each character string, the inter-document importance of the character string is obtained.
With reference to all the intra-document importance tables 32 generated in the process up to step 3, the inter-document importance of each character string is calculated and stored in the inter-document importance table 33 (step 5).
[0036]
Next, using the in-document importance and the inter-document importance, the importance of the character string weighted to the extracted n-gram character string is calculated and stored in the character string importance storage unit of the in-document importance table 33. Store. (Step 6).
Finally, the character strings are classified by comparing the importance of the weighted character strings with a preset threshold value (step 7).
[0037]
Here, step 5 for performing the inter-document importance processing will be described in detail with reference to the flowchart of FIG.
First, one record is read from the in-document importance table 32 (step 51). It is determined whether or not the read character string is already stored in the character string storage unit of the inter-document importance degree table 33. If it is stored, the process proceeds to step 54.
[0038]
If not stored, the process proceeds to step 53, where the character string is stored in the character string storage section of the character string importance table 33, and then the process proceeds to step 54 (steps 52, 53). In step 54, the number of appearance documents of the character string is increased by 1 (step 54). It is determined whether or not there is an unprocessed record in the in-document importance table 32. If there is an unprocessed record, the process returns to step 51, and if not, the process proceeds to step 56 (step 55).
[0039]
Next, it is determined whether or not the processing up to step 55 has been executed for all in-document importance tables. If there is still an unprocessed in-document importance table 32, the process proceeds to step 57, and if there is no unprocessed in-document importance table 32, the process proceeds to step 58 (step 56). If there is an unprocessed in-document importance level table 32, the process moves to the next in-document importance level table 32, and the processing from step 51 to step 55 is performed for all the in-document importance level tables 32 (step 57). .
[0040]
Next, one record is read from the inter-document importance table 33 (step 58). The importance of the character string between the documents is calculated using the number of appearing documents and the total number of document files (step 59), and the obtained inter-document importance is stored in the inter-document importance storage section of the inter-document importance table 33. (Step 511). Next, it is determined whether or not processing has been executed for all records existing in the inter-document importance table 33. If there is still an unprocessed record, the process proceeds to step 58, and if there is no unprocessed record, the process is terminated (step 512).
[0041]
Next, step 6 for performing the character string importance process will be described in detail with reference to the flowchart of FIG.
[0042]
First, one record is read from the in-document importance table 32 (step 61). The inter-document importance corresponding to the read character string is referred to from the inter-document importance table 33, the importance of the weighted character string is calculated (step 62), and the obtained character string importance is subjected to the current processing. Is stored in the character string importance storage section of the in-document importance storage table 32 (step 63).
[0043]
Here, it is determined whether or not there is an unprocessed record in the in-document importance level table 32. If there is an unprocessed record, the process returns to step 61, and if not, the process proceeds to step 65 ( Step 64). Next, the processing up to step 64 is executed on all the document importance level tables 32 to determine whether the weighted character string importance level has been calculated.
[0044]
If there is still an unprocessed in-document importance table 32, the process proceeds to step 66. If there is no unprocessed in-document importance table 32, the process proceeds to step 7 in FIG. 6 (step 65). If there is an unprocessed in-document importance level table 32, the process moves to the next in-document importance level table 32, and the processes from step 61 to step 65 are performed for all the in-document importance level tables 32 (step 66).
[0045]
Next, step 7 for performing classification processing of weighted character strings will be described in detail with reference to the flowchart of FIG.
[0046]
First, one record is read from the intra-document importance table (step 71). The read character string importance level is compared with a preset threshold value. If the character string importance level is larger than the threshold value, the process proceeds to step 73. If the character string importance level is smaller than the threshold value, the process proceeds to step 74 (step 72). The character string importance degree determined to be larger than the threshold value and the character string corresponding to the character string importance degree are assigned to the importance degree storage unit and the character of the keyword table 34 corresponding to the in-document importance degree table 32. Store each in the column storage.
[0047]
Here, it is determined whether or not there is an unprocessed record in the in-document importance table 32. If there is an unprocessed record, the process returns to step 71, and if not, the process proceeds to step 75 ( Step 74). Next, the processing up to step 74 is executed on all the document importance tables 32, and it is determined whether or not the character strings have been classified.
[0048]
If there is still an unprocessed in-document importance table 32, the process proceeds to step 76. If there is no unprocessed in-document importance table 32, the process is terminated. If there is an unprocessed in-document importance level table 32, the process moves to the next in-document importance level table 32, and the processing from step 71 to step 75 is performed for all the in-document importance level tables 32 (step 76).
[0049]
Next, the processing steps of the present invention will be specifically described with reference to actual cases and the flowcharts of FIGS.
[0050]
It is assumed that N document files 31 are stored in the storage device 3.
FIG. 10 shows an example of the document importance level table 32.
[0051]
First, the document file _i 31 (1 ≧ _i ≥N) content, n-gram character string is extracted, and document file of the extracted character string _i The appearance frequency in 31 is obtained and stored in the corresponding document importance table 32 (steps 1, 2, and 3 in FIG. 6).
[0052]
FIG. 10 shows an in-document importance table 32 that stores extracted character strings and appearance frequencies (in-document importance). The processing of steps 1, 2, and 3 in FIG. _i 31 for each. After this process is finished, the document file _i 31 (1 ≦ _i In correspondence with ≦ N), an in-document importance table i32 (1 ≦ i ≦ N) is created.
[0053]
Next, the importance level between documents of each character string is calculated and stored in the inter-document importance level table 33. FIG. 11 shows an example of the inter-document importance table. Here, the importance level table in the document of FIG. _i 32 as the processing target, read character string _k It is determined whether “in” is present in the other document importance level table 32. If it exists, it is counted as the number of occurrences in other documents, and the total value is counted between documents. _i It is stored in the 33 appearing document number storage unit. Here, the number of occurrences n in other documents _k = 43. (FIG. 7, steps 51-57).
[0054]
Next, the importance between documents is obtained. Number of document files 31 N = 50, character string _k = "In", number of appearance documents n _k = 43 is a document file according to equation (1) _i Character string in 31 _k Inter-document importance idf _k = Log (50/43) = 0.15 is obtained, and 0.15 is stored in the inter-document importance storage unit (steps 58 to 511 in FIG. 7). FIG. 11 shows the contents of the inter-document importance table 33 after the inter-document importance calculation is performed in the intra-document importance table 32 shown in FIG.
[0055]
Next, the character string importance degree obtained by weighting the n-gram character string is obtained. Document importance table in FIG. _i Character string from 31 _k = Importance tf in the document of “de” _ik = 11 is read. Further, the character string from the inter-document importance table 33 of FIG. _k = Inter-document importance idf for “de” _k = 0.15 is read. Document file according to equation (2) _i Character string in 31 _k Importance W _ik = 11 × 0.15 = 1.65, and the obtained value is stored in the character string importance storage section of the in-document importance table i31 (steps 61 to 63 in FIG. 8).
[0056]
This processing is performed in the document importance table. _i It repeats with respect to all the character strings of 33 (steps 65 and 66). FIG. 12 shows the document importance table in FIG. _i 32 shows the contents after weighting the character string using the inter-document importance table 33 of FIG. Although not shown, document importance table _i If there is an in-document importance table 32 other than 32, the process moves to the next table and the same processing is repeated.
[0057]
Finally, an n-gram character string classification process is performed.
Document importance table in FIG. _i Character string from 32 _k = "In" and character string _k = Weighted string importance W corresponding to “in” _ik = 1.65 is read and compared with a predetermined threshold value T = 10. If the importance level of the character string is larger than the threshold value T, it is registered as a keyword. However, since the importance level 1.65 of the character string is smaller than the threshold value 10, the keyword table _i 34 is not stored (steps 71 to 74 in FIG. 9).
[0058]
This processing is performed in the document importance table. _i This is repeated for all the records in 32 (steps 75 and 76 in FIG. 9).
[0059]
If the same processing is performed subsequently, the character string _k = String importance W for “de” and “no” _ik Is 0 and is smaller than the threshold value 10, so it is not registered as a keyword. In contrast, the string _k = "Network" string importance W _ik Is 39.33, which is larger than the threshold value 10, so that the character string “network” and the character string importance are indicated in the keyword table. _i 34.
[0060]
FIG. 13 shows the importance level table in the document of FIG. _i 32 shows the contents after performing the classification process with the threshold T = 10.
The above processing is performed on a plurality of document files (1 to N). As a result, corresponding keywords 34 are created for all the document files 31.
[0061]
<Effect of the first embodiment>
According to the first embodiment of the present invention, n-gram character strings extracted from text can be classified into specialized expressions and general expressions. The specialized expression and the general expression in each document can be relatively classified by taking into consideration the importance between documents rather than judging only by the importance in the document. In other words, even if a character string has a low appearance frequency in a document, if it is judged to be highly specialized (if it appears only in a specific document), the value of the specialized expression will be high and be registered as a keyword. Can do.
[0062]
Also, appropriate classification can be performed according to the contents of the prepared document file. For example, in the first embodiment, the character string “network” appears only in a small number of document files, so it can be determined that the expertise in the document is high, and can be classified as a specialized expression. However, if all the document files are network-related papers or the like, the importance of the character string “network” becomes low and cannot be extracted. This feature is effective when extracting keywords to be used in the keyword search device.
[0063]
(B) Second embodiment
FIG. 14 is a block diagram showing a second embodiment of the automatic character string classification device of the present invention. In the second embodiment, blocks having the same functions as those in the first embodiment are given the same numbers, and only blocks different from the first embodiment in the second embodiment will be described in detail.
[0064]
The processing device 2 includes a character string multiple classification unit 24 instead of the character string classification unit 23 in the first embodiment.
[0065]
The character string multiple classifying unit 24 extracts the extracted n-gram character string based on the importance assigned to each character string by the importance calculating unit 22, or a combination of technical expression, general expression, or general expression and technical expression. These are classified into three types.
[0066]
First, one record is read from the in-document importance table 32, and character strings are classified based on the importance of the character string stored in the character string importance storage unit.
[0067]
The classification method is performed, for example, by setting a threshold value in advance and comparing it with the threshold value. Here, when the threshold value is T, when the importance level of the character string is greater than the threshold value T, the character string is determined to have a high importance level, and a classification code (DC = domain dependent collocation; When the technical expression DC) is assigned and smaller than the threshold value T, a classification code (GC = general association, hereinafter referred to as a general expression GC) representing the general expression is assigned.
[0068]
In this process, a classification code of either the professional expression DC or the general expression GC is assigned to all character strings, and then a classification code is reassigned in consideration of the arrangement of characters based on the classification code. Here, in addition to the technical expression DC and the general expression GC, when the expression is a combination of these two expressions, a classification code (CGD = combination of general and domain dependent collocation, hereinafter referred to as a general / special expression). CGD).
[0069]
After the classification code assigning process is completed, the character string and the classification code of the character string are stored in the character string classification table 36. This classification code assigning process is repeated for all the document importance tables 32 to classify the character strings in the document file 31.
[0070]
It should be noted that the general / special expression CGD constitutes the general / special expression CGD in the order of the special expression DC / general expression GC or in the order of the general expression GC / special expression DC regardless of the combination order of the two expressions. To do. Further, the combination of the special expression DC and the general expression / special expression CGD and the combination of the general expression GC and the general expression / special expression CGD also constitute the general / special expression CGD. In this case as well, the order of the two expressions does not matter.
[0071]
The storage device 3 includes a character string classification table 36 instead of the keyword table 34 in the first embodiment.
[0072]
The character string classification table 36 stores character strings (keywords) that are determined to be important in the document file 31 by the character string multiple classification unit 24 and extracted, and a classification code assigned to each character string.
[0073]
FIG. 16 shows an example of the character string classification table 36. The character string classification table 36 includes a character string storage unit that stores the extracted character strings, and a classification code storage unit that stores the classification codes assigned to the character strings by the character string multiple classification unit 24. The character string classification table 36 corresponds to each of a plurality of document files.
[0074]
Next, the character string multiple classification process, which is a feature of the second embodiment, will be described using a flowchart. Processes other than the character string multiple classification process in the present invention are the same as those in the first embodiment.
[0075]
FIG. 15 is a flowchart showing the operation of the character string plural classification unit.
The symbol “+” described in step 720 and the like in FIG. 15 represents a combination of character string elements. For example, “GC + DC” represents a combination of a character string that is an element of general expression and a character string that is an element of specialized expression. The order of classification codes is not related. For example, when “GC + CGD” is described, the order of arrangement of “CGD + GC” is also included.
[0076]
First, one record is read from the in-document importance table 32 (step 711). The read character string importance level is compared with a preset threshold value. If the character string importance level is larger than the threshold value, the process proceeds to step 713, and if it is smaller than the threshold value, the process proceeds to step 714 (step 712). If it is determined that the value is larger than the threshold value, the specialized expression DC is assigned to the character string (step 713).
[0077]
If it is determined that the character string is smaller than the threshold, the general expression GC is assigned to the character string, and the process proceeds to step 715 (step 714). The character string to which the classification code is assigned and the classification code are respectively stored in the character string storage unit and the classification code storage unit of the character string classification table 36 corresponding to the in-document importance level table 32 from which the character string has been read (step 715). ).
[0078]
Here, it is determined whether or not there is an unprocessed record in the in-document importance table 32. If there is an unprocessed record, the process returns to step 711, and the processing from step 711 to step 715 is repeated. A classification code of either the professional expression DC or the general expression GC is assigned to all the character strings in the document importance level table 32. If there is no unprocessed record, the process proceeds to step 717 (step 716). In step 717, the flag is set to 0 (step 717).
[0079]
Next, one record is read from the character string classification table 36 (step 718). It is determined whether the character string element of the read record is “general expression GC” or “general / specialized CGD”. If it corresponds to one of the two expressions, the process proceeds to step 725, and if not, the process proceeds to step 720 (step 719).
[0080]
Next, it is determined whether the element of the character string is “combination of general expression GC and general expression GC”. If this combination is applicable, the process proceeds to step 721, and if not, the process proceeds to step 722 (step 720). If this combination is applicable, the general expression GC is assigned to the character string (step 721), and the classification code stored in the classification code storage unit of the character string classification table 36 is replaced with the general expression GC (step 729). The flag is set to 1 (step 724), and the process proceeds to step 725.
[0081]
Furthermore, the element of the character string is “combination of general expression GC and technical expression DC” or “combination of general expression GC and general / special expression CGD” or “combination of technical expression DC and general / special expression CGD”. Judge if there is. If one of these three expressions is applicable, the process proceeds to step 723, and if not, the process proceeds to step 725 (step 722).
[0082]
When these three expressions are satisfied, the general / specialized expression CGD is assigned to the character string (step 723), and the classification code stored in the classification code storage unit of the character string classification table 36 is added to the general / specialized expression CGD. To substitution (step 729), the flag is set to 1 (step 724), and the process proceeds to step 725.
[0083]
Here, it is determined whether or not there is an unprocessed record in the in-document importance level table 32. If there is an unprocessed record, the process returns to step 718, and the processing from step 718 to step 725 is repeated. A classification code of either the professional expression DC or the general expression GC or the general / special expression CGD is assigned to all the character strings in the in-document importance table 32. If there is no unprocessed record, the process proceeds to step 726 (step 725). In step 726, a flag is set to 0 (step 726).
[0084]
Next, the processing up to step 726 is executed for all the in-document importance degree tables 32, and it is determined whether the character strings have been classified. If there is still an unprocessed in-document importance table 32, the process proceeds to step 728. If there is no unprocessed in-document importance table 32, the process is terminated (step 727).
[0085]
If there is an unprocessed in-document importance level table 32, the process moves to the next in-document importance level table 32, and the processing from step 711 to step 727 is performed for all the in-document importance level tables 32 (step 76).
[0086]
Here, a method for determining whether a character string is a combination of two elements will be described.
First, it is assumed that the processing from Step 711 to Step 717 is executed, and either a DC or GC classification code is assigned to the character string and stored in the character string classification table 36.
[0087]
In order to divide the character string into two elements, the character string delimiter position is shifted one by one, each character string exists in the character string classification table 36, and the classification code of the character string is used as a judgment condition. A classification code is assigned when they match. FIG. 19 shows a method of dividing the character string “network construction” into two elements.
[0088]
Here, the special expression DC is assigned to the character string “network”, the general / special expression CGD is assigned to the character string “network”, and the technical expression DC is assigned to the character string “construct”. The judgment condition for the combination of elements is “a combination of the general expression GC and the technical expression DC” or “a combination of the general expression GC and the general / special expression CGD”.
[0089]
In FIG. 19, the number portion indicates the number of times the character string is divided, the character string A portion and the character string B portion indicate each element of the character string divided into two, and the collation result portion indicates the character string A and the character string B. The result of collating whether both of these elements exist in the character string classification table 36 is shown.
[0090]
First, if the delimiter position of “network construction” is shifted by one character, the character string A becomes “ne” and the character string B becomes “network construction” for the number 1. Since these two elements do not exist in the character string classification table 36, it is determined that they are not a combination of the two elements.
[0091]
Next, in the number 6, the character string A is “network” and the character string B is “construction”. The “network” of the character string A exists in the character string classification table 36, but the “construction” of the character string B does not exist in the character string classification table 36, so it is not a combination of the two elements.
[0092]
In the number 7, the character string A is “network” and the character string B is “construction”. Both the character string A and the character string B exist in the character string classification table 36, and the respective classification codes are the general / special expression CGD and the general expression GC. Therefore, it is determined that the character string “network construction” is “combination of general expression GC and general / special expression CGD”.
[0093]
Next, the process of character string multiple classification processing in the second embodiment will be specifically described with reference to actual cases and the flowchart of FIG.
It is assumed that the contents of the in-document importance table 32 of FIG. 12 are obtained by executing the processing from step 1 to step 5 in the flowchart showing the operation of the automatic character string extracting apparatus of FIG.
[0094]
It is assumed that N document files 31 are stored in the storage device 3. Document importance table in FIG. _i Character string from 31 _k = "In" and character string _k = String importance W corresponding to “de” _ik = 1.65 is read and compared with a predetermined threshold T = 10. Since the importance 1.65 of the character string is smaller than the threshold value 10, the general expression GC is given, and the character string classification table _i A character string and a general expression GC of the classification code are stored in 36. (FIG. 15, steps 711 to 715).
[0095]
This processing is performed in the document importance table. _i This is repeated for all the records in 32 (step 716 in FIG. 15). If you do the same process, _k = String importance W for “de” and “no” _ik Is 0 and is smaller than the threshold value 10, so that the general expression GC is given and the character string classification table _i The character string and the general expression GC of the classification code are stored in 36 (steps 714 and 715 in FIG. 15).
[0096]
In contrast, the string _k = "Network" string importance W _ik Is 39.33, which is larger than the threshold value 10, so that the special expression DC is given and the character string classification table _i 36 stores the character string and the specialized expression DC of the classification code (steps 713 and 715). FIG. 17 shows the importance level table in the document of FIG. _i 32 shows the contents after the character string plural classification processing is performed with the threshold T = 10.
[0097]
Next, set the flag to 0 and the character string classification table _i One record is read from 36. The classification code of the character string “in” of the first record is the general expression GC, and is not the last record, so the process moves to the next record (steps 718, 719, and 725 in FIG. 15). Next read string _k = For “de” and “no”, the classification code is the general expression GC.
[0098]
Next, the string _k = Read “network”. Where the string _k = "Network" does not correspond to any of the conditions shown in Steps 719, 720, and 722 of FIG. Further, since it is not the last record, the processing is shifted to the next record (steps 718, 719, 722, 725).
[0099]
Next, the string _k = “Network” is read. String _k = “Network” is a character string composed of two elements of “Network” of the professional expression DC and “General” of the general expression GC. Therefore, it satisfies the condition of Step 722, and the general / special expression CGD as a classification code. And the classification code DC already stored in the classification code storage section of the character string classification table i36 is replaced with the general / special expression CGD (steps 718, 722, 723, and 729). Since the flag is set to 1 and not the last record, the process moves to the next record (steps 724 and 725).
[0100]
Next, the string _k = Read "Network construction". String _k = "Network construction" is divided into two elements "network" and "construction". Here, “network” corresponds to the condition “combination of general / technical expression CGD and technical expression DC” in step 722 because the classification code is replaced with general / technical expression CGD in the previous processing, and the classification code. And the general / special expression CGD is assigned, and the classification code DC already stored in the classification code storage section of the character string classification table i36 is replaced with the general / special expression CGD.
[0101]
String classification table _i The above processing is repeated up to the last 36 records, and after processing to the last record, it is determined whether the flag is 0 or not. At this time, since the flag is 1, the process proceeds to step 717 and the flag is set to 0. The processing of steps 718 to 726 is repeated until the flag becomes 0 in the determination of step 726.
[0102]
When the flag is 0, the character string multi-classification process is completed for all the character strings stored in the character string classification table 36. In step 727, it is determined whether there is another character string classification table 36, and all character string classification tables are determined. _i The process is executed for 36. 18 shows the character string classification table of FIG. _i For 36, the contents after performing the character string plural classification process are shown.
[0103]
The above processing is performed on a plurality of document files 31 (1 to N). As a result, corresponding character string classification tables 36 are created for all the document files 31.
[0104]
<Effects of Second Embodiment>
According to the second embodiment of the present invention, in addition to the effects obtained in the first embodiment, the n-gram character string extracted from the text is converted into a specialized expression, a general expression, and a combination of the specialized expression and the general expression. It can be classified into three expressions.
[0105]
By classifying the expression into a combination of the technical expression and the general expression, it is possible to remove unnecessary phrases when creating the technical term dictionary.
[0106]
For example, when creating a technical term dictionary from the character string classification table 36 of FIG. A technical term dictionary can be created only with a character string to which an expression DC is assigned.
[0107]
Further, the character string to which the general / special expression CGD is added includes a word used in association with the technical term, and represents a professional wording. Therefore, it is possible to create a specialized expression dictionary (a dictionary storing specialized phrases) from which character strings to which general / expert expressions CGD are assigned are extracted.
[0108]
The character string division when determining whether the extracted character string is a combination of specialized expression and general expression is a method of re-dividing the character string using the result of dividing the character string once. . This makes it possible to determine whether the combination of expressions is more efficient than dividing a character string composed of a plurality of expressions at once.
[0109]
This effect occurs particularly when the character string is repeatedly divided into two. For example, when the character string “network construction” in the character string classification table 36 of FIG. 18 is divided, the character string “network construction” (/ represents a word division delimiter) is divided into three at once.・ It is more efficient than judging it as specialized expression.
[0110]
(C) Other embodiments
(C-1) In the first and second embodiments, character strings extracted from text are classified into various levels of expression, and are necessary when searching for keywords in a document. It can be applied to keyword extraction, automatic extraction of technical term dictionaries used in systems such as machine translation.
[0111]
(C-2) In the first and second embodiments, the character strings after classification are stored in the keyword table and the character string classification table. However, when outputting the extracted character strings, the format of each table is changed. Various modifications such as rearrangement based on the importance of the character string, the classification code, and the similarity of the character string are possible. Further, the output is not limited to only the final classification result stored in the keyword table or the character string classification table, and the result of each process in the keyword classification may be arbitrarily output.
[0112]
(C-3) Regardless of the processing of the present invention, when it is desired to fix the classification of a specific character string to specialized expression or general expression, the classification to be fixed of the character string is registered in the storage device, and classification processing is performed. It may be determined whether or not the character string extracted before is a character string that fixes the classification, and if so, a registered classification may be assigned.
[0113]
【The invention's effect】
As described above, according to the present invention, an automatic character string classification device that automatically classifies extracted character strings, considering the importance of character strings in a document and the importance of a plurality of document files as a whole. Thus, since the importance of the character string is determined, the extracted character string can be classified into specialized expressions and general expressions without being judged only by the importance in the document.
[0114]
Further, in addition to the specialized expression and the general expression, it can be classified into three by determining that the character string is composed of a combination of these two elements.
By obtaining information that character strings are composed of combinations, it is possible to remove unnecessary phrases when creating a technical term dictionary, and a specialized expression dictionary (a dictionary that stores specialized wording) ) Can be created.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a first embodiment of an automatic character string extraction apparatus of the present invention.
FIG. 2 is a diagram illustrating an in-document importance table.
FIG. 3 is a diagram illustrating an inter-document importance table.
FIG. 4 is a diagram showing a keyword table.
FIG. 5 is a block diagram illustrating functions of an importance calculation unit.
FIG. 6 is a flowchart showing the operation of the automatic character string extraction apparatus of the present invention.
FIG. 7 is a flowchart showing an operation of inter-document importance processing.
FIG. 8 is a flowchart showing an operation of character string importance processing;
FIG. 9 is a flowchart illustrating an operation of character string classification processing according to the first embodiment;
FIG. 10 is a diagram illustrating an intermediate result example stored in an in-document importance table.
FIG. 11 is a diagram showing an example stored in an inter-document importance table.
FIG. 12 is a diagram showing an example stored in a document importance level table;
FIG. 13 is a diagram showing an example stored in a keyword table.
FIG. 14 is a block diagram showing a second embodiment of the automatic character string extraction apparatus of the present invention.
FIG. 15 is a flowchart illustrating an operation of character string classification processing according to the second embodiment;
FIG. 16 is a diagram showing a character string classification table.
FIG. 17 is a diagram illustrating an example of an intermediate result of a character string classification table.
FIG. 18 is a diagram illustrating an example of a character string classification table.
FIG. 19 is a diagram illustrating a method of determining whether a character string is composed of two elements.
[Explanation of symbols]
1. Input device
11. Input section
12. Output section
2. ・ Processing equipment
21..Character string extraction unit
22 .. Importance calculator
23..Character string classification
24..Multiple character string classification part
3. Storage device
31. Document file
32 ... Importance table in document
33. Inter-document importance table
34..Keyword table
35. Buffer
36 .. Character string classification table.

Claims (8)

A document storage unit for storing a plurality of documents described in a natural language; a character string extraction unit for extracting a character string from an arbitrary document among the plurality of documents;
In-document importance calculation means for calculating the importance of the character string extracted by the character string extraction means in the extracted document as document importance,
An inter-document importance calculating means for calculating the importance of the whole of the plurality of documents of the character string extracted by the character string extracting means;
Character string importance calculating means for calculating the importance of the character string extracted based on the importance in the document and the importance between the documents as the character string importance,
A character string classification means for classifying the character string extracted based on the character string importance obtained by the character string importance calculation means,
The character string classification means includes:
For all the extracted character strings, a general expression that compares the character string importance assigned to each character string with a predetermined threshold and uses all the extracted character strings regardless of a specific field, or A first classification unit that classifies specialized expressions used in a specific field;
When the first classification unit divides the character string classified into the specialized expression and each component of the divided character string exists in the first classification result, the component of the divided character string A character string comprising: a second classification unit that reclassifies the classification of the character string before division into a general technical expression or a general expression that is a specialized expression used with the general expression from the specialized expression Automatic classification device. In the automatic character string classification device according to claim 1,
The in-document importance calculating unit determines the in-document importance based on the frequency with which the extracted character string appears in the extracted document. In the automatic character string classification device according to claim 1,
The inter-document importance calculation unit determines the inter-document importance based on the number of documents in which the extracted character string appears in the whole of the plurality of documents. In the automatic character string classification device according to claim 1,
The automatic character string classification device, wherein the character string importance calculation means calculates the character string importance weighted by the importance in the document and the importance between documents of the extracted character string. In the automatic character string classification device according to claim 1,
In the second classification unit, when the first classification result is replaced with a new classification, the second classification unit is replaced with the new classification when referring to the first classification result. An automatic character string classification device that refers to a subsequent classification result. In the automatic character string classification device according to claim 1,
When the component of the divided character string is a combination of the general expression and the technical expression, the second classification unit classifies the character string before the division from the technical expression to the general technical expression. fix, the divided combined components of the string in the general expertise represented in the general expression, or if the is a combination of specialized representation in the general specialist expressions classified character string before division from the specialist expression An automatic character string classification device that reclassifies the general expression. In the automatic character string classification device according to claim 1,
The automatic character string classification device, wherein the character string extraction means extracts character strings of all lengths starting from all characters included in a document. Document storage processing for storing multiple documents written in natural language;
A character string extraction process for extracting a character string from an arbitrary document among the plurality of documents;
In-document importance calculation means for calculating the importance of the character string extracted by the character string extraction means in the extracted document as document importance,
An inter-document importance calculation process for calculating the importance of the character strings extracted by the character string extraction means as the inter-document importance;
Character string importance calculation processing for calculating the importance of the character string extracted based on the importance in the document and the importance between the documents as the character string importance,
A character string classification process for classifying the extracted character string based on the character string importance obtained by the character string importance calculating means,
The character string classification process includes:
For all the extracted character strings, a general expression that compares the character string importance assigned to each character string with a predetermined threshold and uses all the extracted character strings regardless of a specific field or A first classification process for classifying specialized expressions used in a specific field;
When the character string classified into the specialized expression in the first classification process is divided, and each component of the divided character string exists in the first classification result, the component of the divided character string A character string further comprising: a second classification process for reclassifying the classification of the character string before division into a general technical expression or a general expression that is a technical expression that appears with the general expression from the technical expression by combination. Automatic classification method.

Images