JP4669642B2 - Document classification apparatus, document classification method, and computer-readable recording medium storing a program for causing a computer to execute the document classification method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an area of technology for classifying a document based on the contents (topics) of the document, in particular, a document classification device for classifying a document by correcting a vector space in a document classification technology based on a vector space model (Vector Space Model). The present invention also relates to a document classification method and a computer-readable recording medium on which a program for causing a computer to execute the document classification method is recorded.
[0002]
[Prior art]
In recent years, with the rapid progress and spread of IT (information technology), it has become possible to access a large amount of electronic documents in Japan and overseas via a network. The amount of information being made is also increasing. Under such circumstances, intelligent work such as classifying a large amount of collected document information into a meaningful category (category) has been performed. There are two purposes (1) and (2) described below for the purpose of classifying this document into meanings or classifying each document by topic.
[0003]
(1) To improve searchability. That is, a huge document group can be searched using the classification name (content name) as a clue, so that a set of documents to which a desired document belongs can be captured.
(2) To grasp the structure of the entire information group. That is, the contents (individual classification) of the entire document group are grasped.
[0004]
However, when a user manually classifies a large amount of document information, although the accuracy of classification is excellent, human and time costs increase, so it is practically impossible in a recent document environment that handles a large number of documents. Therefore, an automatic document classification device has been proposed.
[0005]
As the automatic document classification, for example, JP-A-7-36897, JP-A-10-260991, JP-A-10-17823, JP-A-10-260991, "Project for Efficient Document Clustering, Hinrich
Schutze and Craving Silverstone, 1997, Proceedings of SIGIR 97, pp 74-81, ACM ”. Here, the documents are regarded as document vectors characterized by words, and the similarity (distance) between the vectors is used as a measure, and these documents are grouped using a clustering method to automatically classify the documents.
[0006]
On the other hand, an object for the above (2) is disclosed in, for example, Japanese Patent Application Laid-Open No. 11-15835. Here, as described above, the documents are automatically classified based on the similarity measure.
[0007]
While clustering is an algorithm that generates an ad hoc classification system according to the similarity between document vectors, there is a method called categorization or classification. In this method, several categories (classifications) are provided in advance, and documents are automatically classified by determining which category each document belongs to. Among them, a classifier proposed by Salton et al. Is disclosed in “Introduction to Modern Information Retrieval, G. Salton and MJ McGill, 1983, New York NcGraw Hill”.
[0008]
This apparatus calculates a similarity (cosine) between a document vector and a category vector, and automatically classifies the document according to which category the document is most similar to. Furthermore, Japanese Patent No. 2940501 improves the accuracy of automatic classification by improving the word clustering used for classification.
[0009]
The automatic classification method described above is characterized in that documents are arranged as vectors in a space composed of words extracted from documents, similarity between documents is calculated, and clustering and categorization are performed. Therefore, the result of document classification depends on what kind of space is configured.
[0010]
By the way, in many cases, specific information is given to a document other than the text. For example, bibliographic items such as a document creator name (author) and a document creation date correspond to this. Also, in structured documents such as SGML (Standard Generalized Markup Language) and XML (extensible Markup Language), such specific information that appears in the text is previously tagged. Is granted and can be managed.
[0011]
Furthermore, “information extraction technology” has been disclosed as a technology for automatically extracting such specific information and is still available. For example, in IREX (Information Retrieval and Extraction Excise) held in Japan in 1999, the organization name (ex. US Army), person name (ex. Clinton), place name, proper name, etc. appearing in the text (in the text) An automatic extraction technique is disclosed.
[0012]
[Problems to be solved by the invention]
However, in the conventional technique as described above, when the extracted specific information (word) is used for the spatial configuration at the time of document classification, the specific information (partial information) is generated. There is a problem that the merit of using the extraction of specific information and document classification is halved. That is, when a document set including specific information is obtained, it is only necessary to search for documents using the specific information as a key. Therefore, in the document classification technique, it is desirable to classify from a viewpoint different from the specific information.
[0013]
The present invention has been made in view of the above, and an object of the present invention is to realize effective document classification by excluding extracted specific information from a spatial configuration at the time of document classification.
[0014]
[Means for Solving the Problems]
  In order to achieve the above object, in the document classification apparatus according to claim 1, the document input means for inputting the document information to be classified and the document input meansSaidAnalyzing document information,It is the result of morphological analysisLanguage analysis means for obtaining language analysis information and input by the document input meansSaidAnalyzing document information,Attached information or proper nameInformation extracting means for extracting specific information and said information extracting meansOf the specific information byAccording to output, By removing the specific information from the language analysis information,Language analysis information correction means for correcting the language analysis information, and language analysis information correction meansThe plurality of words whose language analysis information is corrected byGenerate a document feature vector for the document information according to the outputDocument featuresVector generation means; andDocument featuresGenerated by vector generation meansSaidAccording to document feature vectorpluralClassifying the document information;Multiple document groups composed of the document informationA document classification means for generating,The information extraction unit obtains specific information according to tag information attached to the structured document.Is.
[0015]
  According to this invention, the document information input by the document input means is analyzed by the language analysis means.It is the result of morphological analysisLanguage analysis information is acquired, and further information extraction means from document informationAttached information or proper nameThe specific information is extracted, and the language analysis information correcting means extracts the specific information from the language analysis information.By removingCorrect language analysis information,Document featuresA vector generation unit generates a document feature vector, and according to the document feature vector, a document classification unit classifies the document using a technique such as clustering or categorization,Multiple document groups consisting of document informationBy generating, the result of classification by topic and the result of classification by specific information (grouping documents having arbitrary specific information (tag, attribute name)) should not overlap in content. It becomes possible.Further, when the document information to be classified is a structured document such as SGML or XML, the information extraction unit can extract specific information from tag information attached to the structured document.
[0016]
  In the document classification device according to claim 2, the document input means for inputting the document information to be classified and the document input meansSaidAnalyzing document information,It is the result of morphological analysisLanguage analysis means for obtaining language analysis information and input by the document input meansSaidAnalyzing document information,Attached information or proper nameInformation extraction means for extracting specific information and obtained by the language analysis meansSaidGenerate a document feature vector for the document information according to language analysis informationDocument featuresVector generating means and the information extracting meansSaid specific information byAccording to the output of, By removing the specific information from the document feature vector,Modify the document feature vectorDocument featuresVector correction means, andDocument featuresVector correction meansVector correctionAccording to outputpluralClassifying the document information;Multiple document groups consisting of document informationA document classification means for generating,The information extraction unit obtains specific information according to tag information attached to the structured document.Is.
[0017]
  According to this invention, the document information input by the document input means is analyzed by the language analysis means.It is the result of morphological analysisLanguage analysis information is acquired, and further information extraction means from document informationAttached information or proper nameExtract specific information,Document featuresA vector generation unit generates a document feature vector for the document information according to the language analysis information, andBy removing the specific information from the document feature vectorCorrecting the document feature vector, and based on the corrected document feature vector, the document classification means classifies the document;Multiple document groups consisting of document informationBy generating, the result of classification by topic and the result of classification by specific information (grouping documents having arbitrary specific information (tag, attribute name)) should not overlap in content. It becomes possible.Further, when the document information to be classified is a structured document such as SGML or XML, the information extraction unit can extract specific information from tag information attached to the structured document.
[0018]
In the document classification apparatus according to claim 3, the information extraction means obtains specific information by extracting proper noun expressions.
[0019]
According to the present invention, in claim 1 or 2, the information extraction unit obtains the specific information of the proper noun expression from the input document information, thereby generating the document feature vector excluding the influence of the proper noun expression. It becomes possible.
[0022]
  Claims4The document classification apparatus according to the present invention further comprises display means for displaying the specific information extracted by the information extraction means, and extracted information selection means for accepting selection of one or more specific information. is there.
[0023]
According to the present invention, the specific information extracted by the information extracting means is displayed, and the user can select and specify the specific information to be excluded at the time of classification calculation from the displayed specific information. Classification processing becomes possible.
[0024]
  Claims5In the document classification method related toA document classification method realized by executing a program prepared in advance on a computer, and by executing the program, the computerDocument input process for inputting document information to be classified, and input by the document input processSaidAnalyzing document information,It is the result of morphological analysisLanguage analysis process for obtaining language analysis information and input by the document input processSaidAnalyzing document information,Attached information or proper nameInformation extraction process for extracting specific information and the information extraction processOf the specific information byLanguage analysis information correction step for correcting the language analysis information according to the output, and the language analysis information correction stepThe plurality of words whose language analysis information is corrected byGenerate a document feature vector for the document information according to the outputDocument featuresA vector generation step, andDocument featuresAccording to the document feature vector generated by the vector generation processpluralClassifying the document information;Multiple document groups composed of the document informationA document classification process to be generated;And the information extracting step obtains specific information according to tag information attached to the structured document.Is.
[0025]
  According to the present invention, the document information input in the document input process is analyzed in the language analysis process.It is the result of morphological analysisLanguage analysis information is acquired, and the information extraction process is performed from document information.Attached information or proper nameThe specific information is extracted, and the language analysis information correction step extracts the specific information from the language analysis information.By removingCorrect language analysis information,Document featuresThe vector generation process generates a document feature vector, and according to the document feature vector, the document classification process classifies the document using a technique such as clustering or categorization,Multiple document groups consisting of document informationBy generating, the result of classification by topic and the result of classification by specific information (grouping documents having arbitrary specific information (tag, attribute name)) should not overlap in content. It becomes possible.When the document information to be classified is a structured document such as SGML or XML, for example, the information extraction step can extract specific information from tag information attached to the structured document.
[0026]
  Claims6In the document classification method related toA document classification method realized by executing a program prepared in advance on a computer,  By executing the program, the computerDocument input process for inputting document information to be classified, and input by the document input processSaidAnalyzing document information,It is the result of morphological analysisLanguage analysis process for obtaining language analysis information and input by the document input processSaidAnalyzing document information,Attached information or proper nameObtained by the information extraction step of extracting specific information and the language analysis stepSaidGenerate a document feature vector for the document information according to language analysis informationDocument featuresVector generation step and the information extraction stepSaid specific information byAccording to the output ofBy removing the specific information from the document feature vector,Modify the document feature vectorDocument featuresVector correction step, andDocument featuresVector correction processVector correctionAccording to outputpluralClassifying the document information;Multiple document groups composed of the document informationA document classification process to be generated;And the information extracting step obtains specific information according to tag information attached to the structured document.Is.
[0027]
  According to the present invention, the document information input in the document input process is analyzed in the language analysis process.It is the result of morphological analysisLanguage analysis information is acquired, and the information extraction process is performed from document information.Attached information or proper nameExtract specific information,Document featuresA vector generation step generates a document feature vector for the document information according to the language analysis information, andBy removing the specific information from the document feature vectorModify the document feature vector, and based on the modified document feature vector, the document classification step classifies the document;Multiple document groups consisting of document informationBy generating, the result of classification by topic and the result of classification by specific information (grouping documents having arbitrary specific information (tag, attribute name)) should not overlap in content. It becomes possible.When the document information to be classified is a structured document such as SGML or XML, for example, the information extraction step can extract specific information from tag information attached to the structured document.
[0028]
  Claims7In the document classification method, the information extracting step obtains specific information by extracting proper noun expressions.
[0029]
  According to the invention, the claims5 or 6In this case, the information extraction step acquires specific information of proper noun expressions from the input document information, so that it is possible to generate a document feature vector excluding the influence of the proper noun expressions.
[0032]
  Claims8The document classification method further includes a display step for displaying the specific information extracted by the information extraction means and an extraction information selection step for accepting selection of one or more specific information.
[0033]
According to this invention, the specific information extracted by the information extraction process is displayed, and the user can select and specify the specific information to be excluded at the time of classification calculation from the displayed specific information. Classification processing becomes possible.
[0034]
  Claims9A computer-readable recording medium according to claim 15-8A program for causing a computer to execute the document classification method described in any one of the above is recorded.
[0035]
  According to the invention, the claims5-8When the document classification method described in any one of the above is programmed and recorded on a computer-readable recording medium, the document classification method can be executed on a computer.
[0036]
DETAILED DESCRIPTION OF THE INVENTION
DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of a document classification apparatus, a document classification method, and a computer-readable recording medium recording a program for causing a computer to execute the document classification method according to the present invention will be described in detail with reference to the accompanying drawings. . The present invention is not limited to this embodiment.
[0037]
(Embodiment 1)
FIG. 1 is a block diagram showing a system configuration of the document classification apparatus according to the first embodiment of the present invention. The document classification apparatus according to the first embodiment includes a document input unit 101 that inputs document information to be classified on the bus 100, a language that analyzes the document information input by the document input unit 101, and obtains language analysis information. An analysis unit 102; an information extraction unit 103 that analyzes document information input by the document input unit 101 to obtain specific information; a language analysis information correction unit 104 that corrects language analysis information according to an output of the information extraction unit 103; A document feature vector generation unit 105 that generates a document feature vector for the document information according to the output of the language analysis information correction unit 104, and classifies the document information according to the document feature vector generated by the document feature vector generation unit 105, Are connected to a document classification unit 106 that generates a subset of
[0038]
FIG. 2 is a flowchart showing a basic operation procedure of document classification in FIG. Here, first, the document input unit 101 executes document input processing for inputting the document information to be classified (step S11), and the input document information is analyzed by the language analysis unit 102 to obtain language analysis information. Language analysis processing is performed (step S12). Subsequently, the information extraction unit 103 analyzes the input document information and executes an information extraction process for obtaining specific information (step S13). The language analysis information correction unit 104 follows the output of the information extraction process. A language analysis information correction process for correcting the language analysis information is performed (step S14). Further, the document feature vector generation unit 105 executes a vector generation process for generating a document feature vector for the document information in accordance with the output of the language analysis information correction process (step S15), and the document classification unit 106 performs the vector generation process. The document information is classified according to the document feature vector generated by the above, and document classification processing for generating a document subset is executed (step S16).
[0039]
Here, as a suitable example in which extraction of specific information and document classification based on contents (topics) included in a document set are important, an analysis situation of free description answers obtained through questionnaire surveys, etc. is assumed. An example will be described.
[0040]
In recent years, for example, it is possible to collect thousands to hundreds of thousands of free description answers on a computer over a short period of time via the Internet, etc., and collect a large amount of text information using such a function. Can do.
[0041]
Here, as an example of a large amount of text information obtained by a questionnaire survey, it is assumed that a request for a printer owned by a respondent is described. The questionnaire is composed of “requests for printers”, question items of the printer name (product name), printer manufacturer, and satisfaction (Q1) for the printer. Suppose that N responses were received in total.
[0042]
In the example of the free description answer in the questionnaire, the document set has a format as shown in FIG. 3, for example. Here, as one of the analysis activities, the analyst (the operator of the invention) has any kind of opinion (topic) in the answer set (document set), and these are between the manufacturer and the target product. Assume that you are trying to figure out what kind of relationship there is.
[0043]
First, the answer set to be classified is incorporated into the system via the document input unit 101. This input information is usually recorded and stored for later processing on the captured answer set. Subsequently, for the collected answer set, the language analysis unit 102 extracts words and compound words (or specific continuous character strings) included in each document (each answer). For this process, a known language analysis algorithm such as morphological analysis is used. An example in which words of nouns, adjectives, and adjective verbs are extracted by the language analysis unit 102 is shown below.
[0044]
ID0001 →
XL ・ 100 (unregistered word), consumables (general noun), high (adjective), printing (sa variable noun)
ID0002 →
Company A (unregistered word), salesman (general noun), technology (general noun), knowledge (general noun), abundance (adjective verb), trust (general noun)
ID0003 →
Company B (unregistered word), PRX 4000 (unregistered word), printing (sa variable noun), speed (general noun), satisfaction (sa variable noun), company newsletter (general noun), business (general noun), manual ( Common noun), use (sa variable noun)
ID000N →
Company A (unregistered word), trust (general noun), high (adjective), use (sa variable noun)
[0045]
Next, the information extraction unit 103 extracts specific information from the answer set. The first method of information extraction is to use attribute values (value of target product name field, value of manufacturer field) other than bibliographic items attached to each document or text attribute to be classified (other than Q1 in the example of FIG. 3 described above). This is a method of outputting to the language analysis information correction unit 104. The attribute value of any attribute may be stored in the file or the like in advance in the language analysis information correction unit 104. In this example, the attribute values of “target product name” and “manufacturer” are extracted, and the attribute value of “satisfaction” is not extracted.
[0046]
The second method of information extraction is a method that can be used when attached information such as “target product name” and “manufacturer” is not acquired as an attribute value in advance. This is a known technique called information extraction technique, and automatically extracts the following unique names from the text described in Q1.
[0047]
Organization name (company name)
Name
Place name
Product name
date
time
Amount of money
Percentage
etc
[0048]
With this information extraction technology, “XL • 100” is the product name from the ID0001 response, “Company A” is the company name from the ID0002 response, “B Company” is the company name from ID0003, and “ “PRX • 4000” is extracted from ID000N as “Company A” as the company name.
[0049]
Next, a method for correcting a vector space expressing a document or word according to the specific information extracted in this way will be described. There is a known technique called a “stop word list” in which an arbitrary word (token) that is not desired to contribute to document classification is described in a file or the like. In the present invention, a technique is used in which specific information extracted by the information extraction unit 103 functions in a “stop word list” by automatic / user selection.
[0050]
The language analysis information correction unit 104 removes the specific information extracted by the information extraction unit 103 from the language analysis information output from the language analysis unit 102. For example, for the answer of ID0001, the language analysis unit 102 outputs the following language analysis information.
[0051]
ID0001 →
XL · 100 (unregistered), consumables (general nouns), high (adjectives), printing (sa variable noun)
[0052]
The language analysis information correction unit 104 removes “XL · 100” extracted by the information extraction unit 103 from the language analysis information, and obtains ID0001 → consumable item (general noun), high (adjective), and print (sa variable noun). Output.
[0053]
Similarly, ID0002 to ID000N are
ID0002 →
Salesman (general noun), technology (general noun), knowledge (general noun), abundance (adjective verb), trust (general noun)
ID0003 →
Print (sa noun), speed (general noun), satisfaction (sa noun), company newsletter (general noun), business (general noun), manual (general noun), use (sa noun)
ID000N →
Trust (general noun), high (adjective), use (sa variable noun)
It becomes.
[0054]
As a result, the subsequent generation of document feature vectors and document classification can be performed without using the token extracted by the information extraction unit 103.
[0055]
Next, according to the output from the language analysis information correction unit 104, the document feature vector generation unit 105 uses tokens (feature description units) such as words as columns, each document as a row, and elements as, for example, word appearance frequency. The “token” × “document (answer)” matrix is generated.
[0056]
In addition, when the language analysis unit 102 having a general morphological analysis function and a syntax analysis function is used, the token is described in the above description, in addition to the word extraction, the word part of speech information, the compound word (phrase), and the syntax information are simultaneously displayed. For example, the following contents (1) and (2) are possible.
[0057]
(1) “International Union” is tokenized as one compound word instead of two words “international (general noun)” and “association (general noun)”.
(2) The word “explanation (sa variable noun)” is treated as “sa variable noun“ explanation ”appearing in the predicate part” according to its appearance position, and is distinguished from the case where it appears in the subject part.
[0058]
The document feature vector generation unit 105 obtains a document vector according to this “token” × “document” matrix, and there are the following three methods, any of which may be used in the present invention.
[0059]
(1) The column component of the matrix is used as it is as a document feature vector.
(2) Weighting values in consideration of the length of each document (measured by the number of characters, the number of pages, etc.) and the appearance frequency of each token in the document set to be classified, and then using it as a document feature vector .
(3) An inner product matrix between documents is calculated from the above matrix, singular value decomposition is applied to the matrix, a potential semantic space is constructed, and the position of each document in the space is obtained and used as a vector. This technology is realized by referring to the contents of “Project for Effective Document Clustering, Hinrich Schutze and Craving Silverstone 1997, Processeds of SIGIR, pp 74-81, ACM”.
[0060]
The document classification unit 106 classifies the document using the similarity of the document feature vector that is the output of the document feature vector generation unit 105. As measures of similarity, inner products, cosines, Euclidean distances, Mahalanobis distances, and the like can be considered, and any measure can be used.
[0061]
The document classification method includes a bottom-up classification method for grouping similar documents according to the similarity between document feature vectors called clustering, and several categories (classifications) called categorization in advance. There is a method of automatically classifying documents by determining which category each document belongs to. Hereinafter, the clustering method and the categorization method will be described.
[0062]
First, the clustering method will be described. For clustering, various calculation methods including hierarchical and non-hierarchical algorithms are known, and any of them can be used. The K-means algorithm method is taken up as an example of a typical algorithm. This algorithm classifies a set of n document feature vectors generated by the document feature vector generation unit 105 into k vector sets according to their similarity. Note that k must be given in advance.
[0063]
The calculation procedure of this algorithm is
(1) The initial value of the centroid of k clusters is selected from n vectors.
(2) Assign each of the n clusters to the most similar cluster centroid.
(3) For each of k clusters, a new average of vectors included in each cluster is obtained, and this is used as a new cluster centroid.
(4) The above (2) and (3) are repeatedly executed until the end condition is satisfied, such as when the position of the cluster centroid no longer changes.
It is.
[0064]
By using this clustering method, it is possible to obtain a plurality of document sets (clusters) composed of similar document groups. Here, since the specific information extracted by the information extraction unit 103 is not used for calculating the similarity between the document feature vectors, a cluster can be generated without being affected by the specific information. That is, a dominant cluster including specific information (for example, a specific manufacturer) is not formed.
[0065]
Next, the categorization method will be described. Various methods have already been proposed. Among them, G, Salton and M.M. J. et al. The method by McGill (Introduction to Modern Information Retrieval, 1983, New York NcGraw Hill) is well known.
[0066]
In this categorization method, first, categories to be classified are defined in advance. If each category is defined by, for example, specifying a sample document, a vector of each category can be generated based on the sample document. When a plurality of documents are samples, the category document may be added to the sample document vector. In order to properly classify the document to be classified into each category, it is possible to calculate the similarity between each document feature vector to be classified and each category vector, and classify the document into the most similar category. it can. The document-word (token) space, which is a premise for calculating the similarity between vectors that is also used when performing such classification, does not include the specific information extracted by the information extraction unit 103. Documents can be placed in categories regardless of information.
[0067]
Therefore, according to the first embodiment described above, the specific information (for example, product name) extracted by the information extracting unit 103, the classification result (for example, the content of the user's request) based on the content achieved by the document classification unit 106, The relationship between can be examined. Further, in the above-described example, for example, an analysis work such as grasping a difference in user request for each product name by cross tabulation or the like can be performed.
[0068]
(Embodiment 2)
FIG. 4 is a block diagram showing a system configuration of the document classification device according to the second exemplary embodiment of the present invention. The document classification apparatus according to the second embodiment includes a document input unit 101 for inputting document information to be classified on the bus 100, a language for analyzing the document information input by the document input unit 101, and obtaining language analysis information. An analysis unit 102, an information extraction unit 103 that analyzes the document information input by the document input unit 101 and extracts specific information, and a document feature vector for the document information according to the language analysis information obtained by the language analysis unit 102 Generate a document feature vector generation unit 105, a vector correction unit 107 that corrects a document feature vector according to the output of the information extraction unit 103, and classify the document information according to the output of the vector correction unit 107 to generate a document subset And a document classification unit 106 to be connected.
[0069]
FIG. 5 is a flowchart showing a basic operation procedure of document classification in FIG. Here, first, a document input process for inputting document information to be classified is executed by the document input unit 101 (step S21), and the input document information is analyzed by the language analysis unit 102 to obtain language analysis information. Language analysis processing is performed (step S22). Subsequently, the information extraction unit 103 analyzes the input document information and executes information extraction processing for obtaining specific information (step S23). The document feature vector generation unit 105 performs language analysis information by the language analysis unit 102. Accordingly, a document feature vector for the document information is generated (step S24). Further, the document feature vector is corrected according to the output of the information extraction unit 103 (step S25), and the document classification unit 106 classifies the document information according to the document feature vector generated by the vector correction processing, and sets a document subset. The document classification process for generating is executed (step S26).
[0070]
That is, the second embodiment is different from the first embodiment described above in that the correction based on the specific information extracted by the information extraction unit 103 is performed on the generated document feature vector. The document feature vector generation unit 105 generates a vector of each document for the token extracted by the language analysis unit 102 as in the first embodiment. Since the vector is expressed as shown in FIG. 6, for example, the column vector may be deleted in order to exclude the specific information extracted by the information extraction unit 103.
[0071]
For example, in FIG. 6, if the specific information extracted by the information extraction unit 103 is token 3, the vector correction unit 107 can eliminate the influence by deleting the sequence of token 3. In addition, when singular value decomposition is performed and the feature dimensions are obtained not as tokens but as their combined dimensions (m1, m2, m3,... MI,. It is realized by finding mI having the highest correlation with information and eliminating it.
[0072]
By the way, the information extraction unit 103 in the first embodiment or the second embodiment described above may obtain specific information by extracting proper nouns.
[0073]
Furthermore, the information extraction unit 103 may extract specific information based on tag information attached to the structured document. Here, a case where the document input from the document input unit 101 is a structured document such as SGML or XML will be described as an example.
[0074]
For structured documents, for example, “Press release High-speed color printer [XL-1000] is released. Company A Co., Ltd. introduces a new high-speed color printer XL-1000 to the office market from January 7, 2000. Printing speed is The full color is 60 sheets per minute.The sales price is scheduled to be 800,000 yen, and it is planned to be sold from distributors nationwide "is described in the following format.
[0075]
<Document>
<H1> Press Release High-speed color printer "XL-1000" released <h1> <p> <sederaname> Company A </ sedername> is a new model from <releasedate> January 7, 2000 </ releasedate> A high-speed <producttype> color printer </ producttype> <productname> XL-1000 </ productname> is introduced into the office market. The printing speed is 60 sheets per minute in full color. The selling price is planned to be <price> 800,000 yen </ price>, and is scheduled to be sold from distributors nationwide. </ P>
</ Document>
[0076]
In structured documents, important words appearing in the document are tagged in a form sandwiched between <tagname> </ tagname>, so that it is possible to easily extract important words. In the above-described example, tags such as a sender name, a date of release (releasedate), a product name (productname), and a price (price) are embedded.
[0077]
The information extraction unit 103 of the present invention analyzes the tagging structure of the structured document, extracts a tag name such as sendername and a value such as company A in association with each other, and (tag name, value ) To extract. In this case, the above set can be obtained by extracting a portion from a location starting with <xxx> to a location ending with </ xxx>.
[0078]
That is, in the above example,
sendename, Company A
releaseedate, January 7, 2000
producttype, color printer
productname, XL-1000
price, 800,000 yen
Will be extracted.
[0079]
(Embodiment 3)
FIG. 7 is a block diagram showing a system configuration of the document classification apparatus according to the third embodiment of the present invention. The document classification apparatus according to the third embodiment includes a document input unit 101 that inputs document information to be classified on the bus 100, a language that analyzes the document information input by the document input unit 101, and obtains language analysis information. An analysis unit 102, an information extraction unit 103 that analyzes the document information input by the document input unit 101 and extracts specific information, and a document feature vector for the document information according to the language analysis information obtained by the language analysis unit 102 A document feature vector generation unit 105 to be generated, a vector correction unit 107 that corrects a document feature vector according to the output of the information extraction unit 103, an extraction information selection unit 108 that accepts selection of one or more specific information from a user, and information The specific information display unit 109 that displays the specific information extracted by the extraction unit 103 and the output of the vector correction unit 107 Classify document information Therefore, the document classification unit 106 to generate a subset of the documents, are connected.
[0080]
FIG. 8 is a flowchart showing a basic operation procedure of document classification in FIG. Here, first, the document input unit 101 executes a document input process for inputting the document information to be classified (step S31), and the input document information is analyzed by the language analysis unit 102 to obtain language analysis information. Language analysis processing is performed (step S32). Subsequently, the information extraction unit 103 analyzes the input document information and executes information extraction processing for obtaining specific information (step S33), and displays the specific information on the specific information display unit 109 such as a CRT ( Step S34).
[0081]
Further, the extracted information selection unit 108 accepts selection of one or more specific information from the user (step S35). Subsequently, the processing of the first or second embodiment described above is executed.
[0082]
In the case of the first embodiment, the language analysis information correction unit 104 performs language analysis information correction processing for correcting the language analysis information according to the output of the information extraction processing (step S36). Further, the document feature vector generation unit 105 executes a vector generation process for generating a document feature vector for the document information in accordance with the output of the language analysis information correction process (step S37), and the document classification unit 106 performs the vector generation process. Document information is classified according to the generated document feature vector, and document classification processing for generating a document subset is executed (step S39).
[0083]
On the other hand, in the case of the second embodiment, the document feature vector generated by the document feature vector generation unit 105 is corrected according to the output of the information extraction unit 103 (step S38), and the document classification unit 106 Document information is classified according to the document feature vector generated by the process, and a document classification process for generating a document subset is executed (step S39).
[0084]
That is, the third embodiment presents the specific information extracted by the information extraction unit 103 to the user, and allows the user to arbitrarily specify the specific information that is not desired to contribute to the classification.
[0085]
First, the specific information extracted by the information extraction unit 103 is displayed on the CRT, for example, by the specific information display unit 109. An example of this screen display is shown in FIG. The screen shown in FIG. 9 includes “company name, product name, announcement date” as a tag name or attribute name, “company A, company B, XL-100, XL-100” as values, and individual values. The appearance frequency in the entire document set is displayed.
[0086]
Further, the extraction information selection unit 108 takes a method of specifying a tag name that is not used for classification calculation by selecting a check box. That is, in the above example, the token having the company name and product name tag or the attribute name is not used for classification calculation, and the extracted information selection unit 108 uses the language analysis in the case of the first embodiment. In the case of the second embodiment, the information correction unit 104 outputs the information to the vector correction unit 107.
[0087]
By the way, the document classification method described so far can be programmed, recorded on a computer-readable recording medium, and executed on the computer. Also, a part of the document classification method can be realized on a network and can be realized through a communication line.
[0088]
That is, the document classification method described in this embodiment is realized by executing a program prepared in advance on a computer (CPU 20) such as a personal computer or a workstation as shown in FIG. This program is recorded on a computer-readable recording medium such as the memory 21, the hard disk 24, the floppy disk (FD) 27, the CD-ROM 26, the MO, and the DVD by operating the keyboard 25, and is recorded by the computer (CPU 20). It is executed by reading from the medium. In addition, the document classification processing data can be transmitted and received from the communication device 22 to an external device as necessary.
[0089]
In addition, as shown in FIG. 11, this program can be distributed to devices 31 to 33 such as a personal computer via the recording medium via a network such as the Internet 30. When the Internet function is provided, for example, TCP / IP (Transmission Control Protocol / Internet Protocol) is used as a communication protocol. This network is divided into a WAN (Wide Area Network) connected to the outside via a public line or a dedicated line, and a LAN (Local Area Network) that builds a network on the same site. There are methods that are classified, but any of them may be used.
[0090]
【The invention's effect】
  As described above, according to the document classification apparatus (claim 1) of the present invention, the document information input by the document input means is analyzed by the language analysis means.It is the result of morphological analysisLanguage analysis information is acquired, and further information extraction means from document informationAttached information or proper nameThe specific information is extracted, and the language analysis information correcting means extracts the specific information from the language analysis information.By removingCorrect language analysis information,Document featuresA vector generation unit generates a document feature vector, and according to the document feature vector, a document classification unit classifies the document using a technique such as clustering or categorization,Multiple document groups consisting of document informationBy generating, the result of classification by topic and the result of classification by specific information (grouping documents having arbitrary specific information (tag, attribute name)) should not overlap in content. Therefore, an effective document classification device that is not affected by specific information is realized.
[0091]
  According to the document classification apparatus of the present invention (Claim 2), the document information input by the document input means is analyzed by the language analysis means.It is the result of morphological analysisLanguage analysis information is acquired, and further information extraction means from document informationAttached information or proper nameExtract specific information,Document featuresA vector generation unit generates a document feature vector for the document information according to the language analysis information, andBy removing the specific information from the document feature vectorCorrecting the document feature vector, and based on the corrected document feature vector, the document classification means classifies the document;Multiple document groups consisting of document informationBy generating, the result of classification by topic and the result of classification by specific information (grouping documents having arbitrary specific information (tag, attribute name)) should not overlap in content. Therefore, an effective document classification device that is not affected by specific information is realized.Further, when the document information to be classified is a structured document such as SGML or XML, the information extraction unit can extract specific information from tag information attached to the structured document.
[0092]
According to the document classification apparatus (claim 3) of the present invention, in claim 1 or 2, the information extraction unit acquires specific information of the proper noun expression from the input document information. It is possible to generate a document feature vector that eliminates the influence of.
[0094]
  A document classification device according to the present invention (claims)4), It is possible to display the specific information extracted by the information extraction means, and from the displayed specific information, the user can select and specify the specific information to be excluded during classification calculation, It is possible to deal with flexible classification processing according to the type and content of the classification target.
[0095]
  A document classification method according to the present invention (claims)5), The document information input in the document input process is analyzed in the language analysis process.It is the result of morphological analysisLanguage analysis information is acquired, and the information extraction process is performed from document information.Attached information or proper nameThe specific information is extracted, and the language analysis information correction step extracts the specific information from the language analysis information.By removingCorrect language analysis information,Document featuresThe vector generation process generates a document feature vector, and according to the document feature vector, the document classification process classifies the document using a technique such as clustering or categorization,Multiple document groups consisting of document informationBy generating, the result of classification by topic and the result of classification by specific information (grouping documents having arbitrary specific information (tag, attribute name)) should not overlap in content. Therefore, an effective document classification method that is not affected by specific information is realized.When the document information to be classified is a structured document such as SGML or XML, for example, the information extraction step can extract specific information from tag information attached to the structured document.
[0096]
  A document classification method according to the present invention (claims)6), The document information input in the document input process is analyzed in the language analysis process.It is the result of morphological analysisLanguage analysis information is acquired, and the information extraction process is performed from document information.Attached information or proper nameExtract specific information,Document featuresA vector generation step generates a document feature vector for the document information according to the language analysis information, andBy removing the specific information from the document feature vectorModify the document feature vector, and based on the modified document feature vector, the document classification step classifies the document;Multiple document groups consisting of document informationBy generating, the result of classification by topic and the result of classification by specific information (grouping documents having arbitrary specific information (tag, attribute name)) should not overlap in content. Therefore, an effective document classification method that is not affected by specific information is realized.In addition, when the document information to be classified is a structured document such as SGML or XML, the information extraction process can extract specific information from the tag information attached to the structured document.
[0097]
  A document classification method according to the present invention (claims)7)5 or 6, The information extraction process obtains specific information of proper noun expressions from the input document information, and generates document feature vectors that eliminate the influence of this proper noun expression, thus eliminating the influence of this proper noun expression Document feature vectors can be generated.
[0099]
  A document classification method according to the present invention (claims)8), It is possible to display the specific information extracted by the information extraction process, and the user can select and specify the specific information to be excluded during the classification calculation from the displayed specific information. It is possible to deal with flexible classification processing according to the type and content of the classification target.
[0100]
  A computer-readable recording medium according to the present invention (claims)9)5-8Since the document classification method described in any one of the above is programmed and recorded on a computer-readable recording medium, the document classification method can be executed on a computer.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a system configuration of a document classification apparatus according to a first embodiment of the present invention.
FIG. 2 is a flowchart showing a basic operation procedure for document classification in FIG. 1;
FIG. 3 is a chart showing an example of document information according to the first embodiment of the present invention.
FIG. 4 is a block diagram showing a system configuration of a document classification device according to a second exemplary embodiment of the present invention.
FIG. 5 is a flowchart showing a basic operation procedure of document classification in FIG. 4;
FIG. 6 is a chart showing an example of each document vector according to the second embodiment of the present invention.
FIG. 7 is a block diagram showing a system configuration of a document classification apparatus according to a third embodiment of the present invention.
FIG. 8 is a flowchart showing a basic operation procedure of document classification in FIG. 7;
FIG. 9 is an explanatory diagram showing an example of a display screen according to the third embodiment of the present invention.
FIG. 10 is a block diagram showing an example of a computer system that implements the document classification method according to the embodiment of the present invention by software.
FIG. 11 is a block diagram showing an example of a system for realizing the document classification method according to the embodiment of the present invention on a network.
[Explanation of symbols]
101 Document input part
102 Language Analysis Department
103 Information extraction unit
104 Language analysis information correction part
105 Document feature vector generator
106 Document classification section
107 Vector correction section
108 Extraction information selection unit
109 Specific information display section

Claims (9)

A document input means for inputting document information to be classified, analyzed the document information input by the document input means, a language analysis means for obtaining a language analysis information is a result of the morphological analysis, input by the document input means have been analyzing the document information, and the information extraction means for extracting specific information is accompanying information or proper names, in accordance with the output of said identification information by said information extraction means to remove the specific information from the language analysis information by the language analysis information modifying means for modifying the language analysis information, document characteristic to generate a document feature vector for the document information in accordance with the output of a plurality of words that fixes the language analysis information by the language analysis information correction unit multiple and vector generation means, according to said document feature vector generated by the document feature vector generation means Wherein classifying the document information, and a document classifying means for generating multiple documents group including the document information,
The document extracting apparatus, wherein the information extracting unit obtains specific information according to tag information attached to a structured document . A document input means for inputting document information to be classified, analyzed the document information input by the document input means, a language analysis means for obtaining a language analysis information is a result of the morphological analysis, input by the document input means It has been analyzing the document information, and generates an information extracting means for extracting specific information is accompanying information or proper name, the document feature vector for the document information according to the language analysis information obtained by the language analysis unit a document feature vector generation means, in accordance with the output of said identification information by said information extraction means, by removing the specific information from the document feature vector, and document characteristic vector correction means for correcting said document feature vector, the document classifying a plurality of the document information in accordance with the output of the vector correction feature vector modification unit, the sentence And a document classifying means for generating multiple documents group consisting of information,
The document extracting apparatus, wherein the information extracting unit obtains specific information according to tag information attached to a structured document .   The document classification apparatus according to claim 1, wherein the information extraction unit obtains specific information by extracting a proper noun expression. Furthermore, the display means which displays the specific information extracted by the said information extraction means, and the extraction information selection means which receives selection of one or more specific information are provided, The any one of Claims 1-3 characterized by the above-mentioned. The document classification device according to claim 1. A document classification method realized by executing a program prepared in advance on a computer,
By executing the program, the computer, the document input step of inputting document information to be classified, analyzed the document information input by the document input step, the language analysis information is a result of the morphological analysis a language analysis step of obtaining, analyzing the document information input by the document input step, an information extraction step of extracting specific information is accompanying information or proper names, in accordance with the output of said identification information by said information extraction step A language analysis information correction step for correcting the language analysis information, and a document feature vector generation step for generating a document feature vector for the document information in accordance with an output of a plurality of words in which the language analysis information is corrected by the language analysis information correction step. and a plurality of pre accordance document feature vectors generated by the document feature vector generation step It classifies the document information to perform a document classification process generate a plurality of document group consisting of the document information,
The document extracting method, wherein the information extracting step obtains specific information according to tag information attached to the structured document . A document classification method realized by executing a program prepared in advance on a computer,
By executing the program, the computer, the document input step of inputting document information to be classified, analyzed the document information input by the document input step, the language analysis information is a result of the morphological analysis a language analysis step of obtaining, analyzing the document information input by the document input step, an information extraction step of extracting specific information is accompanying information or proper name, the language analysis information obtained by the language analysis step A document feature vector generating step for generating a document feature vector for the document information according to the method, and removing the specific information from the document feature vector in accordance with the output of the specific information by the information extracting step. and document feature vector modification step of modifying a vector of the document feature vector correction process Osamu Of classifying a plurality of the document information in accordance with the output, perform the a document classification process generate a plurality of document group consisting of the document information,
The document extracting method, wherein the information extracting step obtains specific information according to tag information attached to the structured document . 7. The document classification method according to claim 5 , wherein the information extraction step obtains specific information by extracting proper noun expressions. Further, by executing the program, the computer executes a display step of displaying the specific information extracted by the information extraction means, extracts information selection step of accepting a selection of one or more specific information, the The document classification method according to any one of claims 5 to 7 , wherein: 9. A computer-readable recording medium on which a program for causing a computer to execute the document classification method according to claim 5 is recorded.

Images