WO2014097670A1 - 文書分類装置及びプログラム - Google Patents
文書分類装置及びプログラム Download PDFInfo
- Publication number
- WO2014097670A1 WO2014097670A1 PCT/JP2013/068852 JP2013068852W WO2014097670A1 WO 2014097670 A1 WO2014097670 A1 WO 2014097670A1 JP 2013068852 W JP2013068852 W JP 2013068852W WO 2014097670 A1 WO2014097670 A1 WO 2014097670A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- category
- document
- cluster
- document data
- classification device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
- G06F16/355—Creation or modification of classes or clusters
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/93—Document management systems
Definitions
- the present invention relates to a document classification apparatus and program for classifying document data.
- Patent Document 1 discloses a method for determining the category of unclassified data by comparing the characteristics of the category with unclassified data when there is learning data classified in advance into a specific category. Has been.
- Patent Document 2 discloses a method in which each document is divided into clusters based on the feature amount of the document, and the feature word of each cluster is used as the category of the document belonging to the cluster.
- the category of the document data may be “summary” and “history”, or may be “summary” and “configuration”.
- the classification method varies depending on the person, and therefore the category assigned to the document data may differ depending on the person. As described above, it is difficult to ensure the independence of the category, and the category classification fluctuates depending on the person. Therefore, in the learning data that is classified in advance into the specific category, the learning data and the category The combination with may not be accurate.
- the category setting cost increases as the category is defined more finely, and the category classification may fluctuate depending on the person to whom the category is assigned.
- the category setting cost increases as the category is defined more finely, and the category classification may fluctuate depending on the person to whom the category is assigned.
- the category setting cost increases as the category is defined more finely, and the category classification may fluctuate depending on the person to whom the category is assigned.
- An object of the present invention is to provide a document classification device and a program capable of assigning a category expressing the ambiguity to the document data even when the definition of the category previously assigned to the document data is ambiguous. Is to provide.
- feature extraction means for extracting feature information from each of a plurality of document data previously classified into a specific category, and document data having similar appearance frequency of the feature information are classified into the same cluster.
- a category updating unit for adding the category of the other document data classified into the same cluster as its own category to the document data classified into the same cluster. It is.
- the invention according to claim 2 is the document classification apparatus according to claim 1, wherein the feature extraction unit extracts information related to the specific category from each of the plurality of document data as feature information. It is characterized by that.
- the invention according to claim 3 is the document classification device according to claim 1 or 2, wherein the category updating means sets the appearance ratio of the category of each document data in the same cluster to the same cluster. It is added to the classified document data.
- the invention according to claim 4 is the document classification device according to any one of claims 1 to 3, wherein when the search query is received and the information about the category is included in the search query,
- the system further includes a search unit that searches a category previously given to document data and a category given by the category update unit, performs a search using the search query, and outputs a search result.
- the invention according to claim 5 is the document classification device according to claim 4, wherein the search means outputs a search result according to the appearance ratio of the category of each document data in the same cluster.
- the invention according to claim 6 is the document classification device according to claim 4 or 5, wherein the category of the document data searched by the search query matches the information about the category included in the search query. If not, it further comprises update means for changing the category of the retrieved document data to a category included in the search query.
- the invention according to claim 7 is the document classification device according to any one of claims 1 to 6, wherein the identification is created based on the document data to which the category is given by the category updating means.
- the apparatus further comprises category classification means for assigning a category to unclassified document data using a container.
- the invention according to claim 8 is the document classification apparatus according to claim 7, wherein the category classification unit adds a category having the highest appearance ratio to the unclassified document data. .
- the invention according to claim 9 is the document classification apparatus according to claim 7 or claim 8, wherein the unclassified document data processed by the category classification unit is converted into the feature extraction unit, the clustering unit, and the like. And a means for adding to the document data to be processed by the category updating means.
- a tenth aspect of the present invention is the document classification device according to any one of the first to ninth aspects, wherein the feature extraction unit is configured to change the feature information according to the appearance frequency of the feature information in the document data. Characteristic information used for clustering by the clustering means is selected.
- the invention according to claim 11 is the document classification device according to any one of claims 1 to 10, wherein the category update unit is configured to determine an appearance ratio of each document data category in the same category. Accordingly, the category is selected and assigned to the document data.
- a step of extracting feature information from each of a plurality of document data previously classified into a specific category in a computer, and document data having similar appearance frequency of the feature information in the same cluster A program for executing a classification step and a step of assigning the category of the other document data classified into the same cluster to the document data classified into the same cluster as its own category.
- the accuracy of document classification is improved as compared with the case where the configuration of the present invention is not provided.
- the third aspect of the present invention it is possible to assign the category feature ratio to the document data.
- the accuracy of document classification is improved as compared with the case where the configuration of the present invention is not provided.
- the accuracy of document classification is improved as compared with the case where the configuration of the present invention is not provided.
- FIG. 3 is a flowchart showing an operation example 1; 10 is a flowchart showing an operation example 2; It is a table
- FIG. 1 shows an example of a document classification apparatus according to an embodiment of the present invention.
- the document classification apparatus 1 includes a storage unit 10, a feature extraction unit 16, a clustering unit 18, a cluster analysis unit 20, a cluster category storage unit 22, and a category update unit 24.
- a new category is given in consideration of the ambiguity of the category definition.
- the storage unit 10 includes a data storage unit 12 and a category storage unit 14.
- the data storage unit 12 stores learning data classified in advance into a specific category.
- the learning data is, for example, document data composed of a single sentence or a plurality of sentences.
- the learning data may be any data as long as it is classified in advance into a specific category, regardless of the type of category or the quality of learning data. For example, it may be data automatically collected using the document structure, with the titles of the chapters and paragraphs of the document as categories and the documents inside the chapters and paragraphs as learning data.
- Learning data identification information (for example, an ID number) for identifying the learning data is assigned to the learning data, and category information indicating a category of the learning data is associated with the learning data.
- the category storage unit 14 stores category information associated with the learning data.
- the feature extraction unit 16 extracts feature information from each of the plurality of learning data, and vectorizes each learning data using the extracted feature information.
- the feature extraction unit 16 uses one or more of morpheme information, part-of-speech information, syntax / semantic information, character / word N-gram, arbitrary word, number of punctuation marks, number of past tense words, etc. as feature information Extract as These pieces of information become vector features.
- the score of each feature may be a binary value (0 or 1), or may be an appearance number, a Tf-idf (Term Frequency-Inverse Document Frequency) value, or the like.
- the feature extraction unit 16 may treat synonyms as the same feature by using a dictionary or the like, or set a threshold value for the Tf-idf value to remove a feature that may be a noise, and compress the dimension of the vector. May be. By removing features that can cause noise, the accuracy of document classification is improved. For example, a word having an idf value equal to or lower than the lower threshold value can be determined to be a general word, and thus may not be used as a vector feature. In addition, a word whose idf value is equal to or greater than the upper threshold may not be used as a feature because a discriminator may greatly depend on the feature. As described above, the feature extraction unit 16 may select feature information used for clustering by the clustering unit 18 according to the feature score.
- the clustering unit 18 classifies the learning data having similar feature information into the same cluster based on the feature information extracted by the feature extraction unit 16. Specifically, the clustering unit 18 clusters the vector-represented learning data by using a method such as a repeated vision method, a K-means method, or an LDA (Lent Dirichlet Allocation) method. For example, the clustering unit 18 classifies learning data having similar appearance frequency of feature information into the same cluster.
- the clustering unit 18 gives cluster identification information (for example, an ID number) for identifying a cluster to each cluster.
- the cluster analysis unit 20 analyzes the configuration of the category in the cluster based on the category information associated with the learning data belonging to the cluster, and outputs the analysis result to the cluster category storage unit 22. Specifically, based on the category information associated with the learning data belonging to the cluster, the cluster analysis unit 20 determines the number of occurrences of each learning data category belonging to the same cluster in the cluster, Obtain the appearance ratio.
- the appearance ratio of each category is the number of appearances of each category with respect to the number of appearances of all categories belonging to the same cluster.
- the cluster analysis unit 20 can generate noise.
- the learning data belonging to may be removed. Further, among categories belonging to the same cluster, the number of categories is smaller than that of other categories, and a category whose difference is equal to or greater than a predetermined threshold value may be noise. Therefore, the cluster analysis unit 20 removes the category. May be. The accuracy of document classification is improved by removing clusters and learning data that may be noise. For example, a category whose appearance ratio is equal to or less than a predetermined threshold value may be removed from the cluster. Further, the cluster analysis unit 20 may limit the number of categories included in one cluster. For example, the cluster analysis unit 20 may include a predetermined number of categories from a category having a high appearance ratio in the cluster and may not include other categories in the cluster. As described above, the cluster analysis unit 20 may select a category to be used for the cluster category according to the category appearance ratio.
- the cluster category storage unit 22 stores cluster category information as an analysis result of the cluster analysis unit 20.
- the category updating unit 24 newly defines each category belonging to the same cluster and the appearance ratio of each category as a cluster category, and updates the category of each learning data by using the cluster category information indicating the cluster category. . That is, the category update unit 24 gives the learning data belonging to the same cluster also the category of another learning data belonging to the same cluster as its own category. For example, the category update unit 24 assigns the cluster identification information of the cluster to each learning data belonging to the same cluster, and associates the cluster category information of the cluster. In this way, each learning data is classified into the cluster category. Each cluster category is assigned cluster identification information.
- the category storage unit 14 stores cluster category information associated with the learning data. As described above, since the cluster category is given to each learning data, each learning data is classified into the original category and also into the cluster category. That is, each learning data is associated with category information indicating the original category and cluster category information indicating the cluster category.
- storage unit 10 and the cluster category storage unit 22 may be provided outside the document classification device 1 without being provided in the document classification device 1.
- the feature extraction unit 16 reads a plurality of learning data from the data storage unit 12 and extracts feature information from each of the plurality of learning data by analyzing the plurality of learning data. Is vectorized (S01).
- the clustering unit 18 classifies the plurality of learning data expressed as vectors into clusters (S02).
- the cluster analysis unit 20 analyzes the configuration of the categories in the cluster based on the category information associated with the learning data belonging to the cluster (S03).
- the category update unit 24 defines each category belonging to the same cluster and the appearance ratio of each category as a cluster category, and associates the cluster category information of the cluster with each learning data belonging to the same cluster, so that each learning data
- the category is updated (S04).
- the feature extraction unit 16 extracts feature information from each of the plurality of learning data by analyzing the plurality of learning data (S10).
- the feature extraction unit 16 removes feature information that can be noise from the extracted feature information, selects feature information that cannot be noise, and vectorizes each learning data (S11).
- the feature extraction unit 16 treats synonyms in the extracted feature information as the same feature, or removes unnecessary features based on the Tf-idf value, and vectorizes each learning data.
- the clustering unit 18 classifies the plurality of learning data expressed in vectors into clusters (S12), and the cluster analysis unit 20 analyzes the configuration of categories in the cluster (S13).
- the cluster analysis unit 20 removes learning data of clusters that can be noise (S14).
- the cluster analysis unit 20 may remove categories whose number is smaller than other categories and whose difference is equal to or greater than a threshold value.
- the category update unit 24 updates the category of each learning data by associating the cluster category information with each learning data belonging to the same cluster (S15).
- FIG. 4 shows an example of learning data classified in advance into a specific category.
- Each learning data is stored in the storage unit 10, and category information indicating a category of each learning data is stored in the category storage unit 14.
- the content word (word) is used as a vector feature and the number of appearances is used as a score.
- WID in FIG. 5 is identification information given to each word by the feature extraction unit 16.
- WID represents the index number of the vector, and the number of appearances represents the score in the index number.
- the feature extraction unit 16 may give the same WID to the synonym by using a dictionary or the like.
- a threshold value may be set for the score (number of appearances, appearance frequency), and features having a score equal to or lower than the threshold value may be removed. That is, feature information used for clustering by the clustering unit 18 may be selected according to the score.
- the clustering unit 18 classifies the vector-represented learning data into clusters by a predetermined method.
- FIG. 6 shows the result of clustering.
- CID in FIG. 6 is cluster identification information for identifying a cluster.
- the category of learning data classified into each cluster belongs to the corresponding cluster.
- the number of appearances of the category in the cluster means the number of learning data that is classified in advance into the category and classified into the cluster by the clustering unit 18. Note that the cluster analysis unit 20 may remove clusters and learning data that may be noise.
- the cluster 3 includes 91 “configuration” categories and 52 “outline” categories. This is because 91 pieces of learning data previously classified into the “configuration” category are classified into the cluster 3 by the clustering unit 18, and 52 pieces of learning data previously classified into the “summary” category are classified into the clustering unit 18. Means that it was classified into category 3.
- the appearance ratio of the “configuration” category is 0.27 (27%), and the appearance ratio of the “summary” category is 0.16 (16%).
- the learning data is classified into clusters based on the feature information of the learning data previously classified into a specific category, and the category belonging to the cluster and the appearance ratio of the category are newly defined as the cluster category.
- a cluster category expressing the ambiguousness is given to the learning data. Since words have various meanings and concepts, the categories may not be independent of each other, and the categories previously assigned to the learning data may not accurately represent the characteristics of the learning data. According to this embodiment, since the cluster category expressing the ambiguity of the category is assigned to the learning data, a category that more accurately represents the characteristics of the learning data is assigned to the learning data.
- the “summary” category was originally given, but the “history” element may be stronger than the “summary”.
- the correspondence between the category, cluster category, and learning data may be displayed on a display device (not shown).
- the correspondence relationship may be displayed in a table format shown in FIG.
- the information for the user to grasp the concept and meaning of the learning data and the information for the user to reassign the category are provided to the user.
- the document classification device 1A includes a processing unit 30, an input unit 42, and an output unit 44 in addition to the configuration of the document classification device 1 described above.
- the processing unit 30 includes a discriminator creation unit 32, a category classification unit 34, a search unit 36, and a learning data update unit 38.
- the unclassified data storage unit 46 stores document data that is not classified into a specific category (referred to as “unclassified data”).
- the input unit 42 is a user interface, for example, and accepts input of information by the user.
- the output unit 44 is, for example, a display device, a network interface, or a printing device, and outputs the processing result of the processing unit 30.
- the discriminator creation unit 32 creates a discriminator (“category discriminator”) from the learning data classified into a specific category. For example, a principal component vector of learning data is used as a discriminator. Further, the discriminator creating unit 32 may create a discriminator (“cluster category discriminator”) from the learning data classified into the cluster category.
- the category classification unit 34 obtains a category of unclassified data using a discriminator, and classifies the unclassified data into categories.
- the category classification unit 34 may obtain a cluster category of unclassified data using a discriminator and classify the unclassified data into cluster categories.
- the search unit 36 searches for data using the search query received by the input unit 42 using the learning data to which the cluster category is assigned as a search target.
- the search unit 36 calculates a score based on the degree of coincidence between the learning data and the search query, and outputs a search result (a result in which higher score data is higher) in descending score order.
- the search unit 36 also searches the cluster category, and represents the degree of matching between the search query and the search target data according to the configuration of the category belonging to the cluster category. Adjust the score. For example, the score is adjusted by attaching a weight to the score according to the appearance ratio of the category that matches the term included in the search query. Thereby, the score of the data for learning classified into the said category becomes high, so that the appearance ratio of the category which corresponds to the term contained in a search query is large.
- the learning data update unit 38 updates various data when a category or a cluster category is assigned to unclassified data. For example, the learning data update unit 38 causes the data storage unit 12 to store unclassified data to which a category or cluster category is assigned as learning data. The learning data update unit 38 may update the data stored in the category storage unit 14 or the cluster category storage unit 22. Further, the learning data update unit 38 may update the category of the learning data based on the search result of the search unit 36.
- the category classification unit 34 obtains a cluster category of unclassified data using the cluster category classifier created by the classifier creation unit 32, and classifies the unclassified data into cluster categories (S20). Then, a category is selected from the categories belonging to the cluster category (S21). For example, the category classification unit 34 may give the category having the highest appearance ratio among the categories belonging to the cluster category to the unclassified data. Since the category with the highest appearance ratio may more accurately represent the characteristics of the unclassified data, adding the category with the highest appearance ratio to the unclassified data makes the characteristics of the unclassified data more The category that is accurately represented is assigned to the unclassified data.
- the output unit 44 may display a list of categories belonging to the cluster category, and the user may select a category using the input unit 42.
- the category classification unit 34 assigns the category selected by the user to the unclassified data. Only the cluster category may be added to the unclassified data.
- the learning data update unit 38 updates various data (S22).
- the learning data update unit 38 causes the data storage unit 12 to store unclassified data to which a category or cluster category is assigned as learning data.
- the learning data update unit 38 increments the number of appearances of the selected category by one, and updates the number of cluster elements and the appearance ratio of the category.
- the classifier creation unit 32 may update the classifier.
- the information that forms the basis for creating the cluster category is updated, so the accuracy of classification of the cluster category is improved. To do.
- the operation example 3 may be applied when it is desired to add a category or heading to document data to which a category or paragraph heading is not added.
- the category classification unit 34 classifies the target data into cluster categories using a discriminator, and the cluster category, the category belonging to the cluster category, the category having the highest appearance ratio, or the category belonging to the cluster category by the user.
- the selected category is assigned to the target data.
- the learning data update unit 38 updates various data.
- the processing unit 30 may not include the search unit 36.
- the search unit 36 sets the learning data to which the cluster category is assigned as a search target, and scores according to the degree of matching between the search target learning data and the search query. Is calculated (S30).
- the search unit 36 If the search query does not include category terms (for example, “Summary”, “History”, etc.) (No in S31), the search unit 36 outputs the search result in Step S30, and the search process ends. .
- the output unit 44 displays the search results in descending score order.
- the search unit 36 searches the cluster category as a search target (S32).
- the search unit 36 performs a search using the cluster category as a search target, and includes the cluster category including the term “history” of the search query.
- learning data to which the cluster category is assigned is extracted.
- the search part 36 makes the score of the data for learning classified into the said category high, and re-ranks the search result in step S30, so that the appearance ratio of the category which corresponds with the term of a search query is large (S33). .
- the output unit 44 displays the reranked search result.
- the user refers to the reranked search result and selects arbitrary learning data using the input unit 42 (S34).
- the learning data update unit 38 updates various data (S35). This update process will be described with reference to FIG.
- the learning data update unit 38 updates the number of elements of the cluster, the number of appearances of the category, and the appearance ratio according to the selection of the category.
- the cluster category is also set as the search target, so that the search data is not extracted when the search query does not exist in the text of the learning data and only the learning data is set as the search target. Is done.
- learning data that can be related to the search query is extracted. For example, if a search query is not included in the text of the learning data and the search query indicates the concept or meaning of the learning data, the search query matches the concept or meaning if only the learning data is targeted for search.
- the learning data is not extracted, the learning data having the same concept and meaning as the search query is extracted by setting the cluster category as the search target. Therefore, unlike the case of simply searching for the text of the learning data, it can be said that the learning data is searched including the concept and meaning.
- the document classification devices 1 and 1A are realized by cooperation of hardware resources and software as an example.
- the document classification devices 1 and 1A include a processor such as a CPU (not shown).
- the processor executes the functions of the feature extraction unit 16, clustering unit 18, cluster analysis unit 20, category update unit 24, and processing unit 30 described above by reading and executing a program stored in a storage device (not shown).
- the program is stored in a storage device such as a hard disk drive (HDD) via a recording medium such as a CD or DVD, or via communication means such as a network.
- the program may be stored in advance in a storage device such as a hard disk drive.
- a function stored in a storage device such as a hard disk drive is read into a memory such as a RAM and executed by a processor, thereby realizing the functions of the above-described units.
- 1, 1A document classification device 10 storage unit, 12 data storage unit, 14 category storage unit, 16 feature extraction unit, 18 clustering unit, 20 cluster analysis unit, 22 cluster category storage unit, 24 category update unit, 30 processing unit, 32 classifier creation unit, 34 category classification unit, 36 search unit, 38 learning data update unit, 42 input unit, 44 output unit, 46 unclassified data storage unit.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
次に、図2に示すフローチャートを参照して、文書分類装置1の動作例1を説明する。まず、特徴抽出部16は、複数の学習用データをデータ記憶部12から読み出し、複数の学習用データを解析することで複数の学習用データのそれぞれから特徴情報を抽出し、学各学習用データをベクトル化する(S01)。次に、クラスタリング部18は、ベクトル表現された複数の学習用データをクラスタに分類する(S02)。そして、クラスタ分析部20は、クラスタに属する学習用データに関連付けられたカテゴリ情報に基づき、当該クラスタ内のカテゴリの構成を分析する(S03)。カテゴリ更新部24は、同一クラスタに属する各カテゴリと各カテゴリの出現割合とをクラスタカテゴリとして定義し、同一クラスタに属する各学習用データに当該クラスタのクラスタカテゴリ情報を関連付けることで、各学習用データのカテゴリを更新する(S04)。
次に、図3に示すフローチャートを参照して、文書分類装置1の動作例2を説明する。まず、特徴抽出部16は、複数の学習用データを解析することで複数の学習用データのそれぞれから特徴情報を抽出する(S10)。次に、特徴抽出部16は、抽出した特徴情報のうちノイズとなり得る特徴情報を除去し、ノイズとなり得ない特徴情報を選択して各学習用データをベクトル化する(S11)。例えば、特徴抽出部16は、抽出した特徴情報のうち同義語を同一素性として扱い、又は、Tf-idf値に基づいて不要な素性を除去し、各学習用データをベクトル化する。そして、クラスタリング部18は、ベクトル表現された複数の学習用データをクラスタに分類し(S12)、クラスタ分析部20は、クラスタ内のカテゴリの構成を分析する(S13)。また、クラスタ分析部20は、ノイズとなり得るクラスタの学習データを除去する(S14)。クラスタ分析部20は、数が他のカテゴリよりも少なく、その数の差が閾値以上となるカテゴリを除去してもよい。そして、カテゴリ更新部24は、同一クラスタに属する各学習用データにクラスタカテゴリ情報を関連付けることで、各学習用データのカテゴリを更新する(S15)。
次に、文書分類装置1の動作について具体例を挙げて説明する。図4に、予め特定のカテゴリに分類された学習用データの一例を示す。例えば、学習用データ識別情報(ID=1,2)が付された学習用データは「概要」というカテゴリに予め分類されており、ID=3の学習用データは「経歴」というカテゴリに予め分類されており、ID=4の学習用データは「構成」というカテゴリに予め分類されている。各学習用データは記憶部10に記憶されており、各学習用データのカテゴリを示すカテゴリ情報は、カテゴリ記憶部14に記憶されている。
(1)ID=1の学習用データは、23%の確率で「経歴」カテゴリ、20%の確率で「概要」カテゴリに分類されるデータである。
(2)ID=1の学習用データは、23%の「経歴」カテゴリの特徴と20%の「概要」カテゴリの特徴とを有するデータである。
(3)ID=1の学習用データは、「経歴」カテゴリの特徴と「概要」カテゴリの特徴とを有するデータである。
ID=1の学習用データは「概要」カテゴリに予め分類されていたが、「経歴」カテゴリの出現割合(23%)が「概要」カテゴリの出現割合(20%)よりも大きいため、「概要」カテゴリよりも「経歴」カテゴリの特徴が強いと分析されている。他のクラスカテゴリについても同様に解釈され得る。
次に、図9を参照して、変形例に係る文書分類装置について説明する。変形例に係る文書分類装置1Aは、上記の文書分類装置1の構成に加えて、処理部30、入力部42及び出力部44を備えている。処理部30は、識別器作成部32、カテゴリ分類部34、検索部36及び学習用データ更新部38を備えている。未分類データ記憶部46には、特定のカテゴリに分類されていない文書データ(「未分類データ」と称する)が記憶されている。
次に、図10を参照して、変形例に係る文書分類装置1Aの動作例(動作例3)を説明する。動作例3では、未分類データにカテゴリやクラスタカテゴリを付与して学習用データを更新する。
次に、図11を参照して、変形例に係る文書分類装置1Aの別の動作例(動作例4)を説明する。動作例4では、クラスタカテゴリが付与された学習用データを検索対象として検索を行い、その検索結果に基づいて学習用データを更新する。
Claims (12)
- 予め特定のカテゴリに分類された複数の文書データのそれぞれから特徴情報を抽出する特徴抽出手段と、
前記特徴情報の出現頻度が類似する文書データを同一クラスタに分類するクラスタリング手段と、
同一クラスタに分類された文書データに、前記同一クラスタに分類された他の文書データのカテゴリも自己のカテゴリとして付与するカテゴリ更新手段と、
を有することを特徴とする文書分類装置。 - 請求項1に記載の文書分類装置であって、
前記特徴抽出手段は、前記特定のカテゴリに関連する情報を特徴情報として前記複数の文書データのそれぞれから抽出する、
ことを特徴とする文書分類装置。 - 請求項1又は請求項2に記載の文書分類装置であって、
前記カテゴリ更新手段は、前記同一クラスタにおける各文書データのカテゴリの出現割合を、前記同一クラスタに分類された文書データに付与する、
ことを特徴とする文書分類装置。 - 請求項1から請求項3のいずれか一項に記載の文書分類装置であって、
検索クエリを受け、前記検索クエリにカテゴリに関する情報が含まれている場合、文書データに予め付与されたカテゴリ及び前記カテゴリ更新手段によって付与されたカテゴリも検索対象とし、前記検索クエリを用いて検索し、検索結果を出力する検索手段を更に有する、
ことを特徴とする文書分類装置。 - 請求項4に記載の文書分類装置であって、
前記検索手段は、前記同一クラスタにおける各文書データのカテゴリの出現割合に応じた検索結果を出力する、
ことを特徴とする文書分類装置。 - 請求項4又は請求項5に記載の文書分類装置であって、
前記検索クエリによって検索された文書データのカテゴリと前記検索クエリに含まれるカテゴリに関する情報とが一致しない場合、前記検索された文書データのカテゴリを、前記検索クエリに含まれるカテゴリに変更する更新手段を更に有する、
ことを特徴とする文書分類装置。 - 請求項1から請求項6のいずれか一項に記載の文書分類装置であって、
前記カテゴリ更新手段によってカテゴリが付与された文書データに基づいて作成された識別器を用いて未分類の文書データにカテゴリを付与するカテゴリ分類手段を更に有する、
ことを特徴とする文書分類装置。 - 請求項7に記載の文書分類装置であって、
前記カテゴリ分類手段は、出現割合が最大のカテゴリを前記未分類の文書データに付与する、
ことを特徴とする文書分類装置。 - 請求項7又は請求項8に記載の文書分類装置であって、
前記カテゴリ分類手段によって処理された前記未分類の文書データを、前記特徴抽出手段、前記クラスタリング手段及び前記カテゴリ更新手段の処理対象の文書データに追加する手段を更に有する、
ことを特徴とする文書分類装置。 - 請求項1から請求項9のいずれか一項に記載の文書分類装置であって、
前記特徴抽出手段は、文書データにおける特徴情報の出現頻度に応じて、前記クラスタリング手段によるクラスタ分けに用いられる特徴情報を選択する、
ことを特徴とする文書分類装置。 - 請求項1から請求項10のいずれか一項に記載の文書分類装置であって、
前記カテゴリ更新手段は、前記同一カテゴリにおける各文書データのカテゴリの出現割合に応じてカテゴリを選択して文書データに付与する、
ことを特徴とする文書分類装置。 - コンピュータに、
予め特定のカテゴリに分類された複数の文書データのそれぞれから特徴情報を抽出するステップと、
前記特徴情報の出現頻度が類似する文書データを同一クラスタに分類するステップと、
同一クラスタに分類された文書データに、前記同一クラスタに分類された他の文書データのカテゴリも自己のカテゴリとして付与するステップと、
を実行させることを特徴とするプログラム。
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| AU2013365452A AU2013365452B2 (en) | 2012-12-21 | 2013-07-10 | Document classification device and program |
| US14/717,034 US10353925B2 (en) | 2012-12-21 | 2015-05-20 | Document classification device, document classification method, and computer readable medium |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2012279624A JP6007784B2 (ja) | 2012-12-21 | 2012-12-21 | 文書分類装置及びプログラム |
| JP2012-279624 | 2012-12-21 |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| US14/717,034 Continuation US10353925B2 (en) | 2012-12-21 | 2015-05-20 | Document classification device, document classification method, and computer readable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2014097670A1 true WO2014097670A1 (ja) | 2014-06-26 |
Family
ID=50978024
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/JP2013/068852 Ceased WO2014097670A1 (ja) | 2012-12-21 | 2013-07-10 | 文書分類装置及びプログラム |
Country Status (4)
| Country | Link |
|---|---|
| US (1) | US10353925B2 (ja) |
| JP (1) | JP6007784B2 (ja) |
| AU (1) | AU2013365452B2 (ja) |
| WO (1) | WO2014097670A1 (ja) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017029758A1 (ja) * | 2015-08-20 | 2017-02-23 | 三菱電機株式会社 | 学習装置および学習識別システム |
| CN109739975A (zh) * | 2018-11-15 | 2019-05-10 | 东软集团股份有限公司 | 热点事件抽取方法、装置、可读存储介质及电子设备 |
| CN111078987A (zh) * | 2019-12-21 | 2020-04-28 | 武汉比特空间科技有限公司 | 基于互联网的同类别数据提取整理系统 |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9858330B2 (en) * | 2013-10-21 | 2018-01-02 | Agile Legal Technology | Content categorization system |
| US20160019284A1 (en) * | 2014-07-18 | 2016-01-21 | Linkedln Corporation | Search engine using name clustering |
| EP3403187A4 (en) | 2016-01-14 | 2019-07-31 | Sumo Logic | SINGLE CLICK DELTA ANALYSIS |
| US10606899B2 (en) | 2016-05-23 | 2020-03-31 | International Business Machines Corporation | Categorically filtering search results |
| JP7198900B2 (ja) * | 2017-08-14 | 2023-01-04 | ヤフー株式会社 | 生成装置、生成方法、及び生成プログラム |
| JP6985059B2 (ja) * | 2017-08-14 | 2021-12-22 | ヤフー株式会社 | 生成装置、生成方法、及び生成プログラム |
| US20200026767A1 (en) * | 2018-07-17 | 2020-01-23 | Fuji Xerox Co., Ltd. | System and method for generating titles for summarizing conversational documents |
| US11996086B2 (en) * | 2019-08-19 | 2024-05-28 | Nippon Telegraph And Telephone Corporation | Estimation device, estimation method, and estimation program |
| WO2023089729A1 (ja) * | 2021-11-18 | 2023-05-25 | 三菱電機株式会社 | 文書検索装置、文書検索方法及び文書検索プログラム |
| US20240386062A1 (en) * | 2023-05-16 | 2024-11-21 | Sap Se | Label Extraction and Recommendation Based on Data Asset Metadata |
| US12254014B1 (en) * | 2024-02-23 | 2025-03-18 | Servicenow, Inc. | Document creation with guided generative artificial intelligence |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH1078971A (ja) * | 1996-09-02 | 1998-03-24 | Canon Inc | 文書分類装置及び文書分類方法 |
| JP2002041544A (ja) * | 2000-07-25 | 2002-02-08 | Toshiba Corp | テキスト情報分析装置 |
| JP2009070321A (ja) * | 2007-09-18 | 2009-04-02 | Fuji Xerox Co Ltd | 文書分類装置、及び文書分類プログラム |
| JP2009259250A (ja) * | 2008-04-18 | 2009-11-05 | Nec (China) Co Ltd | 文書の分類器を生成する方法とそのシステム |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3488063B2 (ja) | 1997-12-04 | 2004-01-19 | 株式会社エヌ・ティ・ティ・データ | 情報分類方法、装置及びシステム |
| KR20020049164A (ko) * | 2000-12-19 | 2002-06-26 | 오길록 | 유전자 알고리즘을 이용한 카테고리 학습과 단어클러스터에 의한 문서 자동 분류 시스템 및 그 방법 |
| JP4407272B2 (ja) | 2003-12-22 | 2010-02-03 | 富士ゼロックス株式会社 | 文書分類方法、文書分類装置及び文書分類プログラム |
| US20070011020A1 (en) * | 2005-07-05 | 2007-01-11 | Martin Anthony G | Categorization of locations and documents in a computer network |
| JP5019315B2 (ja) * | 2007-04-23 | 2012-09-05 | 公立大学法人広島市立大学 | 情報処理装置、情報処理方法、及びプログラム |
| US20090077028A1 (en) * | 2007-09-18 | 2009-03-19 | Gosby Desiree D G | Web services access to classification engines |
-
2012
- 2012-12-21 JP JP2012279624A patent/JP6007784B2/ja active Active
-
2013
- 2013-07-10 AU AU2013365452A patent/AU2013365452B2/en active Active
- 2013-07-10 WO PCT/JP2013/068852 patent/WO2014097670A1/ja not_active Ceased
-
2015
- 2015-05-20 US US14/717,034 patent/US10353925B2/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH1078971A (ja) * | 1996-09-02 | 1998-03-24 | Canon Inc | 文書分類装置及び文書分類方法 |
| JP2002041544A (ja) * | 2000-07-25 | 2002-02-08 | Toshiba Corp | テキスト情報分析装置 |
| JP2009070321A (ja) * | 2007-09-18 | 2009-04-02 | Fuji Xerox Co Ltd | 文書分類装置、及び文書分類プログラム |
| JP2009259250A (ja) * | 2008-04-18 | 2009-11-05 | Nec (China) Co Ltd | 文書の分類器を生成する方法とそのシステム |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017029758A1 (ja) * | 2015-08-20 | 2017-02-23 | 三菱電機株式会社 | 学習装置および学習識別システム |
| JPWO2017029758A1 (ja) * | 2015-08-20 | 2017-11-09 | 三菱電機株式会社 | 学習装置および学習識別システム |
| CN107924493A (zh) * | 2015-08-20 | 2018-04-17 | 三菱电机株式会社 | 学习装置和学习识别系统 |
| CN109739975A (zh) * | 2018-11-15 | 2019-05-10 | 东软集团股份有限公司 | 热点事件抽取方法、装置、可读存储介质及电子设备 |
| CN109739975B (zh) * | 2018-11-15 | 2021-03-09 | 东软集团股份有限公司 | 热点事件抽取方法、装置、可读存储介质及电子设备 |
| CN111078987A (zh) * | 2019-12-21 | 2020-04-28 | 武汉比特空间科技有限公司 | 基于互联网的同类别数据提取整理系统 |
Also Published As
| Publication number | Publication date |
|---|---|
| JP6007784B2 (ja) | 2016-10-12 |
| AU2013365452A1 (en) | 2015-06-04 |
| US10353925B2 (en) | 2019-07-16 |
| US20150254332A1 (en) | 2015-09-10 |
| AU2013365452B2 (en) | 2017-05-25 |
| JP2014123286A (ja) | 2014-07-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6007784B2 (ja) | 文書分類装置及びプログラム | |
| CN108701161B (zh) | 为搜索查询提供图像 | |
| US10268758B2 (en) | Method and system of acquiring semantic information, keyword expansion and keyword search thereof | |
| US9965459B2 (en) | Providing contextual information associated with a source document using information from external reference documents | |
| US20160189057A1 (en) | Computer implemented system and method for categorizing data | |
| RU2547213C2 (ru) | Присвоение применимых на практике атрибутов данных, которые описывают идентичность личности | |
| US20150199609A1 (en) | Self-learning system for determining the sentiment conveyed by an input text | |
| US9898464B2 (en) | Information extraction supporting apparatus and method | |
| TW201913411A (zh) | 同義詞辭典作成裝置、記錄有同義詞辭典作成程式之電腦可讀取之記錄媒體及同義詞辭典作成方法 | |
| US10515267B2 (en) | Author identification based on functional summarization | |
| CN103608805B (zh) | 辞典产生装置及方法 | |
| US20150205860A1 (en) | Information retrieval device, information retrieval method, and information retrieval program | |
| CN115438174A (zh) | 文本处理装置、方法、设备和计算机可读存储介质 | |
| JP7537513B2 (ja) | レコメンド装置、レコメンドシステム、レコメンド方法およびプログラム | |
| CN119739838A (zh) | 多标签生成匹配的rag智能问答方法、装置、设备及介质 | |
| CN111639056A (zh) | 保存文件的方法及装置 | |
| US10353927B2 (en) | Categorizing columns in a data table | |
| US8135573B2 (en) | Apparatus, method, and computer program product for creating data for learning word translation | |
| JPWO2022113286A5 (ja) | ||
| US20150052140A1 (en) | Information processing apparatus, information processing method, and program | |
| JP2016110256A (ja) | 情報処理装置及び情報処理プログラム | |
| CN108733733B (zh) | 基于机器学习的生物医学文本分类方法、系统和存储介质 | |
| Anđelić et al. | Text classification based on named entities | |
| JP4640593B2 (ja) | 多言語文書検索装置および多言語文書検索方法、並びに、多言語文書を検索するプログラム | |
| US20220083736A1 (en) | Information processing apparatus and non-transitory computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 13864301 Country of ref document: EP Kind code of ref document: A1 |
|
| ENP | Entry into the national phase |
Ref document number: 2013365452 Country of ref document: AU Date of ref document: 20130710 Kind code of ref document: A |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 13864301 Country of ref document: EP Kind code of ref document: A1 |