JP2013196212A - Document division device, document division program and document division method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a document division device, a document division program and a document division method for efficiently performing the parallel processing of clustering to a document.SOLUTION: A generation part 41 generates index data 31 as feature information showing the feature of each of a plurality of documents of document data 30. An inter-document similarity calculation part 42 calculates similarity between a plurality of documents on the basis of generated index data 31, and stores the calculated inter-document similarity as inter-document similarity data 32. An inter-group similarity calculation part 44 calculates similarity between the groups of a document in which similarity between the documents of each document is equal to or less than an upper predetermined order, and stores the calculated inter-group similarity as inter-group similarity data 34. A classification part 45 classifies documents having high inter-group similarity into groups. A division part 46 divides a plurality of documents by group units stored in cluster data 35.

Description

  The present invention relates to a document dividing device, a document dividing program, and a document dividing method.

  2. Description of the Related Art Conventionally, a technique for calculating a similarity between a plurality of documents and clustering documents having a high similarity is known. This clustering is used, for example, when a frequently asked question is specified when an FAQ (Frequently Asked Question) is created for a query received at a help desk or the like.

  In clustering, for example, a plurality of documents are compared with each other in a brute force manner to obtain similar documents. As described above, in clustering, a plurality of documents are compared with each other and the similarity between the documents is obtained, so that the storage capacity used in the processing increases as the number of documents to be clustered increases and the processing time increases. Will also increase.

  Conventionally, parallel processing by a plurality of computers and processors is known as a method for reducing processing time and processing load when data processing is performed. Thus, for example, a method of preparing a plurality of data replicas and performing clustering in parallel with a plurality of computers and processors is conceivable.

JP 2003-263443 A

  However, when clustering is performed in parallel, a larger amount of storage capacity is required as the number of documents to be processed increases, and therefore, a copy of data is created, so that more storage capacity is consumed.

  The disclosed technique has been made in view of the above, and an object thereof is to provide a document dividing apparatus, a document dividing program, and a document dividing method capable of efficiently performing parallel processing of clustering on a document.

  The document dividing device disclosed in the present application includes a generation unit, an inter-document similarity calculation unit, an inter-set similarity calculation unit, a classification unit, and a division unit. The generation unit generates feature information indicating features of a plurality of documents to be clustered. The inter-document similarity calculation unit calculates the similarity between the plurality of documents based on the feature information generated by the generation unit. The inter-set similarity calculation unit calculates the similarity between sets of documents having a similarity between the documents within a predetermined upper level. The classification unit classifies documents having high similarity between sets into groups. The division unit divides the plurality of documents in units of groups classified by the classification unit.

  According to the document dividing device disclosed in the present application, it is possible to efficiently perform parallel processing of clustering on documents.

FIG. 1 is a diagram showing the overall configuration of the document dividing apparatus. FIG. 2 is a diagram illustrating a configuration example of inter-document similarity data. FIG. 3 is a diagram illustrating a configuration example of the similarity data between sets. FIG. 4 is an explanatory diagram for explaining the similarity between sets of documents. FIG. 5 is a diagram illustrating an example of a flow of processing for classifying documents. FIG. 6 is a diagram illustrating an example of a flow of processing for dividing a document. FIG. 7 is an explanatory diagram for explaining clustering. FIG. 8 is a diagram schematically showing the flow of processing for document data. FIG. 9 is a diagram illustrating an example of a document name of a clustering target document included in the document data and a word included in the document. FIG. 10 is a diagram illustrating an example of index data. FIG. 11 is a diagram showing an example of inter-document similarity data. FIG. 12 is a diagram illustrating an example of excluded document data. FIG. 13 is a diagram illustrating an example of inter-set similarity data. FIG. 14 is a diagram illustrating an example of cluster data. FIG. 15 is a diagram illustrating an example of distributed data. FIG. 16 is a flowchart illustrating the procedure of the dividing process. FIG. 17 is a flowchart showing the procedure of index generation processing. FIG. 18 is a flowchart showing the procedure of the inter-document similarity calculation process. FIG. 19 is a flowchart showing a procedure of excluded document specifying processing. FIG. 20 is a flowchart showing the similarity calculation processing between sets. FIG. 21 is a flowchart showing the classification process. FIG. 22 is a flowchart showing the dividing process. FIG. 23 is a flowchart illustrating the procedure of the clustering process. FIG. 24 is a diagram for describing an example of a computer that executes a division program.

  Embodiments of a document dividing device, a document dividing program, and a document dividing method disclosed in the present application will be described below in detail with reference to the drawings. Note that this embodiment does not limit the disclosed technology. Each embodiment can be appropriately combined within a range in which processing contents are not contradictory.

[Configuration of Document Splitting Apparatus 10]
A document dividing apparatus according to the first embodiment will be described. FIG. 1 is a diagram showing the overall configuration of the document dividing apparatus. The document dividing apparatus 10 is an apparatus that divides a plurality of documents into sets of documents having high similarity, and is a computer, for example.

  The document dividing apparatus 10 includes an external I / F (interface) unit 20, a storage unit 21, and a control unit 22.

  The external I / F unit 20 is an interface that inputs and outputs various types of information with the outside. The external I / F unit 20 may be an interface that inputs and outputs various types of information from an external device via a data transmission path such as a network, for example. The external I / F unit 20 may be an interface that inputs and outputs various types of information to and from a storage medium such as a flash memory. The external I / F unit 20 acquires various types of information by communicating with an external device or via a storage medium. For example, a plurality of documents that are clustering targets are input to the external I / F unit 20. Further, the external I / F unit 20 outputs the divided distributed data 36 to a computer or processor that performs clustering. In addition, the external I / F unit 20 inputs and outputs the result of clustering with a computer or processor that performs clustering.

  The storage unit 21 is a storage device such as a semiconductor memory element such as a flash memory, a hard disk, or an optical disk. The storage unit 21 is not limited to the type of storage device described above, and may be a RAM (Random Access Memory) or a ROM (Read Only Memory).

  The storage unit 21 stores an OS (Operating System) executed by the control unit 22 and various programs used for document division described later. Furthermore, the storage unit 21 stores various data used in the program executed by the control unit 22. As an example of such data, the storage unit 21 includes document data 30, index data 31, inter-document similarity data 32, excluded document data 33, inter-set similarity data 34, cluster data 35, and distributed data. 36 is stored.

  The document data 30 is data in which a plurality of documents to be clustered are stored. As an example, the document data 30 is stored in a registration unit 40 (to be described later) as document data of a plurality of documents to be clustered input to the external I / F unit 20. As another example, the document data 30 is referred to by a generation unit 41 and a division unit 46 described later.

  The index data 31 is data in which an index indicating which document includes words included in a plurality of documents stored in the document data 30 is stored. As an example, the index data 31 is generated and stored by a generation unit 41 described later. As another example, the index data 31 is referred to by an inter-document similarity calculation unit 42 described later.

  The inter-document similarity data 32 is data in which the similarity between each document stored in the document data 30 and each document including its own document is stored. As an example, the inter-document similarity data 32 is calculated and stored by an inter-document similarity calculation unit 42 described later. As another example, the inter-document similarity data 32 is referred to by an excluded document specifying unit 43, an inter-set similarity calculating unit 44, and an output unit 47, which will be described later.

  FIG. 2 is a diagram illustrating a configuration example of inter-document similarity data. For each document included in the document data 30, a document name for identifying each document is defined. When the document name is predetermined for each document and stored in the document data 30, the document name may be used. Further, the document name may be determined for each document included in the document data 30 in the document dividing apparatus 10. In the example of FIG. 2, for each document whose document names are “A” to “Z”, the document name of the other party whose similarity is equal to or more than a predetermined first threshold and the similarity are represented by [document name, similarity] It is arranged and stored in the order of high similarity in format. The degree of similarity indicates that the larger the value, the more similar. In this embodiment, the similarity is normalized to a range from 0.0 to 1.0. In the example of FIG. 2, the document A has a similarity of 1.0 to the document A, a similarity of 0.8 to the document B, and a similarity of 0.8 to the document C. The similarity with E is 0.7. Document B has a similarity with document B of 1.0, a similarity with document A of 0.8, a similarity with document D of 0.8, and a similarity with document G. The degree is 0.5. Document C has a similarity with document C of 1.0, a similarity with document E of 0.9, a similarity with document A of 0.8, and a similarity with document D. The degree is 0.7. Further, the document D indicates that the similarity with the document D is 1.0, the similarity with the document B is 0.8, and the similarity with the document C is 0.7. Document I indicates that the similarity with Document I is 1.0. Further, the document J indicates that the similarity with the document J is 1.0, the similarity with the document X is 0.9, and the similarity with the document K is 0.5. Further, the document X indicates that the similarity with the document X is 1.0, the similarity with the document J is 0.9, and the similarity with the document Y is 0.8. Further, the document Y indicates that the similarity with the document Y is 1.0, the similarity with the document Z is 0.8, and the similarity with the document X is 0.8. Further, the document Z indicates that the similarity with the document Z is 1.0, the similarity with the document X is 0.9, and the similarity with the document Y is 0.8.

  The excluded document data 33 is data in which a document in which there is no document similar to the first threshold value other than its own document is stored as an excluded document. As an example, the excluded document data 33 is stored with an excluded document specified by an excluded document specifying unit 43 described later. As another example, the excluded document data 33 is referred to by a dividing unit 46 described later.

  The inter-set similarity data 34 is data in which, for each document stored in the document data 30, the similarity between similar document sets of the respective documents is stored. As an example, the inter-set similarity data 34 is calculated and stored by an inter-set similarity calculating unit 44 described later. As another example, the inter-set similarity data 34 is referred to by a classification unit 45 described later.

  FIG. 3 is a diagram illustrating a configuration example of the similarity data between sets. In the example of FIG. 2, the similarity between similar document sets is stored for each document whose document names are “A” to “Z”. The similarity between sets indicates that the larger the value, the more similar the documents that are included in the set. In this embodiment, the similarity between sets is normalized to a range from 0.0 to 1.0. In the example of FIG. 3, for each document whose document names are “A” to “Z”, the document name of the other party and the similarity between sets are ranked in the order of high similarity in the form of [document name, similarity]. They are stored side by side. The example in FIG. 3 indicates that the similarity between the set of document A and document B is 0.5. The document B indicates that the similarity between the set with the document A is 0.5 and the similarity between the set with the document C is 0.3. The document C indicates that the similarity between the set with the document D is 0.3. The document D indicates that the similarity between the set with the document B is 0.4 and the similarity with the document C is 0.3. The document J indicates that the similarity between the set with the document K is 0.4. The document X indicates that the similarity between the set with the document Y is 0.5. The document Y indicates that the similarity between the set with the document Z is 0.6 and the similarity with the document X is 0.5. The document Z indicates that the similarity between the set with the document Y is 0.6.

  The cluster data 35 is data in which a plurality of documents shown in the document data 30 are classified into groups and the documents of each group are stored as clusters. As an example, the cluster data 35 is generated and stored by a classification unit 45 described later. As another example, the cluster data 35 is referred to by a dividing unit 46 described later.

  The distributed data 36 is data in which a document obtained by dividing a plurality of documents indicated by the document data 30 into groups indicated by the cluster data 35 is stored. As an example, the distributed data 36 is generated and stored by a dividing unit 46 described later. As another example, the distributed data 36 is referred to by an output unit 47 and a clustering unit 48 described later.

  Returning to FIG. 1, the control unit 22 includes an internal memory for storing programs and control data that define various processing procedures, and executes various processes using these. Various types of integrated circuits and electronic circuits can be employed for the control unit 22. For example, an ASIC (Application Specific Integrated Circuit) is an example of the integrated circuit. Examples of the electronic circuit include a central processing unit (CPU) and a micro processing unit (MPU). As shown in FIG. 1, the control unit 22 includes a registration unit 40, a generation unit 41, an inter-document similarity calculation unit 42, an excluded document specification unit 43, an inter-set similarity calculation unit 44, and a classification unit 45. A dividing unit 46, an output unit 47, and a clustering unit 48.

  Among these, the registration unit 40 is a processing unit that registers document data in the storage unit 21. As one aspect, the registration unit 40 registers document data in the storage unit 21 when document data indicating a plurality of documents to be clustered is input to the external I / F unit 20 from a data transmission path or a storage medium. To do. Thereby, the document data 30 is stored in the storage unit 21.

  The generation unit 41 is a processing unit that generates feature information indicating features of a plurality of documents to be clustered stored in the document data 30. As one aspect, the generation unit 41 generates index data 31 indicating which document includes each word included in each document indicated in the document data 30 as feature information indicating the document characteristics. For example, the generation unit 41 extracts a word from each document shown in the document data 30 and stores each word in association with the document name. Then, the generation unit 41 arranges the words and document names extracted from the documents in the order of the character codes of the words. As a result of arranging in the character code order, words and document names are arranged in order for each word. The generation unit 41 refers to the words and document names arranged in the order of the character codes in order, and generates the index data 31 indicating which document each word is included in. The generation unit 41 stores the generated index data 31 in the storage unit 21.

  The inter-document similarity calculation unit 42 is a processing unit that calculates the similarity between a plurality of documents to be clustered stored in the document data 30. As one aspect, the inter-document similarity calculation unit 42 calculates the similarity between a plurality of documents stored in the document data 30 based on the index data 31 generated by the generation unit 41. For example, the inter-document similarity calculation unit 42 specifies words included in each document indicated in the document data 30 based on the index data 31. Then, the inter-document similarity calculation unit 42 regards each document as a word vector and calculates the angle between the vectors (cosine θ) as the similarity between the documents.

For example, it is assumed that document A includes word a, word b, word c, and word d, and document B includes word b, word c, word d, and word e. In this case, the inter-document similarity calculation unit 42 uses the words included in the document A and the document B as elements. If each word includes a word, the value of the element corresponding to the word is set to 1. If the word does not include each word, the word is calculated. A vector whose corresponding element value is 0 is obtained. For example, when the vector element corresponds to (word a, word b, word c, word d, word e), the vector A of the document A is obtained as (1, 1, 1, 1, 0). . Further, the vector B of the document B is obtained as (0, 1, 1, 1, 1). The inter-document similarity calculation unit 42 performs an operation for obtaining an angle (cosine θ) shown in the following equation (1) for the vectors A and B to obtain the similarity between the documents A and B.
Similarity = A · B / (| A | × | B |) (1)

Here, A · B is calculated as 1 × 0 + 1 × 1 + 1 × 1 + 1 × 1 + 0 × 1 = 3. Further, | A | is calculated as (1 × 1 + 1 × 1 + 1 × 1 + 1 × 1 + 0 × 0) ^1/2 = 2. Similarly, | B | is calculated as (0 × 0 + 1 × 1 + 1 × 1 + 1 × 1 + 1 × 1) ^1/2 = 2. Therefore, the similarity between document A and document B is calculated as 3 / (2 × 2) = 0.75.

  The inter-document similarity calculation unit 42 obtains a similarity for each document shown in the document data 30 with each document including the self-document shown in the document data 30 as a partner document. Then, the inter-document similarity calculation unit 42 stores the document name and similarity of the partner document whose calculated similarity is equal to or greater than a predetermined first threshold in the storage unit 21 as inter-document similarity data 32. For example, the first threshold is set to 0.4, and the inter-document similarity calculation unit 42 stores the document name and similarity of the partner document having the similarity of 0.4 or more in the storage unit 21 as inter-document similarity data 32. It shall be. The first threshold is not limited to this example, and a person who uses the document dividing device 10 may set an arbitrary value. As a result, the storage unit 21 stores inter-document similarity data 32 as shown in FIG. Note that the similarity calculation method is not limited to the method of calculating the above-mentioned angle (cosine θ) by regarding a document as a vector of words as long as the degree of similarity between documents can be calculated. It may be used. For example, the ratio of the same word contained in the own document and the partner document may be calculated as the similarity.

  The excluded document specifying unit 43 is a processing unit that specifies a document that is not similar to any document. As an aspect, the excluded document specifying unit 43 specifies a document having no document having a similarity equal to or higher than the first threshold other than the self document based on the inter-document similarity data 32. For example, the excluded document specifying unit 43 specifies a document in which similar documents other than the own document are not registered in the inter-document similarity data 32. In the example of FIG. 2, there is no document whose document I has a similarity of 0.4 or more other than its own document I. For this reason, the excluded document specifying unit 43 specifies the document I as an excluded document. The excluded document specifying unit 43 stores the specified document as excluded document data 33 in the storage unit 21.

  The inter-set similarity calculation unit 44 is a processing unit that calculates the similarity between sets of similar documents for each document stored in the document data 30. As one aspect, the inter-set similarity calculation unit 44 calculates the similarity between sets of documents that are within a predetermined predetermined upper rank. For example, the inter-set similarity calculation unit 44 calculates the similarity between sets of documents having the highest similarity between the documents of each document, excluding documents excluded by the excluded document specifying unit 43. Note that the inter-set similarity calculation unit 44 calculates the similarity between sets of all documents having a similarity equal to or higher than the first threshold when there are not a predetermined number of counterpart documents whose similarity is equal to or higher than the first threshold. calculate. That is, when there are only three documents having a similarity equal to or greater than the first threshold value, the similarity is calculated with the three documents as a set. For example, it is assumed that the similarity between a set of documents in the top four ranks is calculated as a document in the top predetermined rank. The order in which the similarity is calculated is not limited to this example, and a person using the document dividing device 10 may set an arbitrary value.

The inter-set similarity calculation unit 44 is based on the inter-document similarity data 32, and each vector shown in the document data 30 has a similar document as an element and a similarity with the similar document as an element value. The similarity between sets of documents is calculated from the similarity between vectors. For example, the document A has the top four documents and the similarity is [A, 1.0], [B, 0.8], [C, 0.8], [E, 0.7] It is assumed that B is a document in the top four and the similarity is [B, 1.0], [A, 0.8], [D, 0.8], [G, 0.5]. In this case, the inter-set similarity calculation unit 44 obtains a vector having the documents similar to the documents A and B as elements and the similarity as the element value. For example, when the vector elements correspond to (document A, document B, document C, document D, document E, document G), the vector A of document A is (1.0, 0.8,. 8, 0.0, 0.7, 0.0). Further, the vector B of the document B is obtained as (0.8, 1.0, 0.0, 0.8, 0.0, 0.5). The inter-set similarity calculation unit 44 obtains an angle cosine θ shown in the following equation (2) for the vector A and the vector B, and uses this as the similarity between the document A and the document B.
Similarity = A · B / (| A | × | B |) (2)

Here, A · B is 1.0 × 0.8 + 0.8 × 1.0 + 0.8 × 0.0 + 0.0 × 0.8 + 0.7 × 0.0 + 0.0 × 0.5 = 1.6 Calculated. Also, | A | is calculated as (1.0 × 1.0 + 0.8 × 0.8 + 0.8 × 0.8 + 0.7 × 0.7) ^1/2 = (2.77) ^1/2 . Similarly, | B | is calculated as (0.8 × 0.8 + 1.0 × 1.0 + 0.8 × 0.8 + 0.5 × 0.5) ^1/2 = (2.53) ^1/2 The Accordingly, the similarity between the document A and the document B is calculated as 1.6 / ((2.77) ^1/2 × (2.53) ^1/2 ) ≈0.6.

  Further, for example, document A and document B have only three counterpart documents, and document A has counterpart documents and similarities of [A, 1.0], [B, 0.5], [C, 0. 3], and document B is assumed to be [B, 1.0], [A, 0.8], [C, 0.4] with the counterpart document. Also in this case, the inter-set similarity calculation unit 44 obtains a vector having the documents similar to the documents A and B as elements and the similarity as the element value. For example, when the vector element corresponds to (document A, document B, document C), the vector A of document A is obtained as (1.0, 0.5, 0.3). Further, the vector B of the document B is obtained as (0.8, 1.0, 0.4). The inter-set similarity calculation unit 44 calculates the similarity between the document A and the document B by performing the calculation shown in the above equation (2) for the vector A and the vector B.

Here, A · B is calculated as 1.0 × 0.8 + 0.5 × 1.0 + 0.3 × 0.4 = 1.42. Also, | A | is calculated as (1.0 × 1.0 + 0.5 × 0.5 + 0.3 × 0.3) ^1/2 = (1.39) ^1/2 . Similarly, | B | is calculated as (0.8 × 0.8 + 1.0 × 1.0 + 0.4 × 0.4) ^1/2 = (1.8) ^1/2 . Accordingly, the similarity between the document A and the document B is calculated as 1.42 / ((1.39) ^1/2 × (1.8) ^1/2 ) ≈0.92.

  Here, the degree of similarity between a set of documents having a similarity between documents of upper and lower predetermined levels will be described. FIG. 4 is an explanatory diagram for explaining the similarity between sets of documents. In the example of FIG. 4, an example of the similarity between documents of each document is shown. In the example of FIG. 4, the documents A to G and N to Q are schematically illustrated by arranging the distance between documents shorter as the similarity between the documents is higher. In the example of FIG. 4, documents A to G having high similarity between documents are indicated by “●”, and documents N to Q having low similarity between documents are indicated by “◯”. In the example of FIG. 4, for example, when clustering is performed, it is expected that the documents A to G in the range 50 in which the similarity between documents is high are gathered into clusters. Since the documents A to G are similar to each other, when the similarity between sets of similar documents is calculated, the similarity is calculated to be high. Therefore, by grouping documents having high similarity between sets of similar documents into the same group, it is possible to prevent similar documents from being divided into different distributed data 36. Also, by calculating the similarity between sets of similar documents for each document, similar documents can be easily classified into the same group.

  The inter-set similarity calculation unit 44 stores, for each document, the document name of the counterpart document and the similarity between sets in the storage unit 21 as inter-set similarity data 34. Thereby, the storage unit 21 stores the inter-set similarity data 34 as shown in FIG. The inter-set similarity data 34 shown in FIG. 3 is an example, and is not obtained from the inter-document similarity data 32 shown in FIG.

  The classification unit 45 is a processing unit that classifies each document stored in the document data 30 into a group of documents having high similarity between sets. As one aspect, the classification unit 45 classifies each document into a group of documents whose similarity between sets is equal to or more than a predetermined second threshold based on the set similarity data 34. FIG. 5 is a diagram illustrating an example of a flow of processing for classifying documents. For example, the classification unit 45 arranges the documents stored in the inter-set similarity data 34 in descending order of similarity between sets of documents, and sequentially specifies each document as a processing target in descending order of similarity. The classification unit 45 classifies the document to be processed into the same group as the document that has already been processed when the similarity between the document that has already been processed and the set is greater than or equal to the second threshold. Further, the classification unit 45 classifies the document as a new group when the similarity between the document already processed and the set is less than the second threshold. For example, when the second threshold value is 0.4 and the similarity between sets is 0.4 or more, the classification unit 45 classifies the document into the same group as the document that has already been processed, and the similarity between sets is If it is less than 0.4, it is classified as a new group. As a result, each document stored in the document data 30 is classified into a group. Thereby, in the example of FIG. 5, the documents Y, Z, and X are classified into the same group, the documents A, B, C, and D are classified into the same group, and the documents J and K are classified into the same group. The classification unit 45 stores the classified documents of each group in the storage unit 21 as cluster data 35. Note that the second threshold is not limited to this example, and a person using the document dividing device 10 may set an arbitrary value.

  The dividing unit 46 is a processing unit that divides each document stored in the document data 30 into distributed data for distributed processing. As one aspect, the dividing unit 46 divides a plurality of documents in units of groups stored in the cluster data 35. Further, the dividing unit 46 divides the excluded document stored in the excluded document data 33 into distributed data. FIG. 6 is a diagram illustrating an example of a flow of processing for dividing a document. For example, the dividing unit 46 arranges the groups classified by the classifying unit 45 in the descending order of the number of documents, and sequentially specifies each group as a processing target in the descending order of the number of documents. When the dividing unit 46 divides the document into a plurality of distributed data 36, the divided unit 46 divides the document in units of groups into the distributed data 36 having the smallest number of documents among the distributed data 36. Thus, in the example of FIG. 6, the group of documents A, B, C, and D and the group of documents J and K are divided into distributed data 36A, and the documents Y, Z, and X are divided into distributed data 36B. The dividing unit 46 also divides the excluded document stored in the excluded document data 33 into distributed data having the smallest number of documents. When there are a plurality of distributed data 36 having the smallest number of documents, the dividing unit 46 divides the document into any of the distributed data having the smallest number of documents. Thereby, a plurality of documents are divided into the distributed data so that the number of documents is equal. The dividing unit 46 stores each divided distribution data 36 in the storage unit 21.

  The output unit 47 is a processing unit that outputs the distributed data 36 stored in the storage unit 21 to a processing unit that executes clustering. As one aspect, the output unit 47 outputs the distributed data 36 as data to be clustered to a plurality of computers and processors that execute clustering. Further, the output unit 47 outputs the similarity between documents related to the documents included in the distributed data 36 stored in the inter-document similarity data 32 together with the distributed data 36 to a plurality of computers and processors that execute clustering. Output. In this embodiment, clustering is executed in the document dividing apparatus 10 with any one distributed data 36 as a processing target. The output unit 47 outputs the distributed data 36 other than the distributed data 36 that is the clustering processing target to another computer or processor. The output distributed data 36 is subjected to clustering in another computer or processor. The output unit 47 outputs the similarity between documents related to the document included in the distributed data 36 stored in the inter-document similarity data 32 together with the distributed data 36 to another computer or processor. In other computers and processors, the similarity calculation processing can be reduced by performing clustering using the similarity between documents concerning the document included in the distributed data 36.

  The clustering unit 48 is a processing unit that performs clustering of the distributed data 36 that is a clustering target. As one aspect, the clustering unit 48 calculates a similarity between documents included in the distributed data 36 to be processed, and executes a clustering process for collecting similar documents. At this time, when calculating the similarity between documents, the clustering unit 48 stores the similarity between documents related to the document included in the distributed data 36 to be processed in the inter-document similarity data 32. Use the similarity degree. That is, when the similarity between documents is stored in the inter-document similarity data 32, the clustering unit 48 does not calculate the similarity and uses the stored similarity as the similarity between documents. As described above, for the documents whose similarity is stored in the inter-document similarity data 32, the stored similarity is used, so that the similarity calculation process is omitted, and the processing load can be reduced. it can.

FIG. 7 is an explanatory diagram for explaining clustering. In the example of FIG. 7, an example in which the documents A to F are clustered is shown in the upper part. In the example of FIG. 7, the documents A to F are divided into documents A to C and documents D to F having high similarity between sets of similar documents in the lower part, and the documents A to C and the documents D to F are divided. An example of clustering by different processors 1 and 2 is shown. In clustering, for example, a plurality of documents are compared with each other in a brute force manner to obtain similar documents. Therefore, as shown in the upper part of FIG. 7, when documents A to F are clustered, the similarity between the documents A to F is calculated and the similarity is stored. Along with the increase, the storage capacity used for processing increases and the processing time also increases. For example, as the number of documents n increases, the calculation amount and storage capacity of similarity increase by n × n. As an example, when the number of documents n is 1 million and the size of the storage area for storing each similarity is 4 bytes, the storage area for storing the similarity is (10 ⁷ × 2-1) ² × 4 bytes. ≈14.5 TB. Therefore, when the number of documents n is large, there is a limit to clustering documents with one processor.

  On the other hand, as shown in the lower part of FIG. 7, when the documents A to C and the documents D to F are clustered by different processors, the number of documents to be clustered decreases due to the division, so that the storage capacity can be suppressed. Processing time can also be suppressed. Documents A to C and documents D to F have low similarity between sets of similar documents, and the similarity between documents is also low. For this reason, the processing unit for clustering the documents A to C and the documents D to F can perform clustering of similar documents without exchanging the similarities calculated with each other.

  The clustering unit 48 communicates with other computers and processors that have performed clustering, merges the results of the respective clustering processes, and summarizes the results as clustering for the document data 30.

  FIG. 8 is a diagram schematically showing the flow of processing for document data. The generation unit 41 generates index data 31 as feature information indicating the features of each of the plurality of documents in the document data 30. The inter-document similarity calculation unit 42 calculates the similarity between a plurality of documents based on the generated index data 31, and stores the calculated similarity between documents as inter-document similarity data 32. The excluded document specifying unit 43 specifies a document in which similar documents other than the own document are not registered in the inter-document similarity data 32, and stores the specified document as excluded document data 33. The inter-set similarity calculation unit 44 calculates the similarity between sets of documents that are within a predetermined upper level of similarity between documents of each document, and uses the calculated similarity between sets as inter-set similarity data 34. Store. The classification unit 45 classifies documents with high similarity between sets into groups. The dividing unit 46 divides a plurality of documents into distributed data 36 in units of groups stored in the cluster data 35. The dividing unit 46 divides the excluded document stored in the excluded document data 33 into distributed data 36. By dividing into the distributed data 36 in this way, it is possible to suppress a similar document from being divided into different distributed data 36, so that parallel processing of clustering for each distributed data 36 can be performed efficiently. Further, since the number of documents to be clustered is reduced by the division, the storage capacity and the processing time are also suppressed, so that efficient clustering parallel processing can be performed on the documents.

[Concrete example]
Next, a specific example in which a document is divided by the document dividing apparatus 10 according to the first embodiment will be described. FIG. 9 is a diagram illustrating an example of a document name of a clustering target document included in the document data and a word included in the document. In the example of FIG. 9, eleven documents with document names “doc01” to “doc11” are shown as documents to be clustered, and words included in the documents are shown as alphabetic symbols next to the respective document names. ing. In the example of FIG. 9, it is shown that the document with the document name doc01 includes the words A to J.

  The generation unit 41 generates index data 31 indicating which document includes each word included in each document to be clustered. FIG. 10 is a diagram illustrating an example of index data. In the example of FIG. 10, the words included in the clustering target document are shown in a list, and the document name of the document including the word is shown next to each word. In the example of FIG. 10, it is indicated that the word A is included in each document with the document names doc01, doc02, doc03, doc04, and doc05.

  The inter-document similarity calculation unit 42 calculates the similarity between a plurality of documents stored in the document data 30 based on the index data 31, and the document name of the partner document whose calculated similarity is equal to or more than a first threshold value and The similarity is stored in the storage unit 21 as inter-document similarity data 32. The first threshold value is 0.2. FIG. 11 is a diagram showing an example of inter-document similarity data. In the example of FIG. 11, the document names of the documents to be clustered are shown, and the similarity between the document name of the partner whose similarity is equal to or more than the first threshold and the similarity is document name: similar to each document name. They are listed in the order of degree of similarity in degree format. In the example of FIG. 11, the document with the document name doc01 has a similarity with the document with the document name doc01 of 1.0, the similarity with the document with the document name doc02 is 0.8, and the document with the document name doc03. It shows that the similarity to is 0.7. In the example of FIG. 11, the document with the document name doc01 has a similarity of 0.4 with the document with the document name doc04, and the similarity with the document with the document name doc05 is 0.3.

  The excluded document specifying unit 43 specifies a document in which similar documents other than the own document are not registered in the inter-document similarity data 32, and stores the specified document in the storage unit 21 as excluded document data 33. FIG. 12 is a diagram illustrating an example of excluded document data. In the example of FIG. 12, a document with the document name doc11 in which no similar document other than the own document is registered from the inter-document similarity data 32 shown in FIG. 11 is stored as an excluded document.

  Based on the inter-document similarity data 32, the inter-set similarity calculation unit 44 calculates the similarity between the sets of documents having the top four similarities between the documents. Are stored in the storage unit 21 as inter-set similarity data 34. FIG. 13 is a diagram illustrating an example of inter-set similarity data. In the example of FIG. 13, the document names of the documents to be clustered are shown. The similarity between the other document name and a set of similar documents is displayed next to each document name. They are listed in order of decreasing similarity. In the example of FIG. 13, the document with the document name doc01 has a similarity with the document with the document name doc01 of 1.00, the similarity with the document with the document name doc02 is 0.98, and the document with the document name doc03. It is shown that the similarity to is 0.96. In the example of FIG. 13, the document with the document name doc01 has a similarity of 0.76 with the document with the document name doc04, and the similarity with the document with the document name doc05 is 0.44.

  The classification unit 45 classifies each document into a group based on the inter-set similarity data 34, and stores the classified documents in each group as cluster data 35. 21. The second threshold value is 0.7. FIG. 14 is a diagram illustrating an example of cluster data. In the example of FIG. 14, documents with document names doc08 to doc10 are set as clusters as group 1, documents with document names doc06 and doc07 are set as clusters as group 2, and documents with document names doc01 to doc05 are set as clusters as group 3. Indicates that

  The dividing unit 46 arranges the groups classified by the classifying unit 45 in descending order of the number of documents, sequentially groups each group in descending order of the number of documents, and divides the document into groups of distributed data 36 having the smallest number of documents. The dividing unit 46 also divides the excluded document stored in the excluded document data 33 into the distributed data 36 having the smallest number of documents. FIG. 15 is a diagram illustrating an example of distributed data. FIG. 15 shows an example in which a document is divided into two distributed data 36A and 36B. The example of FIG. 15 indicates that the document names doc01 to doc05 of the group 3 and the document names doc06 and doc07 of the group 2 are divided as the distributed data 36A. Further, the example of FIG. 15 indicates that the document names doc08 to doc10 of group 1 and the excluded document of document name doc11 are divided as distributed data 36B.

  As described above, by dividing the document data 30 into the distributed data 36A and 36B, it is possible to execute an efficient clustering parallel process for the document.

[Process flow]
Next, the flow of a dividing process for dividing a plurality of documents to be clustered by the document dividing apparatus 10 according to the present embodiment will be described. FIG. 16 is a flowchart illustrating the procedure of the dividing process. This division processing is executed at a timing when a predetermined operation for instructing division start is performed after the document data 30 is stored in the storage unit 21.

  As illustrated in FIG. 16, the generation unit 41 generates index data 31 that indicates which document includes each word included in each document indicated in the document data 30 as feature information indicating the document characteristics. Generation processing is executed (step S10). The inter-document similarity calculation unit 42 executes inter-document similarity calculation processing for calculating the similarity between a plurality of documents stored in the document data 30 based on the generated index data 31 (step S11). The excluded document specifying unit 43 executes an excluded document specifying process for specifying an excluded document that is not similar to any document (step S12). The inter-document similarity calculation unit 42 sets a combination of the comparison source document, the partner document, and the similarity calculated by the inter-document similarity calculation processing for each comparison source document in the form of [document name, similarity]. Are stored in the storage unit 21 as the inter-document similarity data 32 (step S13). The inter-set similarity calculation unit 44 executes an inter-set similarity calculation process for calculating the similarity between sets of documents whose documents are within a predetermined upper rank (step S14). The classification unit 45 executes a classification process for classifying each document stored in the document data 30 into a group of documents having high similarity between sets (step S15). The dividing unit 46 executes a dividing process for dividing each document into distributed data for distributed processing (step S16), and ends the process.

  Next, the flow of index generation processing according to the present embodiment will be described. FIG. 17 is a flowchart showing the procedure of index generation processing. This index generation process is called and executed from step S10 of the division process shown in FIG.

  As shown in FIG. 17, the generation unit 41 initializes a storage area for storing words and document names (step S20). The generation unit 41 sets a pointer A indicating that it is a processing target in the first document of the plurality of documents indicated in the document data 30 (step S21). The generation unit 41 extracts all words from the document indicated by the pointer A (step S22). The generation unit 41 deletes only one overlapping word from the extracted words, and makes each word unique (step S23). The generation unit 41 registers each extracted word in association with the document name indicated by the pointer A (step S24). The generation unit 41 advances the pointer A to the next document (step S25). The generation unit 41 determines whether there is a document indicated by the pointer A (step S26). If there is a document indicated by the pointer A (Yes at step S26), the process returns to the above-described step S22 to execute processing.

  On the other hand, when there is no document indicated by the pointer A (No at Step S26), the generation unit 41 sorts the words and document names extracted from each document in the order of the character codes of the words (Step S27). The generation unit 41 sequentially converts words and document names arranged in the order of character codes, generates index data 31 indicating which documents each word is included in (step S28), and returns to the calling source.

  Next, the flow of the inter-document similarity calculation process according to the present embodiment will be described. FIG. 18 is a flowchart showing the procedure of the inter-document similarity calculation process. This inter-document similarity calculation process is called and executed from step S11 of the division process shown in FIG.

  As illustrated in FIG. 18, the inter-document similarity calculation unit 42 sets a first threshold value of similarity (step S30). The inter-document similarity calculation unit 42 sets a pointer A indicating that the document is to be processed in the first document of the plurality of documents indicated in the document data 30 (step S31). The inter-document similarity calculation unit 42 extracts all words from the document indicated by the pointer A (step S32). The inter-document similarity calculation unit 42 calculates the similarity between a plurality of documents including the self-document stored in the document data 30 based on the index data 31 (step S33). The inter-document similarity calculation unit 42 stores the document name and similarity of the partner document whose calculated similarity is equal to or greater than a predetermined first threshold in the storage unit 21 as inter-document similarity data 32 (step S34). The inter-document similarity calculation unit 42 advances the pointer A to the next document (step S35). The inter-document similarity calculation unit 42 determines whether there is a document indicated by the pointer A (step S36). If there is a document indicated by the pointer A (Yes at step S36), the process returns to the above-described step S32 to execute processing. On the other hand, if there is no document indicated by the pointer A (No at step S36), the process returns to the calling source.

  Next, the flow of the excluded document specifying process according to the present embodiment will be described. FIG. 19 is a flowchart showing a procedure of excluded document specifying processing. This excluded document specifying process is called and executed from step S12 of the dividing process shown in FIG.

  As illustrated in FIG. 19, the excluded document specifying unit 43 sets a pointer A indicating that the document is to be processed in the first document of the plurality of documents indicated in the document data 30 (step S40). The excluded document specifying unit 43 initializes the excluded document data 33 (step S41). The excluded document specifying unit 43 determines whether the document indicated by the pointer A is a document in which there is no document whose similarity is equal to or more than the first threshold other than the document itself (step S42). When there is no document whose similarity is equal to or greater than the first threshold other than the own document (Yes at Step S42), the excluded document specifying unit 43 registers the document indicated by the pointer A in the excluded document data 33 (Step S43). On the other hand, if there is a document whose similarity is equal to or greater than the first threshold other than the own document (No at Step S42), the process proceeds to Step S44 described later.

  The excluded document specifying unit 43 advances the pointer A to the next document (step S44). The excluded document specifying unit 43 determines whether there is a document indicated by the pointer A (step S45). If there is a document indicated by the pointer A (Yes at step S45), the process returns to the above-described step S42 to execute processing. On the other hand, if there is no document indicated by the pointer A (No at step S45), the process returns to the calling source.

  Next, the flow of inter-set similarity calculation processing according to the present embodiment will be described. FIG. 20 is a flowchart showing the similarity calculation processing between sets. This inter-set similarity calculation process is called and executed from step S14 of the division process shown in FIG.

  As illustrated in FIG. 20, the inter-set similarity calculation unit 44 sets a pointer A indicating that it is a processing target in the first document of the plurality of documents indicated in the document data 30 (step S50). The inter-set similarity calculation unit 44 sets the pointer B to the document next to the pointer A (step S51). The inter-set similarity calculation unit 44 calculates the similarity between the sets of documents having the top four rankings of similarity between the document indicated by the pointer A and the document indicated by the pointer B (step S52). The inter-set similarity calculation unit 44 stores the calculated similarity as a similarity between sets of documents similar to the document indicated by the pointer A and the document indicated by the pointer B (step S53).

  The inter-set similarity calculation unit 44 advances the pointer B to the next document (step S54). The inter-set similarity calculation unit 44 determines whether there is a document indicated by the pointer B (step S55). If there is a document indicated by the pointer B (Yes at Step S55), the process returns to the above-described Step S52 to execute processing. On the other hand, when there is no document indicated by the pointer B (No at Step S55), the inter-set similarity calculation unit 44 advances the pointer A to the next document (Step S56). The inter-set similarity calculation unit 44 determines whether there is a document indicated by the pointer A (step S57). If there is a document indicated by the pointer A (Yes at step S57), the process returns to the above-described step S51 to execute processing. On the other hand, if there is no document indicated by the pointer A (No at step S57), the process returns to the calling source.

  Next, the flow of classification processing according to the present embodiment will be described. FIG. 21 is a flowchart showing the classification process. This classification process is called and executed from step S15 of the division process shown in FIG.

  As shown in FIG. 21, the classification unit 45 arranges the documents stored in the inter-set similarity data 34 in descending order of similarity between sets of documents (step S60). The classification unit 45 sets the pointer A indicating that the document is a processing target in the first document stored in the inter-set similarity data 34 in descending order of similarity between the sets of documents (step S61). The classification unit 45 classifies the document indicated by the pointer A into a group (step S62). The classification unit 45 advances the pointer A to the next document (step S63). The classification unit 45 determines whether there is a document indicated by the pointer A (step S64). If there is a document indicated by the pointer A (Yes at Step S64), the process proceeds to Step S65 described later. On the other hand, when there is no document indicated by the pointer A (No at Step S64), the process proceeds to Step S70 described later.

  The classification unit 45 determines whether a document whose similarity between the document indicated by the pointer A and the set is equal to or greater than the second threshold exists in the documents already divided into groups (step S65). If there is a document whose similarity between sets is equal to or greater than the second threshold (Yes in step S65), the classification unit 45 adds the document indicated by the pointer A to the group where documents whose similarity is equal to or greater than the second threshold exists. (Step S66), the process proceeds to Step S63. On the other hand, when there is no document whose similarity between sets is equal to or greater than the second threshold (No in step S65), the classification unit 45 includes a document indicated by the pointer A whose similarity between sets is equal to or greater than the second threshold. Whether or not (step S67). When there is a similarity between the sets equal to or greater than the second threshold (Yes at Step S67), the classification unit 45 classifies the document indicated by the pointer A into a new group (Step S68), and proceeds to Step S63. On the other hand, when there is no similarity between sets equal to or greater than the second threshold (No at Step S67), the classification unit 45 stores the document indicated by the pointer A as other (Step S69), and proceeds to Step S63.

  On the other hand, the classification unit 45 sets a pointer B indicating that it is a processing target to the first document stored as others (step S70). The classification unit 45 adds the document indicated by the pointer B to the group including the document having the highest similarity between documents in the document indicated by the pointer B (step S71). The classification unit 45 advances the pointer B to the next document (step S72). The classification unit 45 determines whether there is a document indicated by the pointer B (step S73). If there is a document indicated by the pointer B (Yes at step S73), the process returns to the above-described step S71 to execute processing. On the other hand, when there is no document indicated by the pointer B (No at Step S73), the process returns to the calling source.

  Next, the flow of division processing according to the present embodiment will be described. FIG. 22 is a flowchart showing the dividing process. This division process is called and executed from step S16 of the division process shown in FIG.

  As shown in FIG. 22, when dividing the document into n pieces, the dividing unit 46 prepares n areas for storing the divided documents (step S80). The dividing unit 46 arranges the groups classified by the classification process in descending order of the number of documents (step S81). The dividing unit 46 sets the pointer A indicating the processing target in the first group in the descending order of the number of documents (step S82). The dividing unit 46 compares the number of documents stored in the n areas, and stores the group of documents indicated by the pointer A in the area having the smallest number of documents (step S83). The dividing unit 46 advances the pointer A to the next group (step S84). The dividing unit 46 determines whether there is a group indicated by the pointer A (step S85). If there is a group indicated by the pointer A (Yes at step S85), the process returns to step S83 to execute the process.

  On the other hand, when there is no group indicated by the pointer A (No at Step S85), the dividing unit 46 sets the pointer B indicating that it is a processing target in the excluded document stored in the excluded document data 33 (Step S86). The dividing unit 46 compares the number of documents stored in the n areas, and stores the document indicated by the pointer B in the area having the smallest number of documents (step S87). The dividing unit 46 advances the pointer B to the next excluded document (step S88). The dividing unit 46 determines whether there is a document indicated by the pointer B (step S89). If there is a document indicated by the pointer B (Yes at Step S89), the process returns to Step S87 to execute processing. On the other hand, when there is no document indicated by the pointer B (No at Step S89), the dividing unit 46 stores the documents stored in the n areas as the distributed data 36 in the storage unit 21 (Step S90), and the calling source Return to.

  The output unit 47 outputs the distributed data 36 divided by the above-described division processing as data to be clustered to a plurality of computers and processors that execute clustering.

  Next, the flow of clustering processing by the document dividing apparatus 10 according to the present embodiment will be described. FIG. 23 is a flowchart illustrating the procedure of the clustering process. This clustering process may be executed after the above-described division process. Further, it may be executed at a timing when a predetermined operation for instructing clustering is performed.

  As shown in FIG. 23, the clustering unit 48 calculates the similarity between documents included in the distributed data 36 to be processed (step S100). The clustering unit 48 performs clustering for collecting similar documents based on the calculated similarity between documents (step S101). The clustering unit 48 merges the result of clustering with another processor that has performed clustering (step S102), and ends the process.

[Effect of Example 1]
The document dividing apparatus 10 according to the present embodiment generates index data 31 as feature information indicating features of a plurality of documents in the document data 30. The document dividing apparatus 10 according to the present embodiment calculates the similarity between a plurality of documents based on the generated index data 31. The document dividing apparatus 10 according to the present embodiment calculates the similarity between a set of documents that are within a predetermined upper level in the similarity between documents of each document. Then, the document dividing apparatus 10 according to this embodiment classifies documents having high similarity between sets into groups. The document dividing apparatus 10 according to the present embodiment divides a plurality of documents into distributed data 36 for distributed processing in classified group units. Thereby, according to the document dividing apparatus 10 according to the present embodiment, it is possible to efficiently perform parallel processing of clustering on the document.

  In addition, the document dividing apparatus 10 according to the present embodiment regards each document as a vector in which each of the documents having a similarity degree higher than a predetermined level is an element and the similarity is an element value. The similarity between sets of documents is calculated from the similarity between the documents. Thereby, according to the document dividing apparatus 10 according to the present embodiment, documents having high similarity between sets of similar documents can be grouped into the same group, and similar documents are divided into different distributed data 36. This can be suppressed.

  Further, the document dividing apparatus 10 according to the present embodiment outputs the divided distributed data 36 and the similarity between documents related to the documents included in the distributed data 36 to the processing unit that executes clustering. Thereby, according to the document dividing apparatus 10 according to the present embodiment, it is possible to reduce the calculation processing of the similarity between documents.

  Further, the document dividing apparatus 10 according to the present embodiment divides a plurality of documents into the distributed data 36 so that the number of documents becomes equal. Thereby, according to the document dividing apparatus 10 according to the present embodiment, even when the clustering of each distributed data 36 is processed in parallel by a plurality of computers and processors, it is possible to suppress the load on a specific computer or processor from being biased.

  Although the embodiments related to the disclosed apparatus have been described above, the present invention may be implemented in various different forms other than the above-described embodiments. Therefore, another embodiment included in the present invention will be described below.

  For example, in the above embodiment, a case has been described in which a plurality of documents are divided into the distributed data 36 so that the number of documents is equal, but the disclosed apparatus is not limited to this. For example, the documents of each group may be stored in the distributed data 36 in order from the largest number of documents. Further, the excluded document stored in the excluded document data 33 may be stored in the specific distributed data 36.

  In the above-described embodiment, the case where the clustering unit 48 performs clustering of any of the distributed data 36 has been described, but the disclosed apparatus is not limited to this. For example, all the distributed data 36 may be clustered by an external computer or processor.

  In the above-described embodiments, the case where words are extracted from each document and the similarity between the documents is calculated has been described. However, the disclosed apparatus is not limited thereto. For example, when a document includes images, the similarity between documents may be calculated in addition to the similarity between images.

  In the above-described embodiment, a case has been described in which a document having no similar document other than its own document is stored in the excluded document data 33 as an excluded document. However, the disclosed apparatus is not limited to this. For example, documents having no similar document other than the own document may be individually grouped. As a result, by dividing the document into the distributed data 36 in units of groups, the excluded document can also be divided into the distributed data 36.

  In the above-described embodiment, a case has been described in which a plurality of documents included in the document data 30 are divided into the distributed data 36 so that the number of documents is equal. However, the disclosed apparatus is not limited to this. For example, information indicating the processing capability of each processing unit such as a computer or processor that performs clustering is acquired, and a plurality of documents included in the document data 30 are divided into distributed data 36 at a ratio according to the processing capability of each processing unit. May be. Examples of the information indicating the processing capability include the number of CPU clocks, the amount of memory, the numerical value indicating the performance measured by software for measuring the processing performance, and the numerical value determined by the administrator according to the processing capability. As a result, the distributed data 36 having the number of documents corresponding to the processing capability is assigned to each processing unit that executes clustering, and therefore it is possible to suppress an excessive increase in the load on a specific processing unit.

[Distribution and integration]
In addition, each component of each illustrated apparatus does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. For example, the registration unit 40, the generation unit 41, the inter-document similarity calculation unit 42, the excluded document specification unit 43, the inter-set similarity calculation unit 44, the classification unit 45, the division unit 46, the output unit 47, and the clustering of the document dividing device 10 Each processing unit of the unit 48 may be appropriately integrated. Further, the processing of each processing unit may be appropriately separated into a plurality of processing units. In addition, the functions of the document dividing device 10 may be realized by having each processing unit in a separate device and connected through a network to cooperate.

[Split program]
The various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation. Therefore, in the following, an example of a computer that executes a division program having the same function as that of the above embodiment will be described with reference to FIG.

  FIG. 24 is a diagram for describing an example of a computer that executes a division program. As illustrated in FIG. 24, the computer 200 includes an operation unit 210, a display 220, and a communication unit 230. The computer 200 further includes a CPU 250, a ROM 260, an HDD 270, and a RAM 280. These units 210 to 280 are connected via a bus 240.

  The HDD 270 includes a registration unit 40, a generation unit 41, an inter-document similarity calculation unit 42, an excluded document specification unit 43, an inter-set similarity calculation unit 44, a classification unit 45, a division unit 46, an output unit 47, and a clustering unit 48. A division program 270a that exhibits the same function is stored in advance. About this division | segmentation program 270a, you may integrate or isolate | separate suitably like each component shown in Example 1. FIG. In other words, all the data stored in the HDD 270 need not always be stored in the HDD 270, and only the data necessary for processing may be stored in the HDD 270.

  Then, the CPU 250 reads the division program 270a from the HDD 270 and expands it in the RAM 280. Accordingly, as shown in FIG. 24, the division program 270a functions as a division process 280a. The division process 280a expands various data read from the HDD 270 in an area allocated to itself on the RAM 280 as appropriate, and executes various processes based on the expanded data. The division process 280a includes a registration unit 40, a generation unit 41, an inter-document similarity calculation unit 42, an excluded document specification unit 43, an inter-set similarity calculation unit 44, a classification unit 45, a division unit 46, an output unit 47, and a clustering unit 48. Including the processes shown in FIGS. 16 to 23. That is, the division process 280a includes a registration unit 40, a generation unit 41, an inter-document similarity calculation unit 42, an excluded document specification unit 43, an inter-set similarity calculation unit 44, a classification unit 45, a division unit 46, an output unit 47, clustering. The same operation as that of the unit 48 is executed. It should be noted that all the processing units virtually realized on the CPU 250 do not always have to operate on the CPU 250, and only the processing units necessary for processing need only be virtually realized.

  Note that the above divided program 270a is not necessarily stored in the HDD 270 or the ROM 260 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk inserted into the computer 200, so-called FD, CD-ROM, DVD disk, magneto-optical disk, or IC card. Then, the computer 200 may acquire and execute each program from these portable physical media. Each program is stored in another computer or server device connected to the computer 200 via a public line, the Internet, a LAN, a WAN, etc., and the computer 200 acquires and executes each program from these. It may be.

DESCRIPTION OF SYMBOLS 10 Document dividing device 21 Storage part 22 Control part 30 Document data 31 Index data 32 Inter-document similarity data 33 Excluded document data 34 Inter-set similarity data 35 Cluster data 36 Distributed data 41 Generation part 42 Inter-document similarity calculation part 43 Exclusion Document specifying unit 44 Inter-set similarity calculation unit 45 Classification unit 46 Dividing unit 47 Output unit 48 Clustering unit

Claims (6)

A generation unit that generates feature information indicating features of each of a plurality of documents to be clustered;
An inter-document similarity calculating unit that calculates a similarity between the plurality of documents based on the feature information generated by the generating unit;
An inter-set similarity calculating unit that calculates the similarity between sets of documents within a predetermined upper rank in the similarity between the documents of each document;
A classification unit for classifying documents having high similarity between the sets into groups;
A division unit that divides the plurality of documents in units of groups classified by the classification unit;
A document dividing apparatus characterized by comprising: The inter-set similarity calculation unit regards each document as a vector having a higher degree of similarity with the document within a predetermined level as an element and the similarity as an element value, and the similarity between vectors The document dividing apparatus according to claim 1, wherein similarity between sets of documents is calculated from the document.   And further comprising an output unit that outputs the distributed data obtained by dividing the plurality of documents by the dividing unit and the similarity between the documents relating to the documents included in the distributed data to a processing unit that executes clustering, respectively. The document dividing device according to claim 1 or 2. The document dividing device according to claim 1, wherein the dividing unit divides the plurality of documents into distributed data so that the number of documents is equal. On the computer,
Generate feature information indicating the features of each of multiple documents to be clustered,
Based on the generated feature information, the similarity between the plurality of documents is calculated,
Calculating a similarity between a set of documents having a similarity between the documents within a predetermined upper level of each document;
Classify documents with high similarity between the sets into groups,
A document segmentation program for executing each process for segmenting the plurality of documents in units of groups classified by the classification unit. Computer
Generate feature information indicating the features of each of multiple documents to be clustered,
Based on the generated feature information, the similarity between the plurality of documents is calculated,
Calculating a similarity between a set of documents having a similarity between the documents within a predetermined upper level of each document;
Classify documents with high similarity between the sets into groups,
A document dividing method, comprising: performing each process of dividing the plurality of documents in units of groups classified by the classification unit.

Images