JP7272846B2 - Document analysis device and document analysis method - Google Patents

Description

The present invention relates to document analysis technology.

When analyzing a large number of documents, it takes a lot of time and effort to read all the documents and understand their contents. _Therefore , as shown in FIG. 1, several documents (D _A , D _{B ,} . There is a need to comprehend the contents of a large number of document groups D1, D2, . This can be expected to solve the problem that it takes a lot of time and effort to read all of a large amount of documents.
On the other hand, in conventional techniques, for example, a document clustering technique as shown in FIG. 2 is used. Document clustering technology classifies a large number of documents into clusters, which are collections of documents with similar contents. In FIG. 2, a large number of document groups _D1 , _D2 , . classified (L2). By grasping the contents of the documents in each cluster (L3), the main contents (Sa1) of the large amount of original documents are grasped.

In the technique described in Patent Document 1 below, each sentence of a question text is labeled on a rule basis, and the discourse structure of the content of the question text is analyzed. Then, a group of question texts related to the user-designated keyword is extracted and grouped based on the discourse structure. As a result, representative sentences of questions in each group can be listed as FAQ (representative question) candidates.

Patent No. 5574842 ("FAQ candidate extraction system and FAQ candidate extraction program")

When documents are collected on a scale of thousands to tens of thousands, it takes an enormous amount of time to read all the documents in each cluster when grasping the contents of the documents in each cluster obtained as a result of clustering. I have a problem.
Also, when a document is arbitrarily extracted to grasp the content of each cluster, the content of the document is not necessarily the content that is common in the original mass of documents. In clustering, both documents with common content and those with non-common content are assigned to one of the clusters. Therefore, a cluster contains both documents with common contents and documents with uncommon contents. Therefore, a problem with the document clustering technique is that it is impossible to specify a document with common content (hereinafter referred to as a "representative document") that is originally intended to be extracted. The best representative document is a document that has a large number of similar documents in the entire original mass of documents.

In addition, extraction of representative documents requires manual work.
For example, even in the technique described in Patent Document 1, there is a problem that the FAQ extraction work needs to be performed manually.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a technology capable of performing the work of extracting a representative document with high efficiency and high accuracy.

According to one aspect of the present invention, a document data preprocessing unit that vectorizes a document by distributed representation and calculates a document vector; an isolated document removal unit that removes, as an isolated document, documents having a high degree of similarity less than a predetermined number, and a clustering processing unit that performs clustering in consideration of the degree of similarity of the documents from which the isolated documents have been removed by the isolated document removal unit; and a representative document extraction unit for extracting a representative document from the clusters clustered by the clustering processing unit.
Due to the processing of the isolated document removing unit, an isolated document having fewer than a predetermined number of documents having a degree of similarity higher than a predetermined threshold is not selected as a representative document.
Preferably, the representative document extracting unit extracts a document having a document vector having the closest cosine distance to the cluster center point.

It is preferable that the isolated document removal unit determines that a document having a document vector whose cosine distance is closer than the threshold value d and the number of similar documents smaller than the threshold value n is isolated and excludes the document.

The clustering processing unit performs clustering in consideration of document similarity by clustering document vectors using a method of clustering real-valued vectors,
A cluster center point is initialized by using a vector randomly extracted from the document vector calculated by the document data preprocessing unit, and in the random extraction, the probability of extracting the document vector is initialized with the document vector. and the minimum cosine distance to the cluster center point to the power of α and the number of document vectors whose cosine distance to the document vector is closer than the threshold value d to the power of β.

Furthermore, a ratio of the number of documents similar to the representative document is set by using n and d used in the processing in the isolated document removal unit, α and β used in the processing in the clustering processing unit, and the number of clusters k as a set of parameters. and a second index that is an evaluation index of clustering accuracy compared with the clustering correct data, and the first index and the second index Further having a scatter diagram display unit for plotting and displaying the set, plots automatically selected based on predetermined criteria from the plots displayed in the scatter diagram display unit, or manually at the user's discretion Preferably, the parameters are reset based on the selected plot.

According to another aspect of the present invention, there is provided a document analysis method by a computer, comprising: a document data preprocessing step of vectorizing a document by distributed representation and calculating a document vector; An isolated document removing step for removing, as an isolated document, a document whose similarity is higher than a predetermined threshold and the number of which is less than a predetermined number among the processed documents, and the similarity of the document from which the isolated document is removed by the isolated document removing step. A document analysis method is provided, comprising: a clustering processing step of performing clustering with consideration given to a document; and a representative document extracting step of extracting a representative document from the clusters clustered by the clustering processing step.

According to the present invention, it is possible to improve the efficiency of work for grasping the contents of a large number of documents.

FIG. 2 is a diagram showing an example of a method for grasping the contents of a document; 1 is a diagram illustrating an example of a document clustering technique; FIG. BRIEF DESCRIPTION OF THE DRAWINGS It is a functional block diagram which shows one structural example of the document-analysis system applicable to the document-analysis technique by the 1st Embodiment of this invention, and a document-analysis apparatus. FIG. 10 is a flow chart diagram showing an example of the processing flow of the document analysis technique according to the present embodiment; 5 is a flowchart showing a detailed processing example of document data preprocessing in FIG. 4; FIG. FIG. 6A is a diagram showing how representative document extraction processing is performed when isolated document removal processing is not performed. FIG. 6B is a diagram showing how representative documents are extracted when isolated documents are removed. It is a flowchart figure which shows the detailed example of the process of step S2. FIG. 8(a) is a diagram showing arbitrary extraction processing of a representative document, and FIG. 8(b) shows automatic extraction processing with a document having the closest cosine distance to the cluster center point as the representative document. It is a diagram. It is a flowchart figure which shows the detail of the process of step S4. 4 is a functional block diagram showing a positional configuration example of a document analysis apparatus according to a second embodiment of the present invention, and is a diagram corresponding to FIG. 3; FIG. FIG. 9 is a flow chart diagram showing the flow of system processing according to the second embodiment of the present invention; FIG. 10 is a diagram showing an example of a scatter diagram showing an example of parameter setting;

In this specification, a distributed representation of a document is, for example, a vectorization of the document content as a numeric vector of real numbers.
Document similarity is, for example, the cosine distance between document vectors.
An isolated document is a document in which the number of documents with similarities higher than a predetermined threshold is less than a predetermined number.
A representative document is, for example, a document that has many other similar documents in the original document.
Document analysis techniques according to embodiments of the present invention will be described in detail below with reference to the drawings.

(First embodiment)
FIG. 3 is a functional block diagram showing one configuration example of a document analysis system Y and a document analysis device X that can be applied to the document analysis technique according to the first embodiment of the present invention. FIG. 4 is a flow chart showing an example of the processing flow of the document analysis technique according to this embodiment. Tables 1 to 10 are tables showing examples of data tables used for document analysis.

Table 1 is a table showing a list of data tables of the document analysis system. A is 1. document data;2. It is document-related data consisting of document vectors. B is 1. 2. boilerplate rules; 3. named entity rules; It is pre-processing related data with a distributed representation model. C is 1. document cluster;2. Representative documents and clustering related data. D is 1. Parameter tuning related data with parameter sets.
Each of Tables 2 to 10 below is a table showing an example of each data table included in the data table list of Table 1.

Table 2 is a table showing an example of A1, that is, a document data table. The document data table A1 stores original text data of documents such as questions and answers for each ID.

Table 3 is a table showing an example of A2, that is, a document vector table. The document vector table A2 stores document vector element values v1, v2, . . . for each document ID.
B3. Document vector data calculated based on the distributed representation model described later, and the document ID corresponds to A1 and the ID of the document data table. Vector element values v1, v2, . . . have as many columns as the number of vector dimensions.

Table 4 is a table showing an example of B1, that is, a standard sentence rule table. The standard sentence rule table B1 is a list of standard sentences, and stores objects to be excluded if there is a corresponding or similar sentence in the document.

Table 5 is a table showing an example of B2, that is, a named entity rule table. The named entity rule table B2 replaces the portion of the sentence that matches the regular expression with, for example, "(label)". For example, "The construction date is 2019/1/23." is replaced with "The construction date is (date)." Here, the "(" and ")" surrounding the label is an example, and is a symbol for distinguishing between the label word "date" and words that appear in the original sentence in the sentence after replacement. be.

Table 6 is a table showing an example of B3, that is, a distributed representation model table. The distributed representation model table B3 is word vector data created based on a corpus (Wikipedia, etc.), and word vector element values have as many columns as the number of vector dimensions.

Table 7 is a table showing an example of C1, that is, a document cluster table. In the document cluster table C1, clusters formed by clustering are associated with document affiliations. The document ID corresponds to the document data ID in the document data table A1.

Table 8 is a table showing an example of C2, that is, the representative document table. The representative document table C2 manages representative documents of each cluster selected by the representative document extraction process. The representative document ID corresponds to the ID of the document data of A1.

Table 9 is a table showing an example of D1, that is, a parameter set table. The parameter set table D1 is a list of values (n, α, β, d, k) that can be set for various parameters such as clustering and representative document extraction. Additionally, a column "in use" is provided so that you can see which parameters are in use.

Table 10 is a table showing an example of correct data D2, that is, a correct data table. In the correct data table D2, cluster correct data created by a person is associated with document affiliation (document ID). The document ID corresponds to the document data ID of the document data table A1.

As shown in FIG. 3, the document analysis apparatus X according to this embodiment has, for example, a document analysis processing section X1 and a database (storage device) DB. Then, for example, a document analysis system Y includes a document analysis device X and a terminal device (user terminal) Z connected to the document analysis device X via a network (NT). The network NT may be wired or wireless.
The document analysis processing unit X1 includes a document data management unit 1 that manages the document data DB1 in the database (storage device) DB, a preprocessing-related function unit 3 that performs processing related to preprocessing of the document data, and a clustering process. It has a section 5 , a representative document extraction section 7 and a representative document content display section 11 . The preprocessing-related function unit 3 has an isolated document removal unit 3a.

The database (storage device) DB includes a document data DB1 storing the document data table A1, a document vector DB2 storing the document vector table A2, a document cluster DB3 storing the document cluster table C1, and a representative document table C2. , a preprocessing related data DB5 storing a fixed phrase rule table B1, a named entity rule table B2 and a distributed representation model table B3, and a parameter set DB6 storing a parameter set table D1.

Next, the flow of document analysis processing by the document analysis processing section X1 will be described. As shown in FIG. 4, when the process is started (START), in step S1, the preprocessing-related function unit 3 stores the document in the document data table A1 stored in the document data DB1 and the preprocessing-related data DB5. Preprocessing related data (formal sentence rule table B1, named entity rule table B2, distributed representation model table B3) are obtained, and the document data is preprocessed. Preprocessing includes calculation of document vector A2 based on the distributed representation model, and the like. Document vector A2 is stored in document vector DB2.
Next, in step S2, the isolated document removing unit 3a removes isolated documents having fewer than a predetermined number of documents with a degree of similarity higher than a predetermined threshold. The process of removing an isolated document includes, for example, a process of determining that a document having a small number of other documents having document vectors with a close cosine distance is isolated and excluded. The document vector after removing the isolated document is stored in the document vector DB2.

Next, in step S3, the clustering processing unit 5 performs clustering processing based on the parameter set D1 stored in the parameter set DB6 and the document vector A2 stored in the document vector DB2. In the clustering process, the cosine distances between document vectors are taken into consideration, and clusters are formed from a group of documents having close cosine distances. Correspondence between the formed cluster and the affiliation of the document is stored in the document cluster DB3.
Next, in step S4, the representative document extraction unit 7 extracts representative documents from the document clusters in the document cluster table C1 based on the parameter set D1 stored in the parameter set DB6. Extraction processing of a representative document is processing for selecting a document having the closest cosine distance to a cluster central point for a given cluster. The extracted representative document is stored in the representative document DB 4 as data in the representative document table C2. The reason for extracting from clusters in this processing is to eliminate duplication of content among a plurality of representative documents.

Next, in step S5, the representative document content display (control) unit 11 performs summary display processing of the representative document content using the representative document table C2 stored in the representative document DB4.
Thus, the processing ends (END).

Next, each of the above processes will be described in detail.
FIG. 5 is a flow chart showing a detailed processing example of the document data preprocessing in FIG.
first. When the process starts (START), the following processes are performed.
Step S11: Division of document into sentences Step S12: Morphological analysis of sentences Step S13: Removal of symbols in sentences Step S14: Removal of standard sentences in documents Step S15: Extraction of named entities in sentences Step S16: Distributed representation calculation step S17 of documents : Summary of document content

A known technique can be used for the above processing. As an example, for the processes of steps S13, S14, and S17, the technologies of the unnecessary word removal processing unit, unnecessary sentence removal processing unit, and summary generation unit of Japanese Patent Application No. 2018-162525 can be used, respectively. For the process of step S15, known named entity extraction techniques such as Hidden Markov Model and Conditional Random Field can be used. In addition, for the processing of step S16, a technique for calculating distributed representations of documents called Doc2Vec or Paragraph Vector (Quoc Le, Tomas Mikolov, “Distributed representations of sentences and documents,” International conference on machine learning, 2014) may be used. can be done.
Next, an isolated document removal process will be described. The processing for removing isolated documents in which the number of documents whose similarity is higher than a predetermined threshold is less than a predetermined number is, for example, determined as isolated documents in which the number of documents whose cosine distance is closer than a predetermined threshold d is less than a predetermined number n. This is the process of exclusion.
Note that the cosine distance threshold d used in the isolated document determination is the same as the parameter d used in the clustering process.
A parameter d is a threshold that serves as a criterion for determining whether documents are similar. In the removal of isolated documents, it is desirable to determine that a document having few similar documents among all documents is "isolated", so the parameter d, which serves as a document similarity determination criterion, is used as a threshold.
Note that the threshold for the number of documents is set to "threshold n" since it will be referred to as a parameter to be tuned in the following second embodiment.

FIG. 6A is a diagram showing how representative document extraction processing is performed when isolated document removal processing is not performed. The number of clusters is fixed at four.
Clustering assigns every document to a cluster. Therefore, if clustering is performed without removing isolated documents, the isolated documents are also assigned to one of the clusters. A cluster containing isolated documents consists of isolated documents and similar documents. For example, when the predetermined number n is 1, the number of documents whose cosine distance from the isolated document is closer than the predetermined threshold value d is less than 1, and there is no document similar to the isolated document. Therefore, the cluster does not include documents similar to the isolated document, and a cluster consisting only of the isolated document is formed. Therefore, when a representative document is extracted from a cluster, an isolated document may be selected.

FIG. 6B is a diagram showing how representative documents are extracted when isolated documents are removed. As shown in FIG. 6B, by removing isolated documents in advance from cluster formation targets, it is possible to prevent isolated documents from being selected as representative documents.

FIG. 7 is a flow chart showing a detailed example of the processing in step S2. As shown in FIG. 7, first, in step S2-1, one unprocessed document is specified from among the documents registered in the document data DB1. In step S2-2, the number of documents whose cosine distance is closer than a predetermined threshold value d is counted in one identified document. At step S2-3, the number of documents counted at step S2-2 is compared with a predetermined number n. The predetermined number n can be set in advance based on whether or not the document can be regarded as an isolated document.

At step S2-3, it is determined whether the counted number of documents is smaller than a predetermined number n. If No in step S2-3, the process advances to step S2-4 to store the document in the document vector DB2 without excluding the document. Then, the process proceeds to step S2-6.

If Yes in step S2-3, the process advances to step S2-5 to exclude the document as an isolated document. Then, in step S2-6, it is determined whether or not the current document count number n is equal to the total document number m registered in the document data DB1. If Yes in step S2-6, the process ends (END). If No in step S2-6, the process proceeds to step S2-7, sets n=n+1, and returns to step S2-1.

The isolated document can be excluded by continuously performing the above processing.
Thus, by removing isolated documents before clustering, it is possible to prevent isolated documents from being selected as representative documents.

Next, clustering processing in step S3 will be described.
In this embodiment, an improved version of k-means++ algorithm can be used.
The improved k-means++ algorithm has the following improvements over the conventional k-means++.
1) Using the spherical k-means method considering the cosine distance between document vectors (Kurt Hornik, Ingo Feinerer, Martin Kober, Christian Buchta, “Spherical k-Means Clustering,” Journal of Statistical Software, September 2012, Volume 50, (see Issue 10).
2) Randomly extract the initial value of the cluster center point using the document vector with the following probability.

Here, the intention of making each parameter variable is as follows.
α: Improve clustering accuracy and convergence speed β: Make it easier to select the initial value of the cluster center point from where the document vectors are concentrated
d: Adjusting the cosine distance threshold for judging that document vectors are similar When α=2 and β=0, the algorithm using Equation 1 corresponds to conventional k-means++.

Next, the automatic extraction processing of the representative document in step S4 will be described.
FIG. 8(a) is a diagram showing arbitrary extraction processing of a representative document, and FIG. 8(b) is a diagram showing automatic extraction processing of a document having a document vector with the closest cosine distance to the cluster center point as the representative document. It is a figure which shows the state of.
As shown in FIG. 8(a), according to arbitrary extraction processing of the representative document, the document vector of the representative document may be separated from the center of the cluster. Therefore, there is a problem that the number of intra-cluster documents having a document vector with a cosine distance close to the document vector of the representative document is reduced.

FIG. 9 is a flowchart showing the details of the processing in step S4.
As shown in FIG. 9, cluster center points are obtained in step S4-1 of step S4. At step S4-2, the cosine distance between the document vector of each document in the cluster and the cluster center point is compared. At step S4-3, the document having the document vector with the closest cosine distance to the cluster center point is taken as the representative document. Then, the process ends (END).

As shown in FIG. 8B, by the above processing, when representative documents are extracted based on the distance from the cluster central point, the document having the document vector closest to the cluster central point is selected as the representative. In the k-means method, clusters are formed such that the distance between the cluster center point and the vectors belonging to the cluster is minimized. Therefore, there are many documents in the cluster that have document vectors that are close in cosine distance to the cluster center point. Therefore, by making the document whose cosine distance is closest to the cluster center point the representative document, there is an advantage that the number of documents in the cluster having the document vector whose cosine distance is close to that of the representative document increases. be. As a result, the extracted representative document satisfies the desired property of having a large number of similar documents in the entire original mass of documents and having common content.

As described above, according to this embodiment, it is not necessary to read all the documents in the cluster in order to automatically extract the representative document from the cluster. Therefore, processing is simplified.
Since the extracted representative document is a document with common contents in a large amount of original documents, it is possible to grasp the main contents of the large amount of original documents by reading the representative document.
Therefore, it is possible to improve the efficiency of the work of grasping the contents of a large number of documents.

(Second embodiment)
Next, a document analysis technique according to the second embodiment of the invention will be described. In the document analysis technique according to this embodiment, a parameter tuning support function is added in addition to the first embodiment.
FIG. 10 is a functional block diagram showing one configuration example of the document analysis apparatus according to this embodiment, and corresponds to FIG. The document analysis apparatus X of FIG. 10 has an evaluation index calculation section 15, a parameter setting section 17, and a scatter diagram display section 21 in addition to the document analysis apparatus of FIG. In addition to the database of FIG. 3, it has a correct data database DB7 for storing correct data D2 (DBa, see Table 10).

The evaluation index calculator 15 calculates the ratio of the number of documents similar to the representative document (first index) from the representative document C2, and calculates the clustering accuracy evaluation index (second index) from the correct data D2. The parameter setting unit 17 automatically selects a plot that maximizes the weighted sum of the first index and the second index calculated by the evaluation index calculation unit 15 . It should be noted that the parameters can also be selected by the user's judgment. The scatter diagram display unit 21 displays a scatter diagram, which will be described later.

FIG. 11 is a flow chart showing the flow of system processing according to this embodiment. Steps that perform the same processing as in FIG. 4 are denoted by the same reference numerals, and descriptions thereof are omitted.

In FIG. 11, after step S1, in step S9, it is determined whether or not all the parameters in the parameter set DB6 have been processed. If the process has not been completed (No), in step S10, the parameter of the next ID is acquired from the parameter set DB.
Next, in step S2a, the isolated document removing section 3a performs the process of removing isolated documents. This process is performed using the threshold value n and the threshold value d included in the acquired parameter set.

Next, in step S3a, the clustering processing unit 5 performs clustering processing. This process is performed using the parameters n, α, β, d, and k of the acquired parameter set.
Next, in step S4, the representative document extraction unit 7 extracts a representative document and registers it in the representative document DB4.

Next, in step S6, the evaluation index calculation unit 15 performs evaluation index calculation processing. This process calculates the following two indexes.
1) Ratio of the number of documents similar to the representative document (first indicator)
A document whose cosine distance between the document vector of the document registered in the document data DB 1 and the document vector of at least one representative document registered in the representative document DB 4 is equal to or less than the threshold value d. is the ratio of
2) Evaluation index of clustering accuracy (second index)
The document cluster table C1 and the correct data D2 are compared to calculate an evaluation index of clustering accuracy. As an evaluation index, for example, an Adjusted Rand Index or Adjusted Mutual Information can be used.
Then, the process returns to step S9.
On the other hand, in the case of Yes in step S9, the process proceeds to step S7, and the scatter diagram display unit 21 plots all sets of evaluation indices in scatter diagrams.

Next, in step S8, the following parameters are set.
1) Automatically select the plot that maximizes the weighted sum of the first index and the second index.
2) It is also possible to select arbitrary plots at the user's discretion.
Next, step S3 (clustering processing) and step S4 (representative document extraction processing) similar to those in the first embodiment are performed.
With the above, the parameter tuning support processing for continuously resetting the parameters is completed.

In the second embodiment, a parameter tuning support function is added to the first embodiment.
For example, the parameter setting effect is visualized by a scatter diagram using the similar document number ratio (first index) and the clustering evaluation index (second index).

FIG. 12 is a diagram showing an example of a scatter diagram showing parameter setting examples. As shown in FIG. 12, in the scatter diagram, for example, the horizontal axis indicates the number of similar documents (%) (first index), and the vertical axis indicates the clustering evaluation index (second index). Different parameter sets (sets of values of n, α, β, d, k (number of clusters)) are used to calculate indices in each plot.

Here, since all possible combinations of values cannot be covered, it is assumed that combinations of parameter values to be plotted are registered in the parameter set DB in advance. As shown in the table configuration diagram of the parameter set DB (DB10) in Table 9, one record specified by an ID of 1 is one combination of parameter values, and the columns of the table are parameter names (n, α, β, d, k). Also add a "in use" column so you can see which parameters are in use.

Here, it is assumed that the correct data for clustering is registered in the "correct data (D2) DB" in advance. A table configuration diagram of the correct data (D2) DB is shown in Table 10 above. The document ID corresponds to the document data ID in the document data table A1. The documents registered in the correct data D2 do not have to be all the documents registered in the document data table A1, and some manually extracted document data can be used.

As described above, the optimization of parameters in the processing algorithm is supported as follows.
1) Automatic selection of plots based on predetermined criteria 2) Manual selection of plots by user's judgment

A dashed line L11 in FIG. 12 is a straight line indicating a criterion for maximizing the weighted sum of the similar document number ratio (first index) and the clustering evaluation index (second index). It is assumed that the weights of the first index and the second index when calculating the weighted sum are positive real numbers and registered in advance. The user can adjust which of the first index and the second index is emphasized by changing the weight value. The weight ratio represents the slope of the straight line L11. Plot P1 can be automatically selected on this straight line L11.
Also, as shown in P2, another plot on the scatter diagram can be selected at the user's discretion.
With the above configuration, it is possible to save labor in parameter adjustment work.

Examples of the use of document analysis technology include the following.
1) FAQ (representative question) preparation For the question group, similar questions are clustered and classified.
Extract representative questions from each category and write a summary sentence.
2) Similar reports are clustered and classified for the group of failure information analysis maintenance and inspection reports.
By extracting the representative report of each classification and reading the contents described in it, we grasp the details of common failures.

According to this embodiment, it is possible to improve the efficiency and accuracy of the work of extracting a representative document of a document cluster. High efficiency can be achieved by automating the representative document extraction process.
Also, high accuracy can be achieved by maximizing the number of documents similar to the representative document.
Furthermore, it can assist in optimizing parameters in the processing algorithm. Therefore, it is possible to save labor in parameter adjustment work.

Processing and control can be realized by software processing by a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), or by hardware processing by an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). .
In the above-described embodiments, the illustrated configurations and the like are not limited to these, and can be changed as appropriate within the scope of exhibiting the effects of the present invention. In addition, it is possible to carry out by appropriately modifying the present invention as long as it does not deviate from the scope of the purpose of the present invention.

In addition, each component of the present invention can be selected arbitrarily, and the present invention includes an invention having a selected configuration.
Also, a program for realizing the functions described in the present embodiment is recorded in a computer-readable recording medium, and the program recorded in this recording medium is read by a computer system and executed, thereby processing of each section. may be performed. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices.
The "computer system" also includes the home page providing environment (or display environment) if the WWW system is used.

The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. Furthermore, "computer-readable recording medium" means a medium that dynamically retains a program for a short period of time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It also includes those that hold programs for a certain period of time, such as volatile memories inside computer systems that serve as servers and clients in that case. Further, the program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system. At least part of the functions may be implemented in hardware such as integrated circuits.

INDUSTRIAL APPLICABILITY The present invention is applicable to document analysis devices.

X document analysis device X1 document analysis processing unit Y document analysis system NT network DB database (storage device)
1 document data management unit 3 preprocessing related function unit 3a isolated document removal unit 5 clustering processing unit 7 representative document extraction unit 11 representative document content display unit 15 evaluation index calculation unit 17 parameter setting unit 21 scatter diagram display unit

Claims (4)

a document data preprocessing unit that vectorizes a document by distributed representation and calculates a document vector;
an isolated document removing unit for removing, as an isolated document, a document whose similarity is higher than a predetermined threshold value and whose number is less than a predetermined number from among the documents preprocessed by the document data preprocessing unit;
a clustering processing unit that performs clustering in consideration of the degree of similarity of documents from which isolated documents have been removed by the isolated document removal unit;
a representative document extraction unit for extracting a representative document from the clusters clustered by the clustering processing unit ;
The isolated document removing unit determines that a document having a document vector whose cosine distance is closer than the threshold value d and the number of similar documents is less than the threshold value n is isolated and excludes the document,
The clustering processing unit clusters the document vectors using a method of clustering real-valued vectors to perform clustering in consideration of the degree of similarity of the documents, and the cluster center point is calculated by the document data preprocessing unit. Initialize by making it a vector randomly extracted from the document vector,
In the random extraction, the probability of extracting the document vector is the minimum value of the cosine distance between the document vector and the initialized cluster center point to the power of α, and the cosine distance between the document vector and the threshold d. is a probability proportional to the β-th power of the number of document vectors that are close to each other . moreover,
It is the ratio of the number of documents similar to the representative document, with n and d used in the processing in the isolated document removal unit, α and β used in the processing in the clustering processing unit, and the number of clusters k as a set of parameters. an evaluation index calculation unit that calculates a first index and a second index that is an evaluation index of clustering accuracy compared with correct data for clustering;
further comprising a scatter diagram display unit that plots and displays a set of the first index and the second index;
From the plots displayed in the scatter diagram display unit, resetting the parameters based on plots automatically selected based on predetermined criteria or plots manually selected by user's judgment 2. A document analysis apparatus according to claim 1 . The representative document extraction unit
3. The document analysis apparatus according to claim 1, wherein a document having a document vector having the closest cosine distance to a cluster center point is extracted. A computer-based document analysis method comprising:
a document data preprocessing step of vectorizing a document by distributed representation and calculating a document vector;
an isolated document removing step of removing, as an isolated document, a document whose similarity is higher than a predetermined threshold value and whose number is less than a predetermined number from among the documents preprocessed by the document data preprocessing step;
a clustering processing step of performing clustering in consideration of the degree of similarity of documents from which isolated documents have been removed by the isolated document removal step;
a representative document extraction step of extracting a representative document from the clusters clustered by the clustering processing step ;
In the isolated document removing step, a document having a document vector whose cosine distance is closer than the threshold value d and the number of similar documents is smaller than the threshold value n is determined to be isolated and excluded,
In the clustering processing step, the document vectors are clustered using a method of clustering real-valued vectors to perform clustering in consideration of the similarity of the documents, and the cluster center point is calculated by the document data preprocessing step. Initialize by making it a vector randomly extracted from the document vector,
In the random extraction, the probability that the document vector is extracted is α-th power of the minimum cosine distance between the document vector and the initialized cluster center point, and the cosine distance between the document vector and the threshold value d. is a probability proportional to the β-th power of the number of document vectors that are close to each other .

Images