JP5642229B2 - Importance determination system, importance determination method, and computer program - Google Patents

Description

  The present invention relates to a technique for determining the importance of a word in a document including text.

  Conventionally, there is a TF-IDF algorithm as a method for determining an important word in a text document (see Non-Patent Document 1). In the TF-IDF algorithm, the index TF-IDF is calculated by dividing the appearance frequency TF (Term Frequency) of the word w by the document frequency DF (Document Frequency) including the word w. Then, the higher the value of TF-IDF, the more important the word is determined. In other words, in the TF-IDF algorithm, the index TF-IDF is calculated to be higher as a word having a higher appearance frequency TF only in a specific document and not appearing in another document as an important word characterizing the document.

  Specifically, the value of the index TF-IDF is often calculated using, for example, the following equation. In the following description, “a_b” indicates that the subscript “b” is added to the character “a”. “A_ (b, c)” indicates that the subscript “b, c” is added to the character “a”.

  n_ (i, j) is the number of appearances of the word i in the document j. | D | indicates the total number of documents (total number of documents). D_i represents the number of documents including the word i. tfidf_ (i, j) represents a score representing the importance of the word i in the document j (hereinafter referred to as “importance score”). When the value of tfidf_ (i, j) exceeds a predetermined threshold, it is determined that the word i is an important word in the document j.

Gerard Salton, Christopher Buckley, “TERM-WEIGHTING APPROACHES IN AUTOMATIC TEXT RETRIEVAL” Information Processing & Management Vol.24, No.5, pp.513-523, 1988.

However, the conventional technique has a problem that important words that appear in common in a plurality of documents cannot be determined appropriately.
For example, the value of idf_i is 0 for words that appear in all documents. Therefore, the values of tfidf_ (i, j) obtained by multiplying idf_i are all 0, and the importance cannot be determined appropriately.

  Note that it is possible to prevent the calculated value from becoming 0 by not performing the log operation when calculating the value of idf_i. However, if such processing is performed, the influence of tf_ (i, j) on the calculated value of tfidf_ (i, j) increases. Therefore, if it is a frequent word that appears in any document, the value of tf_ (i, j) exceeds the threshold value in any document, and the importance cannot be determined appropriately.

  In view of the above circumstances, an object of the present invention is to provide a technique capable of determining the importance of a word appearing in common in a plurality of documents with higher accuracy.

One aspect of the present invention, for each single word, term frequency indicating the frequency of occurrence of each word is determined whether different or not have significant difference and said word occurrence frequency in the other documents in one document, the determination An importance determination system comprising: an importance determination unit that determines the importance of the word in the one document based on a result.
One aspect of the present invention is the importance determination system described above, further including a word appearance frequency acquisition unit that acquires the word appearance frequency for each document.

One aspect of the present invention is the importance determination system described above, wherein the importance determination unit is determined to have a significant difference in the set of word appearance frequencies of each document acquired for a certain word. and when said word occurrence frequency is satisfied a predetermined condition indicating a high that is, it determines that the word is a key word in the previous SL document term frequency is acquired.

One aspect of the present invention is the importance determination system described above, wherein the importance determination unit is determined to have a significant difference in the set of word appearance frequencies of each document acquired for a certain word. and when said word occurrence frequency is satisfied a predetermined condition indicating that low, it is determined that the word is a key word in each document other than before SL document term frequency is acquired.

  One aspect of the present invention is the importance determination system described above, wherein the importance determination unit determines that only words having a predetermined part of speech are important words.

  One aspect of the present invention is the importance determination system described above, wherein a word conversion unit that converts, for each document, a word having the same or similar meaning among words appearing in the document into one word. The word appearance frequency acquisition unit acquires a word appearance frequency for each word after being converted by the word conversion unit.

One aspect of the present invention, for each single word, term frequency indicating the frequency of occurrence of each word is determined whether different or not have significant difference and said word occurrence frequency in the other documents in one document, the determination An importance determination step of determining importance of the word in the one document based on a result.

  One aspect of the present invention is a computer program for causing a computer to function as the importance determination system.

  According to the present invention, it is possible to determine the importance of words appearing in common in a plurality of documents with higher accuracy.

It is a schematic block diagram which shows the function structure of the importance determination system 10 in 1st embodiment. It is a figure which shows the outline of a significance level. 5 is a flowchart illustrating a specific example of processing of the importance determination system 10. It is a figure which shows the specific example for showing the effect of the importance determination system. It is a schematic block diagram which shows the importance determination system 10a as a modification of 1st embodiment. It is a schematic block diagram which shows the function structure of the importance determination system 20 in 2nd embodiment. 5 is a flowchart illustrating a specific example of processing of the importance determination system 20. It is a schematic block diagram which shows the function structure of the importance determination system 30 in 3rd embodiment. 5 is a flowchart illustrating a specific example of processing of the importance determination system 30.

[First embodiment]
FIG. 1 is a schematic block diagram showing a functional configuration of an importance determination system 10 in the first embodiment. The importance determination system 10 includes one or a plurality of information processing apparatuses. For example, when the importance determination system 10 is configured by a single information processing apparatus, the information processing apparatus includes a CPU (Central Processing Unit) connected via a bus, a memory, an auxiliary storage device, and the like, and an importance determination program Execute. By executing the importance determination program, the information processing apparatus functions as an apparatus including the document information storage unit 101, the word extraction unit 102, the word appearance frequency acquisition unit 103, and the importance determination unit 104. Note that all or part of the functions of the importance determination system 10 may be realized using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). good. The importance determination system 10 may be realized by dedicated hardware. The importance determination program may be recorded on a computer-readable recording medium. The computer-readable recording medium is a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a semiconductor storage device (for example, SSD: Solid State Drive), a hard disk built in a computer system, or a semiconductor storage. A storage device such as a device. The importance determination program may be provided via a telecommunication line.

  The document information storage unit 101 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The document information storage unit 101 stores document information. The document information represents text included in the document for each piece of document identification information. The document identification information is identification information given in advance for each document. The document information may include information such as the document author name, title, publication date, and publication date.

  For each document stored in the document information storage unit 101, the word extraction unit 102 extracts individual words from the text included in the document. The word extraction unit 102 extracts individual words by performing morphological analysis, for example. In other words, the word extraction unit 102 separates each document into words by performing morphological analysis for each document, for example. The word extraction unit 102 may output the part of speech of each word as a result of morphological analysis.

  The word appearance frequency acquisition unit 103 acquires a word appearance frequency indicating the frequency of appearance of each word for each document based on the extraction result by the word extraction unit 102. For example, the word appearance frequency acquisition unit 103 may count the number of times each word appears for each document, and may acquire the total result as the word appearance frequency of each word. For example, for each document, the word appearance frequency acquisition unit 103 may divide the number of times each word has appeared by the total number of words that have appeared in the document, and acquire the calculation result as the word appearance frequency of each word. The word appearance frequency calculated as the latter makes it possible to increase the accuracy of the determination result of the importance determination system 10 when the total number of words appearing for each document is different.

  The importance determination unit 104 determines the importance of each word in each document based on the word appearance frequency of each word acquired by the word appearance frequency acquisition unit 103.

  Hereinafter, a specific example of processing performed by the importance determination unit 104 will be described. For example, when the word appearance frequency in a certain document j is significantly higher than the word appearance frequency in another document for the word i, the importance determination unit 104 determines that the word i is an important word in the document j. Judge as. As a specific example for determining whether or not the word appearance frequency is significantly higher than the other word appearance frequencies, the importance determination unit 104 uses a technique for determining whether or not it is an outlier. It may be used. Specific examples of techniques for determining whether an outlier is a Smirnov-Grubbs test or a Thompson test. Below, the formula used in the Smirnov-Grubbs test is shown as an example.

In Expression (4), T_x represents a test statistic, n represents the number of samples, t represents a decentralized point of the t distribution when the required significance level is α (α of the t distribution with n−2 degrees of freedom). / N × 100 percentile). The significance level is, for example, a value such as 1% or 5%, and is appropriately determined by the designer or user of the importance determination system 10.

  FIG. 2 is a diagram showing an outline of the significance level. In FIG. 2, the graph indicated by the curve shows the t distribution. The range of the value on the horizontal axis in the region located near the left and right ends and indicated by diagonal lines indicates a range of values having a significant difference. For example, if the significance level (α) is a two-sided test with 5%, the total area of the portion surrounded by the t-distribution curve and the horizontal axis is 100, and the left and right areas (shaded areas in the figure) are 2.5. A straight line parallel to the vertical axis is drawn so that the intersections of the straight line and the horizontal axis are x1 and x2, respectively. On the right side of the graph of FIG. 2, when it is larger than x2, it is regarded that the value is large because there is a significant difference with respect to the average value by the 5% two-sided test. On the left side of the graph of FIG. 2, when it is smaller than x1, it is considered that the value is small because there is a significant difference with respect to the average value by the 5% two-sided test. In the present embodiment, in the graph in which the horizontal axis indicates the word appearance frequency and the vertical axis indicates the frequency, the value included in the significant difference range with the higher word appearance frequency value is higher than the other word appearance frequencies. Is determined to be high with a significant difference. On the contrary, it is determined that the value included in the significant difference range having the lower word appearance frequency value is significantly lower than the other word appearance frequencies.

  The importance determination unit 104 determines, for each word i, a word appearance frequency that is an outlier from the set of word appearance frequencies in each document j. Then, the importance determination unit 104 determines an important word based on the outlier determination result. For example, the importance determination unit 104 determines the word i as an important word of the document j for all combinations of the word i and the document j regarding the word appearance frequency determined as an outlier.

  FIG. 3 is a flowchart illustrating a specific example of processing of the importance determination system 10. The flowchart shown in FIG. 3 shows a process in which the importance determination unit 104 determines an important word based on an outlier. Hereinafter, a specific example of processing of the importance determination system 10 will be described with reference to FIG.

  First, for each document stored in the document information storage unit 101, the word extraction unit 102 extracts individual words from the text included in the document (step S101). Next, the word appearance frequency acquisition unit 103 acquires the word appearance frequency of each word for each document based on the extraction result by the word extraction unit 102 (step S102). Next, the importance determination unit 104 determines, for each word i, a word appearance frequency that is an outlier from the set of word appearance frequencies in each document j (step S103). Then, the importance determination unit 104 determines that the word i is an important word of the document j based on the word i and the document j having the word appearance frequency determined as an outlier (step S104).

  In the importance determination system 10 configured as described above, for each word, an important word is determined based on the degree of appearance frequency in each document that is different from the appearance frequency in other documents. That is, for the word i, when the word appearance frequency in a certain document j is a high value with a significant difference from the word appearance frequency in another document, the word i is determined as an important word in the document j. Therefore, the importance of words that appear in common in a plurality of documents can be determined more accurately.

  FIG. 4 is a diagram illustrating a specific example for illustrating the effect of the importance determination system 10. FIG. 4 shows the word appearance frequency of each word (phone, communication, Internet, customer, contract, bank) in the documents 1 to 5. In the conventional technique, since the word “communication” appears frequently in all the documents 1 to 5, it is difficult to accurately determine whether or not it is an important word. On the other hand, according to the importance determination system 10, the word appearance frequency is compared for each word for each word, and when it is determined that there is a significant difference, it is determined as an important word. For this reason, it is possible to accurately determine whether a word that appears frequently in all documents is an important word. For example, the word “communication” can be determined as an important word in the document 2.

  Further, according to the importance determination system 10, it is possible to determine that a word that does not have a large difference in word appearance frequency is not an important word even if the word appears frequently in all documents. Become. For example, as with the word “communication”, the word “telephone”, which has a high appearance frequency as a whole, has a small difference in word appearance frequency for each document, so it can be determined that it is not an important word in any document. .

  The importance determination system 10 can also solve the following problems. In the conventional technique, it is difficult to accurately determine whether a word with a low word appearance frequency is an important word. For example, the word “bank” in FIG. 4 has a low word appearance frequency in all documents. Therefore, even if the word appearance frequency in the document 1 is high unlike the word appearance frequency in other documents, it is conventionally difficult to determine the word “bank” as an important word of the document 1. On the other hand, the importance determination system 10 can solve such a problem and determine whether or not a word having a low word appearance frequency is an important word with high accuracy. For example, according to the importance determination system 10, the word appearance frequencies of the word “bank” are compared for each document, and when it is determined that there is a significant difference, the word is determined to be an important word. Therefore, the word “bank” having a low word appearance frequency can be determined as the important word of the document 1.

<Modification>
The importance determination unit 104 does not necessarily determine an important word based on all outliers. For example, the importance determination unit 104 may determine the word i as an important word of the document j for a combination of the word i and the document j determined as an outlier by a reoccurrence process within a predetermined number of times. The importance determination unit 104 executes the restart process until a predetermined number of outliers are obtained, and determines the word i as an important word of the document j for the combination of the word i and the document j determined as the outliers. You may do it.

  Further, the importance determination unit 104 may determine a predetermined number as an outlier from a large value among the test statistics calculated based on the equation (4). And the importance determination part 104 may determine an important word based on the determination result performed in this way.

  The importance determination unit 104 determines that the word i is a word i in each document other than the document j when the word appearance frequency in a certain document j is significantly lower than the word appearance frequency in another document. It may be determined as an important word. With this configuration, it is possible to determine a word that does not easily appear only in a specific document as an important word in another document. By using the important word determined in this way, for example, it is possible to clearly classify whether the document is about a topic related to the important word.

  The importance determination unit 104 may determine only words having a predetermined part of speech (for example, nouns) as important words among the words that satisfy the conditions in the above-described processing. In this way, by limiting the part of speech of a word determined as an important word, it is possible to output only words suitable for subsequent processing as important words.

  The importance determination unit 104 may determine a value representing the importance of each word in each document instead of determining the important word of each document. For example, the importance determination unit 104 calculates, for each word i, a statistical value indicating the degree to which the word appearance frequency in a certain document j is different from the word appearance frequency in another document, and uses the calculated value as the word i in the document j. It may be determined as a value representing the importance of. In this case, the importance determination unit 104 outputs a value representing the importance of each word for each document as a determination result. By being configured in this way, it is possible to express the degree of importance of each word in each document as a multi-stage value, instead of using the binary value of whether or not it is an important word as an output value. Become.

  In the example illustrated in FIG. 1, the importance determination system 10 is implemented as a device including the document information storage unit 101, but the document information storage unit 101 may be provided outside the importance determination system 10. FIG. 5 is a schematic block diagram showing an importance determination system 10a as a modified example of the first embodiment. In this case, the importance determination system 10a and the document information storage unit 101 are communicably connected via a network. The word extraction unit 102a, the word appearance frequency acquisition unit 103a, and the importance determination unit 104a included in the importance determination system 10a function in the same manner as the functional units of the same name in the importance determination system 10. The word extraction unit 102a receives document information from the document information storage unit 101 via the network. With this configuration, the importance determination system 10a can determine an important word and a degree of importance for document information stored in an arbitrary document information storage unit 101.

[Second Embodiment]
FIG. 6 is a schematic block diagram showing a functional configuration of the importance determination system 20 in the second embodiment. The importance determination system 20 includes one or a plurality of information processing apparatuses. For example, when the importance determination system 20 is configured by a single information processing apparatus, the information processing apparatus includes a CPU, a memory, an auxiliary storage device, and the like connected by a bus and executes an importance determination program. By executing the importance determination program, the information processing apparatus includes a document information storage unit 201, a word extraction unit 202, a word appearance frequency acquisition unit 203, an importance determination unit 204, a conversion dictionary storage unit 211, and a word conversion unit 212. Function as. All or some of the functions of the importance determination system 20 may be realized using hardware such as an ASIC, PLD, or FPGA. The importance determination system 20 may be realized by dedicated hardware. The importance determination program may be recorded on a computer-readable recording medium. The importance determination program may be provided via a telecommunication line.

The document information storage unit 201, the word extraction unit 202, the word appearance frequency acquisition unit 203, and the importance determination unit 204 function in the same manner as each functional unit with the same name in the first embodiment.
The conversion dictionary storage unit 211 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The conversion dictionary storage unit 211 stores one or more words (hereinafter referred to as “synonyms”) having the same meaning as the representative word in association with each representative word. For example, for the representative word “economy”, words such as “economy”, “keizai”, and “keizai” are stored in association with each other as synonyms.

  The word conversion unit 212 converts synonyms into representative words among the words extracted by the word extraction unit 202 based on the representative words and synonyms stored in the conversion dictionary storage unit 211. For example, if the word extracted by the word extraction unit 202 includes one or more of “economy”, “keizai”, and “keizai”, these are all converted into the representative word “economy”. .

FIG. 7 is a flowchart illustrating a specific example of processing of the importance determination system 20. Hereinafter, a specific example of the processing of the importance determination system 20 will be described with reference to FIG.
First, for each document stored in the document information storage unit 201, the word extraction unit 202 extracts individual words from the text included in the document (step S201). Next, the word conversion unit 212 converts synonyms into representative words among the words extracted by the word extraction unit 202 (step S211). Next, the word appearance frequency acquisition unit 203 acquires the word appearance frequency of each word for each document based on the conversion result by the word conversion unit 212 (step S202). Next, the importance determination unit 204 determines, for each word i, a word appearance frequency that is an outlier from the set of word appearance frequencies in each document j (step S203). Then, the importance determination unit 204 determines that the word i is an important word of the document j based on the word i and the document j having the word appearance frequency determined as an outlier (step S204).

The importance determination system 20 configured as described above can achieve the same effects as the importance determination system 10 in the first embodiment.
Further, in the importance determination system 20, among words extracted by the word extraction unit 202, words corresponding to synonyms are converted into representative words stored in the conversion dictionary storage unit 211 in association with the synonyms. . For this reason, it is possible to determine an important word with high accuracy even for a document in which notation fluctuation occurs.
The importance determination system 20 in the second embodiment may be modified and configured similarly to the importance determination system 10 in the first embodiment.

[Third embodiment]
FIG. 8 is a schematic block diagram showing a functional configuration of the importance determination system 30 in the third embodiment. The importance determination system 30 can communicate with the Web server 40 via a network. The Web server 40 is a server that provides text so that it can be viewed. The Web server 40 may be, for example, a server that provides a document registered in a blog, a server that provides a news article so that it can be browsed, or a server that provides a dictionary. Alternatively, the server may provide a search history.

  The importance determination system 30 includes one or a plurality of information processing apparatuses. For example, when the importance determination system 30 is configured by a single information processing apparatus, the information processing apparatus includes a CPU, a memory, an auxiliary storage device, and the like connected by a bus and executes an importance determination program. By executing the importance determination program, the information processing apparatus functions as an apparatus including a document information storage unit 301, a word extraction unit 302, a word appearance frequency acquisition unit 303, an importance determination unit 304, a clustering unit 321, and a word conversion unit 312. To do. All or some of the functions of the importance determination system 30 may be realized using hardware such as an ASIC, PLD, or FPGA. The importance determination system 30 may be realized by dedicated hardware. The importance determination program may be recorded on a computer-readable recording medium. The importance determination program may be provided via a telecommunication line.

The document information storage unit 301, the word extraction unit 302, the word appearance frequency acquisition unit 303, and the importance determination unit 304 function in the same manner as the functional units having the same names in the first embodiment.
Based on the usage example of each word obtained from the server 40, the clustering unit 321 performs clustering so that the words extracted by the word extracting unit 302 are classified into one cluster for each word having the same meaning. . As a clustering method, for example, the K-means method may be applied. As a result of clustering by the clustering unit 321, for example, words such as “economy”, “economy”, “keizai”, and “keizai” are classified into one cluster. The clustering unit 321 defines a word having the highest appearance frequency among the words of each cluster as a representative word, and defines words other than the representative word as synonyms.
Based on the representative words and synonyms defined by the clustering unit 321, the word conversion unit 312 converts synonyms into representative words among the words extracted by the word extraction unit 302.

FIG. 9 is a flowchart illustrating a specific example of processing of the importance determination system 30. Hereinafter, a specific example of the processing of the importance determination system 30 will be described with reference to FIG.
First, for each document stored in the document information storage unit 301, the word extraction unit 302 extracts individual words from the text included in the document (step S301). Next, the clustering unit 321 performs clustering so that the words extracted by the word extraction unit 302 are classified into one cluster for each word having the same meaning (step S321).

  Next, the word conversion unit 312 converts synonyms into representative words among the words extracted by the word extraction unit 302 (step S311). Next, the word appearance frequency acquisition unit 303 acquires the word appearance frequency of each word for each document based on the conversion result by the word conversion unit 312 (step S302). Next, the importance determination unit 304 determines, for each word i, a word appearance frequency that is an outlier from the set of word appearance frequencies in each document j (step S303). Then, the importance determination unit 304 determines that the word i is an important word of the document j based on the word i and the document j having the word appearance frequency determined as an outlier (step S304).

  The importance determination system 30 configured as described above can achieve the same effects as the importance determination system 10 in the first embodiment and the importance determination system 20 in the second embodiment.

  Further, in the importance determination system 30, since the representative words and synonyms are defined by the clustering unit 321, unlike the second embodiment, it is not necessary to prepare the conversion dictionary storage unit 211 in advance. Therefore, the cost and time required for preparing the conversion dictionary storage unit 211 can be saved.

The importance determination system 30 in the third embodiment may be modified and configured similarly to the importance determination system 10 in the first embodiment.
The clustering unit 321 may define a word located in the center among the words of each cluster as a representative word, and define a word other than the representative word as a synonym.

[Application example]
The important words determined by the first to third embodiments configured as described above may be used as follows.

  For example, when a word ranking is output by a text mining tool, if ranking is performed without considering the importance of each word, the top ranking is occupied by common words regardless of the topic. With respect to such a problem, it is possible to analyze the topic more accurately by generating a ranking based only on the important words determined by the first embodiment to the third embodiment.

  For example, when document summarization or document clustering is performed, not all words included in a document are processed, but only important words determined by the first to third embodiments are used. This makes it possible to create a summary that more accurately represents the features and to perform document clustering.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The present invention is applicable to a technique that performs processing based on the importance of words included in a document.

  10, 10a, 20, 30 ... importance determination system, 101 ... document information storage unit, 102, 102a, 202, 302 ... word extraction unit, 103, 103a, 203, 303 ... word appearance frequency acquisition unit, 104, 104a, 204, 304 ... Importance determination unit, 211 ... Conversion dictionary storage unit, 212, 312 ... Word conversion unit, 321 ... Clustering unit, 40 ... Web server

Claims (8)

Every single word, a word frequency is determined whether different or not have significant difference and said word occurrence frequency in other documents, the one based on the determination result indicating the occurrence frequency of each word in one document An importance determination system comprising: an importance determination unit that determines the importance of the word in a document. The importance determination system according to claim 1, further comprising a word appearance frequency acquisition unit that acquires the word appearance frequency for each document. When the importance determination unit satisfies a predetermined condition indicating that the word appearance frequency determined to have a significant difference in the set of word appearance frequencies of each document acquired for a word is high. the said words, determines that the pre-Symbol term frequency is important word in the acquired document, the importance determination system according to claim 1 or 2. When the importance determination unit satisfies a predetermined condition indicating that the word appearance frequency determined to be different with a significant difference in the set of word appearance frequencies of each document acquired for a word is low the importance determination system according to the word is determined to be important word in each document other than before SL document term frequency is obtained, in any one of claims 1-3. The importance determination system according to claim 3 or 4 , wherein the importance determination unit determines that only words having a predetermined part of speech are important words. For each document, a word conversion unit that converts a word having the same or similar meaning among the words appearing in the document into a single word,
The term frequency acquisition unit acquires the term frequency of each word converted by the word conversion section, importance determination system according to any one of claims 1-5. Every single word, a word frequency is determined whether different or not have significant difference and said word occurrence frequency in other documents, the one based on the determination result indicating the occurrence frequency of each word in one document the importance determination method having a significance determination steps, to determine the significance of the word in the document. Computer program for causing a computer to function as a key determination system according to any one of claims 1-6.

Images