JP5022319B2 - Text mining apparatus, method, program, and recording medium thereof - Google Patents

Description

  The present invention relates to a text mining technique that obtains certain knowledge and ideas by dividing unformatted text data into words and the like, and analyzing their appearance frequency and correlation using a data mining technique.

  When collecting multiple documents written in text, such as free description questionnaires and blog articles, etc. for products, etc., and trying to examine their subject trends, documents to find out what themes are and how much in them A word ranking of frequency (Document Frequency, also called DF for short) is used.

  The document frequency for a word is the number of documents that contain that word. For example, in FIG. 5, each of the three documents 1, 2, 3 includes the word “telephone”. Therefore, the document frequency of “telephone” is 3. The document frequency is a concept that does not consider the number of words included in the document. For example, each of the documents 1, 2, and 3 includes two “phones”, but the document frequency does not consider it. If a document includes a certain word, 1 is counted as the document frequency regardless of the number of the word included in the document.

  The word ranking of the document frequency is a word ranking in the descending order of the document frequency and ranking as necessary. FIG. 6 exemplifies word rankings for the words described in the right column of FIG. 5 and their document frequencies. From this word ranking, it can be seen that the number of documents in which the words “telephone”, “Yokosuka”, and “city hall” appear is 3, which is the largest.

  By the way, when there is a fixed part that appears almost in common in each document and has nothing to do with the subject (hereinafter referred to as the fixed part), the words contained in the fixed part occupy the top of the word ranking. Therefore, there may arise a problem that an accurate subject tendency cannot be grasped. FIG. 7 illustrates the fixed portion with an underline. The underlined part of document 1, “Yes, thank you for the call. Yokosuka City Hall citizen window.” “Thank you for the call. It can be said that there is a fixed part. Then, “phone”, “Yokosuka”, and “city hall” included in this standard part occupy the top of the word ranking of FIG. As described above, when words included in the fixed form part occupy the top of the word ranking, it is difficult to understand what themes have many subjects. Therefore, it is necessary to remove the influence of the fixed part from the word ranking.

Hereinafter, two methods for removing the influence of the fixed portion will be described.
The first method is a method of registering words that frequently appear in the fixed form part as stop words, and generating a word ranking by excluding the words registered as stop words (see, for example, Non-Patent Document 1). . An example of a stop word is shown in the upper right of FIG. In this example, “telephone”, “Yokosuka”, “city hall”, “citizen”, “window”, “thank you”, and “disrespect” that frequently appear in the fixed part of documents 1, 2, and 3 are registered as stop words. An example of word ranking generated by removing these stop words is shown in the lower right of FIG. This word ranking is obtained by removing these stop words from the word ranking of FIG.

  The second method is a method using a text segmentation technique such as a text tiling method (for example, see Non-Patent Document 2). Using the text tiling method, as illustrated on the left side of FIG. 9, the document is divided into a fixed part and a theme part, and the document frequency is calculated using the theme part to generate a word ranking. That is, when a word appears in the subject part of a document, 1 is counted as the document frequency. Words that appear only in the standard part of the document are not considered when calculating the document frequency.

A method for dividing a document by the text tiling method will be briefly described with reference to FIG. A reference point is defined in the document, and a window composed of a predetermined number of sentences is set to the left and right of the reference point. In the example of FIG. 10, the size of the window is 3, and two windows each having three sentences are provided on the left side and the right side of the reference point, respectively. The window on the left side of the reference point is called the left window, and the window on the right side of the reference point is called the right window. The left window appearance frequency, which is the frequency at which each word included in the left window appears in the left window, and the right window appearance frequency, which is the frequency at which each word included in the right window appears in the right window, are calculated. Then, the similarity between the left window appearance frequency and the right window appearance frequency is calculated. While sliding the reference point at regular intervals, the change in the similarity is observed, and the position where the similarity is lowered is found as a break in the document. This is because at the position where the topic changes, the left window and the right window are considered to have a low lexical relationship and a low similarity. Similarly, the fixed part and the theme part are considered to have low lexical relation to each other, and therefore, the fixed part and the theme part are divided at a position where the similarity is low.
Kita Kenji, Tsuda Kazuhiko, Sasabori Masatomi, “Information Retrieval Algorithm”, Kyoritsu Shuppan, p. 29-30 Maria A. Hearst. Multi-Paragraph Segmentation of Expository Text. 32nd Annual Meeting of the Association for Computation Linguistics. P.9-16. 1944

  In the first method, when a word registered as a stop word also appears in the subject part and constitutes the subject of the document, the subject cannot be found. For example, in the example of FIG. 8, “City Hall” constitutes a part of the theme “City Hall Opening Office Time” in the subject part of Document 2, and “Window” It constitutes a part of the theme “Reception hours”. However, “city hall” and “window” are registered as stop words because they appear frequently in the fixed part. Therefore, in this example, “city hall” and “window” are excluded from the word ranking, and the subject of the document 2 and the subject of the document 3 have failed to be grasped. Thus, in the first method, there is a problem that the words constituting the subject are excluded more than necessary, and the influence of the fixed part is not necessarily removed appropriately.

  In the second method, when the fixed part is composed of a smaller number of sentences than the window size, it is difficult to delimit the fixed part. On the other hand, there may be a method of trying to divide a smaller fixed portion by reducing the window size. However, if the window size is reduced, the number of words included in the window is reduced, the similarity between the left window appearance frequency and the right window appearance frequency is significantly reduced, and statistically reliable similarity can be calculated. become unable. In addition, since the window size of the left window and the window size of the right window cannot be the same at the beginning or end of the document, even if the fixed part is near the beginning or end of the document, the fixed part is not displayed. It is difficult to separate. As described above, in the second method, there is a problem that it cannot be said that the influence of the fixed portion is properly removed due to the difficulty in appropriately dividing the fixed portion.

  In view of the above problems, an object of the present invention is to provide a text mining device, a method, a program, and a recording medium thereof that can more appropriately remove the influence of the fixed portion.

  According to the text mining device recited in claim 1, an analysis target document storage unit that stores a plurality of analysis target documents, and a plurality of analysis target documents that are read from the analysis target document storage unit are divided into a plurality of words. A morpheme analysis unit, a word appearance frequency calculation unit for obtaining a frequency at which all or some of a plurality of divided words appear in an analysis target document (hereinafter referred to as word appearance frequency), and a certain analysis target The standard part of the document is a standard part that is not related to the subject of the analysis target document, and the standard part average word appearance frequency of a certain word is the estimated average frequency that the word appears in the standard part of each of the plurality of analysis target documents. As a standard part for subtracting the standard part average word appearance frequency of the word from the word appearance frequency obtained for the divided word, and obtaining the word appearance frequency after removing the standard part effect for the word The reverberation removing unit and the document frequency of a certain word as the number of a plurality of analysis target documents including the word, among the plurality of divided words, the word appearance frequency after removing the fixed part influence is higher than a predetermined frequency. Or a document frequency calculation unit for obtaining a document frequency for each of the above words.

By subtracting the fixed part average word appearance frequency from the word appearance frequency, the influence of the fixed part can be more appropriately removed.
Since the fixed part average word appearance frequency is an estimated average frequency that a word appears in the fixed part, even if this fixed part average word appearance frequency is subtracted from the word appearance frequency, the subject is configured unlike the first method. Do not exclude words that you want more than necessary. Unlike the second method, since the text tiling method is not used, there is no problem that the fixed part cannot be appropriately separated.

  An embodiment of the present invention will be described below. FIG. 1 shows an example of a functional configuration of a text mining device according to the present invention. FIG. 2 shows an example of a text mining method according to the present invention.

The analysis target document storage unit 10 stores a plurality of documents to be analyzed. A document to be analyzed is referred to as an analysis target document. The analysis target document is, for example, text data such as a transcript of a telephone response, a free description questionnaire for a product, or a blog article.
Some analysis target documents are selected from a plurality of analysis target documents stored in the analysis target document storage unit 10 (step S1). Then, a fixed part is extracted from each of these selected analysis target documents and stored in the fixed part storage unit 20.

  In the present invention, a fixed part that is not related to the subject of a certain analysis target document is set as a fixed part of the analysis target document. When a single analysis target document includes a plurality of fixed part portions, all of the plurality of fixed form portions are referred to as a fixed portion of the analysis target document. For example, in document 1 of FIG. 7, the part of the first fixed part "Yes, thank you for the call. It is the Yokosuka City Hall citizen window." The part of the second fixed part "Thank you for the call. There is. Therefore, in the case of the standard part of this document 1, it means both “Yes, thank you for the call. I am Yokosuka City Hall Citizens.” And “Thank you for the call. It does not mean either one.

  The analysis target document is selected based on the word frequency of each word in the standard part of the selected analysis target document and the word frequency of each word in the standard part of the plurality of analysis target documents stored in the analysis target document storage unit 10. It is desirable to make sure that there is no significant difference between the two. This is to improve the accuracy of estimating the fixed portion average word appearance frequency, which will be described later. If it does in that way, selection of an analysis object document may be performed at random using a computer etc., and a person may perform arbitrarily. For example, about 50 analysis target documents are selected.

  The standard part is extracted from the analysis target document by a method that enables the standard part to be accurately extracted, such as manual work. In the present invention, it is not necessary to extract the standard part for all the analysis target documents, and it is sufficient to perform the selection for the selected analysis target document, that is, a part of the analysis target documents. Therefore, in order to take out the fixed portion, a method with high accuracy can be used even if it takes a relatively long time such as manual work. Of course, if there is a method that can accurately extract the fixed portion using a computer, the fixed portion may be mechanically extracted by the method using a computer.

  The fixed form morpheme analyzing unit 30 reads each fixed part from the fixed part storage unit 20, and divides each fixed part into a plurality of words (step S2). The divided words are sent to the fixed part word appearance frequency calculation unit 40.

For dividing into words, a well-known morphological analysis method can be used. For example, there is a longest matching method for selecting a candidate including the longest number of characters from among morpheme analysis result candidates, a method for selecting a candidate with the smallest number of phrases, a method for selecting a candidate with a long independent word, and a predetermined rule. For example, a method for performing morphological analysis and a method for selecting a statistically plausible candidate using a hidden Markov model are used (for example, see Reference 1).
[Reference 1] Shun Ishizaki, “Natural Language Processing”, Shosodo, P. 27-29

  The fixed part word appearance frequency calculation unit 40 obtains the frequency at which the divided words appear in the fixed part (step S3). That is, for each fixed part, the number of each word that appears in the fixed part appears in the fixed part. The obtained frequency is sent to the fixed part average word appearance frequency calculation unit 50.

  For example, in the standard part of the document 1 in FIG. 3, “telephone” appears three times, so the frequency of “telephone” appears as three. Similarly, the frequency at which each word such as “Thank you”, “Yokosuka”, “City Hall”, etc. appearing in the standard part of the document 1 appears. Similarly, for document 2, the frequency at which each word appearing in document 2 appears in the standard part of document 2 is obtained, and for document 3, the frequency at which each word appearing in document 3 appears in the standard part of document 2 is also calculated. Ask. FIG. 3 shows an example in which the selected analysis target documents are three documents 1, 2, and 3.

  The fixed part average word appearance frequency calculation unit 50 calculates the fixed part average word appearance frequency by adding the obtained frequency for each word and then dividing the result by the number of selected analysis target documents (step S4). The fixed part average word appearance frequency for a word is an estimated average frequency at which the word appears in the fixed part of the analysis target document. In other words, “adding the obtained frequency for each word” means “adding the frequency at which the word appears in each fixed part”. Since the number of selected analysis target documents is equal to the number of fixed form parts, it may be considered that the selected analysis target document is divided by the number of fixed part parts instead of the selected analysis target document number. The determined fixed part average word appearance frequency is stored in the fixed part average word appearance frequency storage unit 51.

  For example, in the example of FIG. 3, “telephone” appears three times in the standard part of the document 1, appears once in the standard part of the document 2, and appears in the standard part of the document 3. The number of times to do is two times. The standard part average word appearance frequency calculation unit 50 adds these times (3 times + 1 time + 2 times = 6 times), and the number of times that “telephone” appears in the whole standard part of the selected analysis target document. (6 times) is requested. Then, by dividing this added value (six times) by 3, which is the number of selected documents to be analyzed, the fixed portion average word appearance frequency (twice) of “phone” is obtained. For other words such as “Thank you” and “Yokosuka”, the standard part average word appearance frequency is similarly obtained.

  Thus, in this example, the fixed part average word appearance frequency is estimated not from all the analysis target documents but from a selected part of the analysis target documents. The calculation of the standard part average word appearance frequency may be performed in advance, or may be performed in parallel with the processing from step S5 to step S8 described later.

  The morphological analysis unit 60 divides each of the plurality of analysis target documents read from the analysis target document storage unit 10 into a plurality of words (step S5). The divided words are sent to the word appearance frequency calculation unit 70. The division into words can be performed by using a well-known morpheme analysis method in the same manner as the fixed form morpheme analysis unit 30.

  The word appearance frequency calculation unit 70 obtains the frequency at which each word divided by the morpheme analysis unit 60 appears in the read analysis target document (step S6). That is, for each analysis target document, the number of times each word included in the analysis target document appears in the analysis target document is counted. This obtained frequency is expressed as a word appearance frequency (Term Frequency, also abbreviated as TF). That is, the word appearance frequency for a certain word and a certain analysis target document means the number of times that the word appears in the analysis target document. The obtained frequency, that is, the word appearance frequency is sent to the fixed part influence removing unit 80.

  For example, in FIG. 4, since “telephone” appears twice in the document, the word appearance frequency of “phone” is twice, and “Yokosuka” appears once in the document, so the word frequency of “Yokosuka” is 1. Times.

  The fixed part influence removing unit 80 subtracts the fixed part average word appearance frequency of the word from the frequency obtained for the divided word, that is, the word appearance frequency of the divided word, and determines the fixed part influence on the word. The word appearance frequency after removal is obtained (step S7). That is, for each divided word, the standard part average word appearance frequency of the divided word is subtracted from the word appearance frequency of the divided word, and the subtraction result is used as the word appearance frequency after removing the fixed part influence of the word. . The obtained word appearance frequency after removing the fixed portion influence is sent to the document frequency calculation unit 90.

  In addition, when the fixed part average word appearance frequency is pre-calculated, the fixed part influence removing unit 80 appropriately reads the corresponding fixed part average word appearance frequency from the fixed part average word appearance frequency storage unit 51. When the calculation of the standard part average word appearance frequency is performed in parallel with the processing from step S5 to step S8, the corresponding standard part average word appearance frequency calculated by the standard part average word appearance frequency calculation unit 50 is influenced by the standard part. It may be sent directly to the removing unit 80.

  The document frequency calculation unit 90 uses the document frequency of a certain word as the number of documents to be analyzed including the word, and among the plurality of divided words, the appearance frequency of the word after removing the fixed part influence is higher than a predetermined frequency. The document frequency for each word is obtained (step S8). In addition, you may obtain | require the sentence frequency about each of the word more than the frequency with which the word appearance frequency after a fixed part influence removal is predetermined among several divided words.

  Thus, by subtracting the fixed part average word appearance frequency from the word appearance frequency, the influence of the fixed part can be removed more appropriately. That is, since the standard part average word appearance frequency is an estimated average frequency at which a word appears in the standard part, even if this standard part average word appearance frequency is subtracted from the word appearance frequency, the first part described in the background art column is used. Unlike the above method, the words constituting the subject are not excluded more than necessary. In addition, unlike the second method described in the background art section, the text tiling method is not used, and therefore, there is no problem that the fixed part cannot be appropriately separated.

[Modifications, etc.]
When the fixed form morphological analysis unit 30 divides the fixed part and outputs a word, depending on the division method, a word such as a particle or a conjunction that does not constitute a subject may be output. In this case, the fixed part word appearance frequency calculation unit 40 obtains the appearance frequency for a word that constitutes a subject matter such as a noun or a verb, and obtains the appearance frequency for a word that does not constitute a subject matter such as a particle or a conjunction. It does not have to be. That is, the fixed part word appearance frequency calculation unit 40 may obtain the appearance frequencies of some words instead of all the words divided by the fixed part morpheme analysis unit 30.

  Similarly, when the morphological analysis unit 60 divides the analysis target document and outputs a word, depending on the division method, a word such as a particle or conjunction that does not constitute a subject may be output. In this case, the word appearance frequency calculation unit 70 obtains the word appearance frequency for a word that constitutes a subject such as a noun or a verb, and obtains the word appearance frequency for a word that does not constitute a subject such as a particle or a conjunction. It does not have to be. That is, the word appearance frequency calculation unit 70 may obtain the word appearance frequency for some words instead of all the words divided by the morphological analysis unit 60.

  In addition, as shown by a dotted line in FIG. 1, a word rearrangement unit 100 that outputs words in descending order of document frequency may be provided. Thereby, a word ranking can be produced | generated and it becomes easy to grasp | ascertain a theme tendency. The word rearrangement unit 100 may rank the word according to the word document frequency. Moreover, the word rearrangement part 100 does not need to output all the rearranged words, and may output only a part of the rearranged words. For example, only words having a document frequency equal to or higher than a predetermined document frequency, or only words having a rank higher than a predetermined rank may be output. This makes it easier to grasp the theme trend.

  In the above example, the standard part average word appearance frequency is calculated from a part of the analysis target document. However, the standard part average word appearance frequency may be calculated from a document that is not the analysis target document in the same manner as described above. For example, if the document was an analysis target in the past but is not an analysis target this time, and there is no significant change in the word appearance frequency of each word in the standard part, the document that was the analysis target in the past The standard part average word appearance frequency may be calculated from the above. Similarly, the fixed portion average word appearance frequency may be calculated from a document including an analysis target document and a document that is not an analysis target in the same manner as described above.

  When the above configuration is realized by a computer, the processing contents of the functions of each unit of the text mining device are described by a program. By executing this program on a computer, the functions of the above-described units are realized on the computer.

  That is, when the CPU sequentially reads and executes the program, the fixed part morpheme analyzer 30, the fixed part word appearance frequency calculator 40, the fixed part average word appearance frequency calculator 50, the morpheme analyzer 60, the word appearance frequency calculator 70, the fixed part influence removing unit 80, the document frequency calculating unit 90, and the word rearranging unit 100 are realized. In addition, the auxiliary storage device or the memory functions as the analysis target document storage unit 10, the fixed part storage unit 20, and the fixed part average word appearance frequency storage unit 51.

  The CPU functioning as each unit of the text mining device processes the data read from the memory or the auxiliary storage device, and stores the processed data in the memory or the auxiliary storage device. That is, data is exchanged between the units of the sound reproducing device via the memory or the auxiliary storage device.

The program describing the processing contents can be recorded on a computer-readable recording medium. The computer-readable recording medium may be any medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, or a semiconductor memory. Specifically, for example, the magnetic recording device may be a hard disk device or a flexible Discs, magnetic tapes, etc. as optical discs, DVD (Digital Versatile Disc), DVD-RAM (Random Access Memory), CD-ROM (Compact Disc Read Only Memory), CD
-R (Recordable) / RW (ReWritable), etc., MO (Magneto-Optical disc), etc. as a magneto-optical recording medium, EEP-ROM (Electronically Erasable and Programmable-Read Only Memory), etc. as a semiconductor memory it can.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  As an execution form different from the above-described embodiment, the computer may read the program directly from the portable recording medium and execute processing according to the program. Each time is transferred, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to a computer but has a property that is based on computer processing).

In this embodiment, the present apparatus is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.
In addition, the various processes described above are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary.
Needless to say, other modifications are possible without departing from the spirit of the present invention.

The figure which shows the example of a function structure of a text mining apparatus. The figure which shows the example of the text mining method. The figure for demonstrating a fixed part average word appearance frequency. The figure for demonstrating word appearance frequency. The figure for demonstrating document frequency. The figure for demonstrating word ranking. The figure for demonstrating a fixed form part and a theme part. The figure for demonstrating the conventional 1st method of removing the influence of a fixed form part. The figure for demonstrating the conventional 1st method of removing the influence of a fixed form part. The figure for demonstrating the text tiling method.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Analysis object document memory | storage part 20 Fixed form part memory | storage part 30 Fixed form part morpheme analysis part 40 Fixed form part Word appearance frequency calculation part 50 Fixed form part average word appearance frequency calculation part 51 Fixed form part average word appearance frequency storage part 60 Morphological analysis part 70 Word appearance Frequency calculation unit 80 Fixed part influence removal unit 90 Document frequency calculation unit 100 Word rearrangement unit

Claims (6)

An analysis target document storage unit for storing a plurality of analysis target documents;
A morpheme analyzer that divides each of the plurality of analysis target documents read from the analysis target document storage unit into a plurality of words;
A word appearance frequency calculation unit for obtaining a frequency at which all or a part of the plurality of divided words appear in the read analysis target document (hereinafter referred to as word appearance frequency);
A fixed part of a certain analysis target document is a fixed part that is not related to the subject of the analysis target document, and the fixed part average word appearance frequency of a certain word appears in the fixed part of each of the plurality of analysis target documents. As a presumed average frequency, the fixed part influence is obtained by subtracting the fixed part average word appearance frequency of the word from the word appearance frequency obtained for the divided word and obtaining a word appearance frequency after removing the fixed part influence on the word. A removal section;
The document frequency of a certain word is the number of the plurality of documents to be analyzed including the word, and among the plurality of divided words, the word appearance frequency after removing the fixed portion influence is higher than or higher than a predetermined frequency. A document frequency calculator that calculates the document frequency for each word,
Text mining device including The text mining device according to claim 1,
A fixed form morpheme analysis unit that divides each fixed part of a part of analysis target documents selected from the plurality of analysis target documents into a plurality of words;
A fixed part word appearance frequency calculating unit for obtaining a frequency at which each of all or a part of each of the plurality of divided words appears in the fixed part;
After adding the obtained frequency for each word, dividing by the number of the part of the analysis target document, a fixed part average word appearance frequency calculation unit for obtaining a fixed part average word appearance frequency of the word,
A text mining device further comprising: The text mining device according to claim 1 or 2,
A word rearrangement unit that outputs words in descending order of the document frequency;
A text mining device characterized by that. The analysis target document storage unit stores a plurality of analysis target documents,
A morphological analysis step of dividing each of the plurality of analysis target documents read from the analysis target document storage unit into a plurality of words;
A word appearance frequency calculation step for obtaining a frequency at which all or a part of the plurality of divided words appear in the read analysis target document (hereinafter referred to as word appearance frequency);
A fixed part of a certain analysis target document is a fixed part that is not related to the subject of the analysis target document, and the fixed part average word appearance frequency of a certain word appears in the fixed part of each of the plurality of analysis target documents. As a presumed average frequency, the fixed part influence is obtained by subtracting the fixed part average word appearance frequency of the word from the word appearance frequency obtained for the divided word and obtaining a word appearance frequency after removing the fixed part influence on the word. A removal step;
The document frequency of a certain word is the number of the plurality of documents to be analyzed including the word, and among the plurality of divided words, the word appearance frequency after removing the fixed portion influence is higher than or higher than a predetermined frequency. A document frequency calculation step for obtaining a document frequency for each word,
Text mining method including.   The program for functioning a computer as each part of the text mining device in any one of Claim 1 to 3.   A computer-readable recording medium storing the program according to claim 5.

Images