JP5499546B2 - Important word extraction method, apparatus, program, recording medium - Google Patents

Description

The present invention relates to a method, an apparatus, a program, and a recording medium for extracting an important word that represents a feature of a document.
The present invention is particularly useful when a word obtained by concatenating morphemes extracted from a document created in a natural language is extracted as an important word representing the characteristics of the document.

With the spread of the Internet and computers, enormous amounts of electronic documents are accumulating, so in order to utilize these documents, it is becoming increasingly important to extract words (= important words) that express the characteristics of the documents. ing.
By using key words as clues, a document user (hereinafter, an analyst) can efficiently classify and search for documents.
Here, when the content of a sentence is related to a specialized matter, a combination of a plurality of words (= compound word) often represents a complex concept, so it is possible to use a compound word as an important word of a document. It is valid.

Therefore, a method for extracting compound words from a document and a method for determining important words will be described.
<< A. Compound word extraction >>
A method of extracting compound words is a compound word in which a document created in a natural language is divided into morphemes by a morpheme analysis program, and then a combination of word parts of speech and attributes is determined for the divided morphemes. By applying the rules, a compound word that connects multiple morphemes is created.
For example, Non-Patent Document 1 discloses a technique for dividing Japanese sentences into word units (= morphemes) and adding parts of speech.
Patent Document 1 discloses a technique for creating a compound word having morphological analysis information such as part of speech or reading by applying a compound word rule to a morpheme of a document.
Here, morphological analysis is a basic technology of natural language processing that divides a sentence written in natural language into morpheme strings, which are the smallest units meaningful in the language, and discriminates each part of speech. Is widely used.
<< B. Determination of important words >>
The method of selecting an important word from compound words is to calculate the importance of the compound word using important word knowledge such as the document position where the important word appears and the frequency of occurrence of the compound word. To decide.
For example, Patent Document 2 discloses a technique for determining an important word by using important word knowledge such as a place where the important word is allocated and a font size of the important word.

JP 2008-287406 A (paragraph 0081-paragraph 0088, FIG. 6) JP 2006-309347 A (paragraph 0059-paragraph 0062)

Morphological analysis system tea bowl [Search June 20, 2008] Internet URL: http://chasen-legacy.sourceforge.jp/

As described above, << A. In compound word extraction, when compound words are extracted from a document using compound word rules, compound words that do not comply with the compound word rules cannot be extracted. Cannot be set as
In addition, << B. In determining important words, in order to determine important words, it is necessary to analyze important words in advance and acquire important word knowledge.

The present invention has been made in order to solve the above-described points, and an object of the present invention is to set a compound word as an important word that represents a document feature without using compound word rules or important word knowledge. It is to provide a method, an apparatus, a program, and a recording medium.

This invention solves the said subject by the means as described in each following aspect.
That is, the first invention of the present invention is a key word extraction performed by a key word extraction device comprising a document input means, a compound word extraction means, a similar compound word creation means, an appearance frequency counting means, a key word setting presentation means, and a synonym database. A method,
A document input step in which the document input means acquires document data; and
A compound word extracting step in which the compound word extracting means extracts a compound word composed of three or more morphemes whose type of part of speech is not limited from document data;
A similar compound word creating step in which the similar compound word creating means creates a similar compound word by replacing one morpheme constituting the compound word with a synonym obtained by searching the synonym database;
An appearance frequency counting step in which the appearance frequency counting means counts the appearance frequency in the document data using a compound word and a similar compound word;
The important word setting presentation unit, if the frequency of appearance of the compound word similar compound word satisfies the important word setting conditions, the compound word, by setting the key word of the document data, and the key word setting presenting step of presenting ,
It is a key word extraction method characterized by being performed in the procedure including.

In this way, compound words can be extracted from a document without using compound word rules. Here, the part of speech of the compound word is not limited to a noun or a verb, and may be any part of speech. The part of speech of the morpheme constituting the compound word may be any part of speech.
Whether or not the compound word is an important word (= a word representing the feature of the document) by using the occurrence frequency of the similar compound word (= a word obtained by replacing a part of the compound word with a synonym) using the important word determination condition. It is possible to determine easily.

A second invention of the present invention is the key word extraction method according to the first invention,
The important word setting conditions are:
The appearance frequency of the compound word is 2 and the appearance frequency of the similar compound word is 0.
Is an important word extraction method characterized by comprising:

A third invention of the present invention provides a synonym storage area for storing a synonym database,
A storage unit having
A document input means for obtaining document data;
A compound word extracting means for extracting a compound word composed of three or more morphemes from which the type of part of speech is not limited ;
A synonym acquisition means for searching a synonym database and acquiring a synonym using one morpheme constituting a compound word as a search key;
A similar compound word creating means for creating a similar compound word by replacing the morpheme used for the search key with the obtained synonym;
Appearance frequency counting means for calculating appearance frequency in document data using compound words and similar compound words,
If the frequency of occurrence of compound words similar compound word satisfies the important word setting conditions, the compound word, by setting the key word of the document data, and the important word setting presenting means for presenting,
It is an important word extracting device characterized by comprising.

A fourth invention of the present invention is the key word extraction device according to the third invention,
The important word setting conditions are:
The appearance frequency of the compound word is 2 times or more, and the appearance frequency of the similar compound word is 0 times.
This is an important word extraction device.

A fifth invention of the present invention is a computer program that causes a computer to operate as the important word extracting device according to the third or fourth invention by being incorporated in the computer.

A sixth invention of the present invention is a computer-readable recording medium on which the computer program according to the fifth invention is recorded.

According to the present invention,
(1) It is possible to extract a compound word obtained by connecting a plurality of morphemes from a document.
(2) It is possible to easily determine whether or not a compound word is a word representing a feature of a document by measuring the appearance frequency of a similar compound word (= a word obtained by replacing a part of a compound word with a synonym). Is possible.
Therefore, according to the present invention, it is possible to easily identify whether or not a compound word is an important word that expresses the characteristics of a document by determining the appearance frequency of the compound word using a compound word and a synonym. effective.

It is a figure explaining the outline | summary of the compound word extraction system 1 by embodiment of this invention. Example 1 Is a diagram illustrating the high-level steps of the working and processing of the double focus word extraction system 1. It is a diagram illustrating a display example of document data. Is a diagram illustrating the details of the kind word database. It is a figure explaining the meaning and synonym of a word . It is a diagram illustrating a detailed procedure of a double focus word extraction processing. It is a diagram illustrating a multi-case words. It is a diagram illustrating a detailed procedure of the kind similar compound word generation process. A detailed block diagram of the important word extracting unit 100.

Hereinafter, embodiments of the present invention will be described in more detail with reference to the drawings.

FIG. 1 is a diagram illustrating an overview of a compound word extraction system 1 according to an embodiment of the present invention.
The compound word extraction system 1 is configured by connecting a keyword extraction device 100 and a web server device 300 via a network.

The keyword extraction device 100 is a personal computer equipped with an existing morpheme analysis program, in which a user agent (for example, a web browser, a crawler, a mail user agent, etc.) and a dedicated program described later are installed.

The web server device 300 is a server computer provided with an existing server program (for example, a web server program that provides a web page to a user agent).
The document data provided by the web server device 300 includes web pages (including blogs and electronic bulletin boards).
The web server device 300 may be provided with a mail server program and provide net news.

FIG. 2 illustrates a rough procedure of the work and processing of the compound word extraction system 1.
(1) The analyst designates a document (such as a web page of the web server device 300) on the Internet (for example, URL = Uniform Resource Locator is input).
(2) The keyword extraction device 100 acquires the document data 391 specified by the URL from the web server device 300.
Here, the keyword extraction device 100 may be configured to acquire the document data 391 from an external recording medium (CD, DVD, semiconductor memory).
(3) The keyword extraction device 100 extracts a compound word composed of a plurality of morphemes from the document data 391.
Here, the type of part of speech of the compound word is not limited, and any part of speech may be used. Further, it is not necessary to associate morphological analysis information such as part of speech or reading with a compound word. In addition, it is not necessary to associate morpheme analysis information such as part of speech or reading with a plurality of morphemes constituting a compound word. (Details will be described later)
(4) The keyword extraction device 100 uses the synonym database to obtain a synonym of a morpheme constituting the compound word, and replaces the corresponding morpheme of the compound word with the synonym to generate a similar compound word.
(5) The keyword extraction device 100 aggregates the appearance frequencies of compound words and similar compound words in a document, and the appearance frequency is an important word setting condition (for example, the appearance frequency of a compound word is two or more, If the appearance frequency of the similar compound word is zero), the compound word is set as an important word of the document and presented.
(6) The analyst arrives at the necessary document using the key words as clues.

FIG. 3 is a diagram for explaining important words of a document.

FIG. 3A shows a display example of document data 391 document in which a compound word is determined to be an important word.
The displayed document data 391 includes the compound word “I am a cat”.
Here, a similar compound word created using the synonym of the morpheme “cat” (for example, the similar compound word “I am a dog” created using the synonym “dog” of the morpheme “cat”) is included in this document. Indicates that it is not included.

The meaning of the displayed document data 391 is that the compound word “I am a cat” has an appearance frequency of a predetermined number of times (for example, two times or more), and a similar compound word (for example, “I am a dog” “)” Is 0 times, so it matches the important word setting condition, and the compound word “I am a cat” indicates that it is an important word.

FIG. 3B is a display example of document data 391 including compound words and similar compound words.
The displayed document data 391 includes a compound word “Japanese food self-sufficiency ratio” and a similar compound word “American food self-sufficiency ratio” and “Korea food self-sufficiency ratio” corresponding to this compound word. That is, since the synonyms of the morpheme “Japan” are “America” and “Korea”, the similar compound words generated from the compound word “Japan's food self-sufficiency rate” are compound words “US food self-sufficiency rate” and “ Corresponds to the “Korean food self-sufficiency ratio”.

The meaning of the displayed document data 391 is that the compound word “Japanese food self-sufficiency rate” has a predetermined frequency (for example, two or more times) and the similar compound words “US food self-sufficiency rate” and “ Since the frequency of “Korean food self-sufficiency” appears at least once, the compound word “Japanese food self-sufficiency” indicates that it is not an important word.

Here, it is examined whether or not the compound word “food self-sufficiency ratio” is an important word. The appearance frequency of the compound word “food self-sufficiency” (two times or more) and the appearance frequency of similar compound words (for example, “food self-sufficiency rate” and “food procurement rate”) (zero times) are important word setting conditions Meet. Therefore, the compound word “food self-sufficiency ratio” is set as an important word of this document.

The part of speech of the compound word described in the above description is a noun or a verb, but is not limited to this, and may be an adjective or an adverb. Moreover, although the part of speech of the morpheme which comprises a compound word was a noun, a verb, a particle, etc., it is not limited to this, An adjective, an adverb, etc. may be sufficient.
In addition, in order to extract a compound word, it turns out that the compound word rule of the technique of patent document 1 and the important word knowledge of the technique of patent document 2 are not required.

FIG. 4 explains the details of the synonym database.
The synonym database 191 is a database that searches for synonyms having the same meaning as one morpheme.
The synonym database 191 includes a meaning table 1911 and a synonym table 1912.
The meaning table 1911 is tabular data in which a word 1914 (primary key) and a meaning word 1913 are associated with each other.
The synonym table 1912 is tabular data in which a meaning word 1917 (primary key) and a set 1915 of synonyms 1916 are associated with each other.

As an explanation of the synonym database 191, a relational database has been taken as an example, but a network database may be used.
As the synonym database 191, for example, “Japanese vocabulary system” (Iwanami Shoten) supervised by NTT Communication Science Laboratories may be used.

Here, with reference to FIG. 5, for a set 191 5 word 1913 synonymous 1916 meaning explained.

The meaning word 1913 is a systemized word system (for example, a word system based on a tree structure).
In the tree structure of the word 1913 representing the meaning illustrated in FIG. 5, for example, the word 1913 “living” and “substance” representing the meaning are associated with the word 1913 “noun” representing the meaning, Systematized by structure. Further, below the meaning word 1913 “organism”, for example, the meaning word 1913 “animal” and “plant” are associated and organized in a tree structure. Further, below the meaning word 1913 “animal”, for example, the meaning words 1913 “mammals”, “reptiles”, and “birds” are associated with each other and organized in a tree structure.

The intention of the tree structure of the words representing these exemplified meanings indicates that “subject” 1913 and “substance” 1913 are subordinate meanings of “noun” 1913. Further, it is shown that “animal” 1913 and “plant” 1913 are subordinate to “living organism” 1913. The subordinate meaning of “animal” 1913 is “mammal” 1913, “reptile” 1913, and “bird” 1913.

Next, a description will be given of the set 191 5 of the Thesaurus.
The set of synonyms 191 5 illustrated in FIG. 5 is, for example, “Dog” 191 6 and “Cat” 191 6 “Cow” 191 6 . This set of synonyms 1915 is associated with the word 1913 “mammal” representing meaning.

The intent of these illustrated set of synonyms 1915 is that the meaning of a certain synonym 191 6 “cat” is, for example, “mammal” 1913, so other synonyms 191 6 associated with this “mammal” 1913 are , is that "dog" 191 6 and the "cow" 191 6, and the like.

FIG. 6 is a diagram for explaining the detailed procedure of the compound word extraction process.
(3) Compound Word Extraction Processing (3-1) The keyword extraction device 100 performs morphological analysis on the document data 391 and divides it into morphemes.
(3-2) The keyword extraction device 100 continuously extracts N morphemes from these divided morpheme strings and connects them to form a compound word. (N = 2, 3,..., N, where n is a predetermined integer value) Accordingly, the morphemes constituting the compound word are mechanically extracted from the sequence of morphemes. Any type.

Here, the compound word will be described with reference to FIG.
FIG. 7A illustrates a phrase as a part of the document data 391.
As an example of the phrase of the document data 391, a novel title “I am a cat” is shown.
FIG. 7B illustrates divided morphemes.
As an example of a divided morpheme, “I am / is / cat / is / is” is shown. / Is a symbol indicating a morpheme break.
FIG. 7C illustrates a compound word in which N morphemes are continuously extracted and connected.
When N = 2, as examples of compound words, “Senior is” “is a cat” “is a cat” “is” is shown.
When N = 3, examples of compound words are “Senior is a cat”, “is a cat”, and “is a cat”.
When N = 4, “I am a cat” and “I am a cat” are shown as examples of compound words.
When N = 5, “I am a cat” is shown as an example of a compound word.

FIG. 8 is a diagram for explaining the detailed procedure of the similar compound word generation process.
(4) Similar Compound Word Generation Processing (4-1) The keyword extraction device 100 searches the synonym database using one morpheme constituting the compound word as a search key, and selects a synonym corresponding to the morpheme of the search key. get.
The synonym database 191 obtains a semantic code corresponding to the morpheme of the search key by referring to the semantic table using the morpheme of the search key, and then uses the acquired semantic code to generate a synonym table. Referring to, a synonym corresponding to the semantic code is acquired.
(4-2) The keyword extraction device 100 uses the acquired synonym to replace the corresponding morpheme in the morpheme string with this synonym and generates a similar compound word.

FIG. 9 is a detailed configuration diagram of the keyword extraction device 100.
The keyword extraction device 100 includes a CPU 101, a display unit 103, an input unit 102, a network communication unit 104, a storage unit 109, and a dedicated program.
The CPU 101, the display unit 103, the input unit 102, the network communication unit 104, and the storage unit 109 are connected by a BUS199.

The CPU 101 is a central processing unit.
The display unit 103 is a liquid crystal display device or an organic EL display device.
The input unit 102 is a mouse or a keyboard. The input unit 102 may be a keyboard, a mouse, or a soft keyboard (= touch panel) displayed on the display unit 103.
The network communication unit 104 is a LAN adapter.

The storage unit 109 is a semiconductor memory or a magnetic memory.
The storage unit 109 includes a synonym database storage area 1091, a document storage area 1092, and a morpheme storage area 1093, and stores an operating system 185, a morpheme analysis program 180, a web browser 381, and a dedicated program.

The synonym database storage area 1091 stores a synonym database 191.
The document storage area 1092 stores document data 391.
The morpheme storage area 1093 stores morpheme string data 195.

The storage unit 109 may store user agents such as crawlers and mail user agents.

The operating system 185 manages and controls the hardware of the keyword extraction device 100 (for example, the CPU 101, the input unit 102, the display unit 103, the network communication unit 104, the storage unit 109, the BUS 199, etc.) It is basic software that provides a service that can use these hardware for application software (for example, a dedicated program).
The morpheme analysis program 180 is an existing program that outputs morpheme analysis data in which disassembled morphemes are associated with parts of speech and reading information when text data of a predetermined format is given as input data.
The web browser 381 is an existing program, and is a program for browsing a web page (for example, document data) of an internet website.

In addition, a document input unit 110, a compound word extraction unit 120, a synonym extraction unit 135, a similar compound word creation unit 130, an appearance frequency totaling unit 140, and an important word setting presentation unit 150 are provided. Each of these means is realized by each dedicated program and functions by the CPU 101 interpreting and executing the dedicated program.

The document input unit 110 acquires document data 391 from the web server device 300.

The compound word extraction unit 120 calls the morpheme analysis program 180, passes the document data 391 as input data, receives the morpheme analysis data output from the morpheme analysis program 180, and uses the morpheme of the morpheme analysis data to generate a compound word. create.
The details of the processing compound words extraction means 120, described in the section describing the detailed flow of the (3) "compound word extraction" in Fig.

The synonym extracting unit 135 searches the synonym database 191 using one morpheme constituting the compound word as a search key, and acquires a synonym.
The details of the processing synonym extraction means 135, described in the section describing the detailed flow of the (4) "similar compound word generation processing" in FIG. 8.

The similar compound word creation means 130 replaces one morpheme constituting the original compound word with the obtained synonym and creates a similar compound word.

The appearance frequency totaling unit 140 totals the appearance frequencies in the document data 391 using compound words and similar compound words.

When the appearance frequency of the compound word and the similar compound word satisfies a predetermined condition, the keyword setting presentation unit 150 sets the compound word as the keyword of the document and presents it.

DESCRIPTION OF SYMBOLS 1 Compound word extraction system 100 Important word extraction apparatus 110 Document input means 120 Compound word extraction means 130 Similar compound word creation means 135 Synonym extraction means 140 Appearance frequency totaling means 150 Important word setting presentation means 180 Morphological analysis program 191 Synonym database 300 Document server Device 391 Document data

Claims (6)

A keyword extraction method performed by a keyword extraction device comprising a document input means, a compound word extraction means, a similar compound word creation means, an appearance frequency counting means, a keyword setting presentation means, and a synonym database ,
A document input step in which the document input means acquires document data; and
A compound word extracting step in which the compound word extracting means extracts a compound word composed of three or more morphemes whose type of part of speech is not limited from document data;
A similar compound word creating step in which the similar compound word creating means creates a similar compound word by replacing one morpheme constituting the compound word with a synonym obtained by searching the synonym database;
An appearance frequency counting step in which the appearance frequency counting means counts the appearance frequency in the document data using a compound word and a similar compound word;
The important word setting presentation unit, if the frequency of appearance of the compound word similar compound word satisfies the important word setting conditions, the compound word, by setting the key word of the document data, and the key word setting presenting step of presenting ,
A key word extraction method characterized by being performed in a procedure including The key word extraction method according to claim 1,
The important word setting conditions are:
The appearance frequency of the compound word is 2 times or more, and the appearance frequency of the similar compound word is 0 times.
An important word extraction method characterized by this. A synonym storage area for storing a synonym database,
A storage unit having
A document input means for obtaining document data;
A compound word extracting means for extracting a compound word composed of three or more morphemes from which the type of part of speech is not limited;
A synonym acquisition means for searching a synonym database and acquiring a synonym using one morpheme constituting a compound word as a search key;
A similar compound word creating means for creating a similar compound word by replacing the morpheme used for the search key with the obtained synonym;
Appearance frequency counting means for calculating appearance frequency in document data using compound words and similar compound words,
If the appearance frequency of the compound word and the similar compound word satisfies the important word setting condition, the important word setting presenting means for setting and presenting the compound word as the important word of the document data; and
An important word extracting device comprising: The key word extraction device according to claim 3,
The important word setting conditions are:
The appearance frequency of the compound word is 2 times or more, and the appearance frequency of the similar compound word is 0 times.
An important word extraction device characterized by that.   The computer program which makes a computer operate | move as an important word extraction apparatus of Claim 3 or Claim 4 by incorporating in a computer.   A computer-readable recording medium on which the computer program according to claim 5 is recorded.

Images