JP2011165087A - Important word extraction device, important word extraction method and important word extraction program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an important word extraction technology for selecting an important word on the basis of other than appearance frequencies. <P>SOLUTION: Important word candidates which are collocations of one or more nouns or proper nouns are extracted from an input text. Action expression which appears in describing an action of an input text creator is extracted as a feature word from the input text. One or more features showing the property of the important word candidates are extracted with respect to each important word candidate. An importance score is calculated using the features on the basis of a classification rule determined beforehand by machine learning, and the important word is determined from the importance score. In extracting the features, modification structure information showing whether the features are the important word candidates concerning the action expression is added to the important word candidates as the features. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique for extracting a particularly important word from words appearing in a text.

  Non-patent document 1 is known as a conventional technique for extracting a particularly important word from words appearing in a text. In Non-Patent Document 1, the appearance frequency of words in text is mainly used. For example, in the case of the text of FIG. 1, “chocolate” having the highest appearance frequency is acquired as an important word.

Daiya Takamura and Manabu Okumura, "Document Summary Model with Maximum Cover Problem and its Variants", Journal of the Japanese Society for Artificial Intelligence, 2008, Vol.23, No.6, pp.505-513

  However, the prior art has a problem that important words may not be acquired properly based on the appearance frequency.

  For example, when a word that a text creator (hereinafter referred to as “author”) is interested in is an important word, the prior art may not be able to select the important word. This is because authors rarely state words they are interested in many times in the text, and often use more generalized words. As shown in FIG. 1, when an author who likes the type of chocolate called “Truffle” mentions this in the text, the term “chocolate” is used frequently instead of “Truffle”. Therefore, in the prior art, “chocolate” is selected as an important word instead of “truff” which the author is interested in. In this way, only the appearance frequency of words can find a topic in the text, but cannot find a word that the author is interested in.

  In order to solve the above problems, the keyword extraction technique according to the present invention extracts a keyword candidate that is a collocation or proper noun of one or more nouns from the input text, and the action of the input text creator from the input text. The behavioral expression that appears when describing the feature word is extracted as a feature word, one or more features representing the properties of the key word candidate are extracted for each key word candidate, and based on a classification rule predetermined by machine learning, An importance score is calculated using the feature, and an important word is determined from the importance score. When extracting a feature, dependency structure information indicating whether or not it is a keyword candidate related to behavioral expression is assigned to the keyword candidate as a feature.

  The present invention has an effect that an important word can be selected more flexibly by using whether or not it is a word related to behavioral expression.

The figure which shows the example of input text before a morphological analysis. The figure which shows the structural example of an important word extraction apparatus. The figure which shows the processing flow of an important word extraction apparatus. The figure which shows the example of input text after morphological analysis. The figure which shows the example of the important word candidate memorize | stored in a memory | storage part, and its feature. (A) is a figure which shows the example of the feature word memorize | stored in a memory | storage part, (B) is a figure which shows the example of a rule of the action expression memorize | stored in a memory | storage part. The figure which shows the example of a dependency structure. The block diagram which illustrated the hardware constitutions of the keyword extraction device.

  This embodiment uses Japanese lexical features that words related to behavioral expressions and words that receive instruction words are likely to be important words. In addition, the Japanese lexical feature that the instruction word is likely to exist in the sentence including the important word and the sentence including the instruction word in the sentence before and after the important word is used.

Considering these lexical features, it is used as a machine learning feature.
Hereinafter, embodiments of the present invention will be described in detail.

<Keyword extraction apparatus 100>
The keyword extraction device 100 according to the first embodiment will be described with reference to FIGS. The keyword extraction device 100 includes an input unit 101, a storage unit 103, a keyword candidate extraction unit 110, a feature word extraction unit 120, a feature extraction unit 130, and a classifier 140.

The keyword extraction device 100 receives the input text T that has been subjected to morpheme analysis, and outputs the keyword J (p) and its importance score score (J (p)). Here, p = 1, 2,..., P, and P represents the number of selected important words.
<Input unit 101 and storage unit 103>
The keyword extraction device 100 receives the input text after the morphological analysis via the input unit 101 (s101). For example, a sentence “I always buy at various shops, but today I bought G's truffles because of Ginza.” Is input as the morphological-analyzed input text in the state of FIG. The input unit 101 is an input interface or the like through which data is input.

  However, the key word extraction apparatus 100 may receive input text T ′ (see FIG. 1) that has not been subjected to morphological analysis. In that case, the input text T ′ is input to a morpheme analysis unit (not shown) via the input unit 101, and the morpheme analysis unit divides the input text T ′ into words by an existing morpheme analysis method, and parts of speech for each word , The morphological-analyzed input text T (see FIG. 4) is obtained and output to the important word candidate extraction unit 110 and the feature word extraction unit 120.

The storage unit 103 stores / reads each input / output data and each data of the calculation process one by one. Thereby, each calculation process is advanced. However, the data need not necessarily be stored in the storage unit 103, and data may be directly transferred between the units.
<Keyword candidate extraction unit 110>
As shown in FIG. 4, the keyword candidate extraction unit 110 receives the input text T that has been subjected to morphological analysis, and inputs the keyword candidate j (1),... Which is a collocation or proper noun of one or more nouns from the input text T. , J (M) are extracted (s110) and output to the feature extraction unit 130 and the storage unit 103. However, M represents the number of types of important word candidates included in the input text T. Therefore, important word candidates do not overlap.

For example, the keyword extraction unit 110 receives the morpheme-analyzed text T as an input, extracts an expression representing a specific object such as a person name, a place name, or an organization name by an existing specific expression extraction method (see Reference 1), Output as important word candidates.
[Reference 1] Kenji Imamura, Kuniko Saito, Hisako Asano, “Basic Japanese Analysis Technology as a Base for Knowledge Extraction from Texts”, NTT Technical Journal, Telecommunications Association, 2008.6, pp.20-23
Kinds of unique expressions (person names, place names, brand names, etc.) may be assigned to the extracted important word candidates. Furthermore, one or more consecutive nouns are extracted and output as important word candidates. For example, when the text of FIG. 4 is input, the proper expressions “Kyo (date)”, “Ginza (place name)”, “P company (organization)”, “Truffle (noun)” and “O (crown noun)” ) / Store (noun) ”as consecutive nouns“ shop (noun) ”, and these are output as important word candidates. However, the inside of () represents the kind of proper expression. For example, when information obtained by morphological analysis of the input text in FIG. 1 is used as an input, the extracted important word candidates are stored in the storage unit 103 as in the first column in FIG.

<Feature word extraction unit 120>
The feature word extraction unit 120 extracts action expressions and instruction words as feature words h (1),..., H (N) from the input text T that has been subjected to morphological analysis (s120), and stores them in the feature extraction unit 130 and the storage unit 103. Output. N represents the number of feature words included in the input text T. For example, when information obtained by morphological analysis of the input text in FIG. 1 is used as an input, the extracted feature words are stored in the storage unit 103 as illustrated in FIG. The action expression is a word that appears when describing the author's action. The action expression mainly corresponds to a noun representing a past form, a progressive form, and an action of a verb representing a spontaneous action. This includes past forms of adjectives that express the impressions that would have been obtained as a result of the writer's experience. For example, the behavioral expression can be expressed by a rule as shown in FIG. 6B (see Reference 2).
[Reference 2] Kayo Ikeda, Katsuyoshi Tabuchi, Hidenori Okuda, “Extraction of Blog Experience Information Using Experience Expressions”, IEICE 18th Data Engineering Workshop (DEWS2007) Proceedings, 2007
For example, the feature word extraction unit 120 includes an instruction word extraction unit and an action expression extraction unit (not shown). The storage unit 103 stores in advance all instruction words (for example, “this”, “this”, “that”, etc.), and the instruction word extraction unit is included in the input text T while referring to the storage unit 103. The instruction word to be extracted is extracted.

  The storage unit 103 stores all action expressions (for example, “Bought”, “Used”, etc.) in advance, or stores action expression rules as shown in FIG.

The behavioral expression extraction unit extracts the behavioral expression included in the input text T while referring to the behavioral expression itself stored in the storage unit 103 or refers to the rules of behavioral expression stored in the storage unit 103 Then, a word matching the rule is extracted from the text T.
<Feature Extraction Unit 130>
The feature extraction unit 130 extracts one or more features representing the properties of the important word candidate for each important word candidate (s130). A feature is a unit of information (for example, type, frequency, weight, dependency structure information, directive information, title information, etc.) used for predetermined analysis processing in the classifier, and is an important word candidate. Means the nature of In machine learning, by modeling a phenomenon based on this feature, it is possible to learn a dependency relationship between rules as a probabilistic behavior.

For example, the feature extraction unit 130 includes a frequency assignment unit 131, a weight assignment unit 133, a dependency structure information addition unit 135, an instruction word information addition unit 137, and a title information addition unit 139.
(Frequency giving unit 131)
The frequency assigning unit 131 receives the input text T and the keyword candidate j (m) (where m = 1, 2,..., M) as input, and the frequency of occurrence of the keyword candidate j (m) from the input text T (word Frequency) is counted, and the appearance frequency is assigned as the feature α1 (j (m)) to the important word candidate (s131) and output. When the morphological analysis of the document of FIG. 1 is used as the input text, the word frequency of each important word candidate is as shown in the second column of FIG.
(Weighting unit 133)
The weight assigning unit 133 assigns a predetermined weight to the keyword candidate j (m) as a feature α2 (j (m)) and assigns it to the keyword candidate as shown in the fourth column of FIG. 5 (s133). ),Output.

For example, the storage unit 103 stores word weights in advance. As the weight, for example, information obtained from the outside such as a frequency used as a search query in a search engine or a ranking ranked in descending order of frequency used in a search query is used. Further, for example, the ranking or the like ranked in descending order of frequency of appearance in blogs and newspapers may be used as the weight. The weights do not have to be stored in the storage unit 103. For example, when the keyword extraction device 100 can access a site in which an external word search ranking is known via a communication line or the like, the site It is good also as a structure which accesses this, acquires the ranking with respect to each important word candidate, and makes it a weight. In this case as well, the weight is considered to be predetermined at the site or the like. In addition, when the word corresponding to the important word candidate is not in the storage unit 103 or the site, the weight may be a predetermined value (for example, “0”).
(Dependency structure information adding unit 135)
The dependency structure information adding unit 135 receives an important word candidate j (m), a feature word h (n) (where n = 1, 2,..., N) and an input text T that has been subjected to morphological analysis as inputs. The structural information is assigned as the feature α3 (j (m)) to the important word candidate (s135) and output. The dependency structure information is information indicating whether or not it is a keyword candidate related to behavioral expression and whether or not it is a keyword candidate that receives an instruction word.

For example, the dependency structure information adding unit 135 collects important word candidates into phrases by using an existing dependency analysis method (see Reference 3), and extracts a dependency structure between the phrases.
[Reference 3] Makoto Nagao, “Natural Language Processing”, Iwanami Course Software Science 15, Iwanami Shoten, 1996/04
And each important word candidate
-2 if it is an important word candidate that receives the instruction word extracted by the feature word extraction unit
-1 if it is an important word candidate related to the action expression extracted by the feature word extraction unit
-0 for other important word candidates
As dependency structure information in association with important word candidates as shown in the third column of FIG. “According to behavioral expression” means becoming a subject, object, and adverb of behavioral expression, and “receiving a directive” means being modified to a directive. The values 0 to 2 may be other numerical values or character strings as long as each can be identified.

For example, the input is “Kyo (noun) / ha (joint noun) / Ginza (noun) / ni (case particle) / yo (verb stem) / tsu (verb inflection ending) / ta (verb suffix) / (connection (Suffix) / that (combined) /
In the case of “Company P (noun) / (case particle) / Truffle (noun) / (case particle) / purchase (noun)”, a dependency structure as shown in FIG. 7 is extracted.

This is an important word that can be obtained from the key words extracted by the key word extraction unit and important word candidates and directives related to behavioral expressions such as “Bought” and “Used” extracted by the feature word extraction unit. The purpose is to identify word candidates. For example, from the result in the third column of FIG. 5, “Kyo” and “Truffle” whose dependency structure information is “1 (important word candidate related to action expression)” are important word candidates related to action expression. In addition, it can be seen that “Company P” whose dependency structure information is “2 (important word candidate for receiving an instruction word)” is an important word candidate for receiving an instruction word.
(Indicator information adding unit 137)
The instruction word information adding unit 137 receives the instruction word included in the feature word, the important word candidate j (m), and the input text T after morphological analysis, and the instruction word information as the feature α4 (j (m)). As shown in the fifth column of 5, it is assigned to the important word candidate (s 137) and output. The instruction word information is information indicating whether or not an instruction word is present in a sentence including an important word candidate, and whether or not an instruction word is present in a sentence before and after the sentence including the important word candidate. . In FIG. 5, the value of the instruction word information is
・ If there is a directive in the sentence before and after the sentence containing the key word candidate, 2
-1 if the instruction word is present in the sentence that contains the keyword candidate
-0 otherwise
It has become. This value may be another value or a character string as long as the above three states can be identified.

Further, for example, when a directive word is particularly likely to be present in a sentence after a sentence including an important word (or a preceding sentence), the feature value may not necessarily be provided according to the above-described rules. In other words, when a directive word exists in a sentence before a sentence including a keyword candidate, if a directive word exists in a sentence after the sentence including the keyword candidate, the directive word exists in a sentence including the keyword candidate. In this case, feature values may be provided for the other four cases, and feature values may be provided for the three cases excluding “when a directive word exists in a sentence preceding a sentence including an important word candidate”. May be. Further, whether or not the instruction word exists in two or more sentences before and after the sentence including the important word candidate may be provided as a feature value.
(Title information adding unit 139)
The title information assigning unit 139 receives the title and the keyword candidate j (m) as input, and assigns the title information as the feature α5 (j (m)) to the keyword candidate as shown in the sixth column of FIG. s139) and output. Note that the title information is information indicating whether important word candidates are included in the title.

For example, the title information giving unit 139, when the input text has a title,
-1 for important word candidates included in the title
-0 for important word candidates not included in the title
Is given as title information. If the title of FIG. 1 is “Coffee Companion”, it becomes like the sixth column of FIG. If the input text does not have a title, the processing in the title assigning unit may not be performed, title information (for example, 2) indicating that the title is not attached may be given, and the title may be given. You may give the same title information (for example, 0) as the case of the important word candidate which is not contained. Note that the title information adding unit 139 is not essential.
<Classifier 140>
The classifier 140 receives the features α1 (j (m)) to α5 (j (m)) as input, and based on a classification rule predetermined by machine learning (see Reference 4), the importance score using the features is used. score (j (m)) is calculated.
[Reference 4] Shunichi Amari, Hideki Aso, Koji Tsuda, Noboru Murata, “Statistics of Pattern Recognition and Learning: New Concepts and Methods”, Iwanami Shoten, April 2003, and further, importance score score (j (m) ) Determines the important word J (p) (s140), and outputs the important word J (p) and the importance score score (J (p)) (s150). When the value of the importance score score (J (p)) is a certain value or more, the important word candidate may be set as an important word, or the importance score score (J (p)) is predetermined within a single document. An important word candidate that is higher than the ranking may be used as an important word. Moreover, it is good also considering the important word candidate of the upper few% as an important word.

  For example, the classifier 140 includes the feature α (j (m)) = [α1 (j (m)), α2 (j (m)), α3 (j (m))) of each important word candidate j (m), Using α4 (j (m)), α5 (j (m))] as inputs, an importance score is calculated based on a model (classification rule) f () created in advance.

score (j (m)) = f (α (j (m)))
The importance score of all important word candidates is obtained, and the important word is determined from the importance score. By using Japanese lexical features, important words (for example, words that the author is interested in) can be output.
<How to create a classification rule>
The classification rule f () is learned in advance by machine learning. That is, with respect to the learning text set, the important word candidates extracted by the feature extraction unit and the features assigned to each important word candidate, and the important words ( For example, what is given a label indicating a word that the author is interested in) is used as learning data. A classification rule for selecting an important word from important word candidates given features is learned by known machine learning described in Reference 1 or the like.
<Effect>
Select important words more flexibly than conventional technologies that select important words only for appearance frequency by selecting important words for machine learning based on whether they are important word candidates related to behavioral expressions In addition, it is possible to select an important word more flexibly and appropriately by making it a feature whether or not it is an important word candidate that receives an instruction word.

  An important word can be selected more appropriately by using Japanese lexical features that a directive word is likely to be present in a sentence including the important word and a sentence including the directive word in the preceding and succeeding sentences.

  Since the frequency of words used in the prior art is also an important factor in determining important words, it is possible to select important words more appropriately by using machine learning features. However, in the prior art, an important word is determined on the assumption that a word having a high appearance frequency is an important word. However, in this embodiment, a word having a high appearance frequency is simply considered to be an important word. It is assumed that there is an appearance frequency that tends to become an important word, and learning is performed by machine learning. With such a configuration, an important word can be selected more appropriately.

  Since the weight including the rank is an important factor in determining the important word, the important word can be selected more appropriately by using the machine learning feature. Note that weights that are likely to become important words are learned by machine learning without considering that words having high (heavy) weight values as in the case of frequency are important words.

  Since the important word candidate included in the title is considered to be more likely to be an important word, the important word can be selected more appropriately by using the title information as a feature of machine learning.

By selecting an important word in consideration of the relationship between these features, it is possible to select an important word more appropriately and more flexibly than in the prior art.
<Hardware configuration>
As illustrated in FIG. 8, the keyword extraction device 100 of this example includes a CPU (Central Processing Unit) 11, an input unit 12, an output unit 13, an auxiliary storage device 14, a ROM (Read Only Memory) 15, a RAM ( Random Access Memory) 16 and a bus 17.

The CPU 11 in this example includes a control unit 11a, a calculation unit 11b, and a register 11c, and executes various calculation processes according to various programs read into the register 11c. The input unit 12 is an input interface for inputting data, a keyboard, a mouse, and the like. The output unit 13 is an output interface for outputting data, a display, a printer, and the like. The auxiliary storage device 14 is, for example, a hard disk, a semiconductor memory, or the like, and stores a program for causing the computer to function as the keyword extraction device 100 and various data. Further, the above-mentioned program and various data are expanded in the RAM 16 and used from the CUP 11 or the like. The bus 17 connects the CPU 11, the input unit 12, the output unit 13, the auxiliary storage device 14, the ROM 15, and the RAM 16 in a communicable manner. In addition, as a specific example of such hardware, a server apparatus, a workstation, etc. other than a personal computer can be illustrated, for example.
<Program structure>
As described above, the auxiliary storage device 14 stores each program for executing each process of the keyword extraction device 100 of the present embodiment. Each program constituting the keyword extraction program may be described as a single program sequence, or at least a part of the programs may be stored in the library as a separate module.
<Cooperation between hardware and program>
The CPU 11 expands the above-described program and various data stored in the auxiliary storage device 14 in the RAM 16 according to the read OS program. The address on the RAM 16 where the program and data are written is stored in the register 11c of the CPU 11. The control unit 11a of the CPU 11 sequentially reads these addresses stored in the register 11c, reads a program and data from the area on the RAM 16 indicated by the read address, and causes the calculation unit 11b to sequentially execute the operation indicated by the program. The calculation result is stored in the register 11c.

  FIG. 2 is a block diagram illustrating the functional configuration of the keyword extraction device 100 configured by reading and executing the above-described program in the CPU 11 in this manner.

Here, the storage unit 103 corresponds to any one of the auxiliary storage device 14, the RAM 16, the register 11 c, other buffer memory and cache memory, or a storage area using these in combination. The keyword candidate extraction unit 110, the feature word extraction unit 120, the feature extraction unit 130, and the classifier 140 are configured by causing the CPU 11 to execute a keyword extraction program.
<Modification>
In the present embodiment, the keyword extraction device 100 outputs the keyword and the importance score, but it is sufficient to output at least the keyword.

  The important word candidate extraction unit 110 may assign a type (for example, a person name, a place name, etc.) as a specific expression to the extracted important word candidate as a feature. At this time, the classifier 140 can also use this type of machine learning feature. By adopting such a configuration, for example, when an author tends to make a place name an important word rather than a person name, the place name is easily selected as the important word.

  In the present embodiment, the features extracted by the feature extraction unit 130 are five: frequency, weight, dependency structure information, directive information, and title information, but at least of the dependency structure information, an important word related to action expression If the information indicating whether or not is a feature, it is possible to select key words more flexibly than in the prior art.

  In the present embodiment, the case where the dependency structure information provided by the dependency structure information adding unit 135 is one for one important word candidate has been described. However, there are two relationships for one important word candidate. When receiving information (for example, 1 and 2) can be provided, two pieces of dependency information may be provided. Further, when the same dependency information can be given twice or more for one important word candidate, the same processing may be performed as in the case of giving only once. The same applies to the instruction word information provided by the instruction word information adding unit 137.

  The present invention can be used when extracting important words from a plurality of text documents and obtaining a keyword currently focused on from the important words, in addition to simply extracting important words of a text document. Further, for example, as in Non-Patent Document 1, it can be used when creating a summary from a text document.

100 keyword extraction unit 110 keyword candidate extraction unit 120 feature word extraction unit 130 feature extraction unit 131 frequency assignment unit 133 weight assignment unit 135 dependency structure information addition unit 137 instruction word information addition unit 139 title information addition unit 140 classifier

Claims (10)

An important word candidate extraction unit for extracting an important word candidate that is a collocation or proper noun of one or more nouns from the input text;
A feature word extraction unit that extracts, as a feature word, an action expression that appears when describing the action of the input text creator from the input text;
A feature extraction unit that extracts one or more features representing the properties of the keyword candidates for each keyword word;
A classifier that calculates an importance score using the feature based on a classification rule predetermined by machine learning, and determines an important word from the importance score;
The feature extraction unit includes:
A dependency structure information assigning unit that assigns the dependency structure information indicating whether or not it is a key word candidate related to the behavior expression to the key word candidate as a feature,
An important word extraction device characterized by that. The key word extraction device according to claim 1,
The feature word extraction unit extracts an instruction word as a feature word in addition to the action expression,
The dependency structure information represents whether or not it is a keyword candidate that receives the instruction word, in addition to whether or not it is a keyword candidate related to the behavior expression,
The dependency structure information assigning unit assigns the dependency structure information as a feature to an important word candidate.
An important word extraction device characterized by that. The key word extraction device according to claim 1 or 2,
The feature word extraction unit extracts an instruction word as a feature word in addition to the action expression,
The feature extraction unit further includes:
The keyword candidate is identified by using, as a feature, indicator word information indicating whether or not a directive word exists in a sentence including the keyword candidate, and whether or not a directive word exists in the sentences before and after the sentence including the keyword candidate. An instruction word information assigning unit to be provided to,
An important word extraction device characterized by that. The key word extraction device according to any one of claims 1 to 3,
The feature extraction unit further includes:
A frequency giving unit that obtains the appearance frequency of the important word candidate from the input text and assigns it to the important word candidate as a feature;
An important word extraction device characterized by that. The key word extraction device according to any one of claims 1 to 4,
The feature extraction unit further includes a weight assigning unit that assigns a predetermined weight to the important word candidate as a feature to the important word candidate.
An important word extraction device characterized by that. The key word extraction device according to any one of claims 1 to 5,
The feature extraction unit further provides title information indicating whether or not an important word candidate is included in the title by using the title of the input text and the important word candidate as a feature, and adding title information to the important word candidate And comprising
An important word extraction device characterized by that. The key word extraction device according to any one of claims 1 to 6,
The important word candidate extraction unit, when extracting the important word candidate, gives the type as a specific expression as a feature to the important word candidate.
An important word extraction device characterized by that. An important word candidate extraction step of extracting an important word candidate that is a collocation or proper noun of one or more nouns from the input text;
A feature word extraction step of extracting an action expression that appears when describing the action of the input text creator from the input text as a feature word;
A feature extraction step of extracting one or more features representing the properties of the keyword candidate for each keyword candidate;
A classifier that calculates an importance score using the feature based on a classification rule predetermined by machine learning, and determines an important word from the importance score;
The feature extraction step includes:
A dependency structure information giving step for assigning to the important word candidate the dependency structure information representing whether or not it is a key word candidate related to the behavior expression,
An important word extraction method characterized by this. The key word extraction method according to claim 8,
The feature word extraction step extracts an instruction word as a feature word in addition to the action expression,
The dependency structure information represents whether or not it is a keyword candidate that receives the instruction word, in addition to whether or not it is a keyword candidate related to the behavior expression,
The dependency structure information giving step assigns the dependency structure information as a feature to an important word candidate.
An important word extraction method characterized by this.   A program for causing a computer to function as the important word extracting device according to claim 1.

Images