JP2010040020A - Keyword extraction device, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a keyword extraction server, a method and a program capable coping with the text of a free form (including multilingual), by which high speed processing is made possible, and which can extract only a word effective for classification. <P>SOLUTION: The keyword extraction device is provided with: a division part 11 which divides an input text with punctuation; a morpheme extraction part 12 which extracts morpheme from divided division parts; a noun extraction part 13 which decides parts of speech for the extracted morpheme to extract the morpheme decided to be the noun; an arithmetic operation part 14 which calculates the score of the noun as a keyword on the basis of the number of characters of the noun, appearance frequency of the noun in the text, and a ratio of the total number of sentences in the text to the appearance frequency indicating over how many sentences the noun appears, about the extracted noun; and a decision part 15 which decides whether or not the noun is considered as the keyword on the basis of the score as the result of the arithmetic operation. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a keyword extraction apparatus, method, and program. More particularly, the present invention relates to a keyword extraction apparatus, method, and program capable of efficiently extracting keywords effective for classification of various document data such as data stored in a database or data acquired via the Internet. .

  With the spread of personal computers and the Internet, and the development of electronic filing technology, an environment where a large amount of digitized document data can be used is being prepared. On the other hand, important keywords are automatically selected from a vast amount of information. There is a need for a system to extract.

  Various methods have been proposed so far to classify document data stored in a database or document data acquired via the Internet. For example, a technique is widely known in which a plurality of keywords that are features included in document data are selected, their distribution and appearance positions are analyzed, and classification is performed based on the analysis results.

  What is important when performing document data classification processing is selection of “words effective for classification”, that is, keyword selection. Conventional keyword selection methods can be broadly classified into the following three methods (1) to (3).

(1) Keyword selection method using dictionary data The keyword selection method using dictionary data is a method in which word groups that are considered to be effective for classifying document data are registered in advance as dictionary data, and the registered words are used as keywords. A method of using this dictionary data as a keyword is described in Patent Document 1 and Patent Document 2, for example.

(2) Keyword selection method by grammatical analysis of documents included in document data to be classified The keyword selection method by grammatical analysis of documents included in document data to be classified is based on the grammar of documents included in the document data to be classified. Morphological analysis or analysis based on original grammatical rules, and a word extracted as a result is used as a keyword or a candidate thereof. This technique is described in Patent Document 3 and Patent Document 4, for example.

(3) Keyword selection method based on total comparison of classification target document data The keyword selection method based on total comparison of classification target document data performs a total comparison of various document data to be classified, This is a method of analyzing the combination data and extracting keywords or keyword candidates based on the analysis result. This technique is described in Patent Document 5, for example.

  As described above, various methods have already been proposed as keyword extraction methods. However, for example, the above-mentioned “(1) Keyword selection method using dictionary data” takes specialized knowledge and time to create a prerequisite dictionary, and the created dictionary is in an unexpected field. There is a problem that the text is not effective enough. For example, there are problems that keywords that are effective for classification of documents in a specific specialized field such as medical care and finance are insufficient, or that new words that appear cannot be handled.

  In addition, “(2) Keyword selection method by grammatical analysis of documents included in document data to be classified” requires specialized knowledge to standardize grammar rules, and it is not possible to There is a problem that it does not have a sufficient effect on free-form sentences that are not grammatically established.

Furthermore, in “(3) Keyword selection method based on total comparison of document data to be classified”, when the amount of document data to be processed increases, the time required for the comparison processing increases exponentially and processing efficiency decreases. In addition, there is a problem that phrases other than words that are effective for classification are extracted, such as “is” and “mass” that appear frequently in documents in Japanese.
As an algorithm for solving this problem, there is tf · idf (Term Frequency-Inverse Document Frequency, an algorithm for extracting characteristic words (words regarded as important) in a sentence) (described later).

JP 2002-215647 A JP 2002-108888 A JP 2003-36261 A JP 2002-245061 A Japanese Patent Laid-Open No. 2001-22752

The present invention has been made in view of such a situation, and an object of the present invention is to provide a keyword extraction apparatus, method, and program that solve the problems in the conventional keyword extraction method described above.

  Specifically, it does not use prerequisite knowledge (dictionary, grammar data), can handle free-form (including multilingual) sentences, is capable of high-speed processing, and is effective for classification It is an object of the present invention to provide a keyword extraction server, method, and program capable of extracting only the keyword.

(1) dividing means for dividing the input text by punctuation marks;
Morpheme extraction means for extracting morphemes from the divided parts divided by the dividing means;
Determining a part of speech for the morpheme extracted by the morpheme extraction unit, and extracting a morpheme determined to be a noun;
About the noun extracted by the noun extraction means, the number of characters of the noun, the appearance frequency of the noun in the text, the total number of sentences in the text, and how many sentences the noun appears over Calculating means for calculating a score as a keyword of the noun based on a ratio to the appearance frequency indicating
A keyword extracting device comprising: a determination unit that determines whether or not the noun is a keyword based on the score that is a result of the calculation.

  According to the invention according to (1), a noun is extracted from the input text, the number of characters of the noun, the frequency of appearance of the noun in the text, the total number of sentences in the text and the number of nouns The keyword is determined based on the ratio to the appearance frequency indicating whether it has appeared across sentences (in the text).

  In this way, according to the invention according to (1), it is possible to handle free-form (including multilingual) sentences without using the prerequisite knowledge (dictionary, grammar data), and high-speed processing. Only words that are possible and effective for classification can be extracted.

  That is, it is possible to quickly extract only keywords that are important clues for elucidating the problem and understanding the contents, and that are effective as words used as clues for information retrieval.

  (2) The calculation means includes the number of characters of the noun or a logarithm calculation value around the number of characters of the noun, the appearance frequency of the noun in the text, the total number of sentences in the text, and how many nouns the sentence includes (1) The keyword extracting device according to (1), wherein a value obtained by multiplying a ratio with an appearance frequency indicating whether or not it has appeared or a logarithm operation value of a number before and after the ratio is used as the score.

  According to the invention according to (2), for the noun extracted in (1), the number of characters of the noun or the logarithm calculation value before and after the number of characters of the noun, the appearance frequency of the noun in the text, the sentence in the text The determination of the keyword is executed based on a value obtained by multiplying the ratio between the total number and the appearance frequency indicating how many sentences the noun appears over or the logarithm operation values of the numbers before and after the ratio.

  In this way, according to the invention according to (2), the calculation method is a simple calculation of calculating the number of characters, logarithmic calculation, noun appearance frequency in the text, total number of sentences, and ratio, and having a small number of calculations. Since keywords can be extracted based on this, a keyword extraction server that can handle free-form (including multilingual) sentences without using the prerequisite knowledge (dictionary, grammar data), and provides high-speed processing is provided. It becomes possible to do.

(3) a character information database storing character information transmitted and received on the Internet;
Selecting means for selecting a maximum score noun with the highest score among the nouns determined to be the keyword by the determining means;
The maximum score noun and the noun are searched in the character information database, and the search number of the maximum score noun, the search number of the noun, and the search number including both the maximum score noun and the noun are searched and investigated. Search means to
A correction coefficient calculating means for calculating a correction coefficient based on the search number of the maximum score noun, the search number of the noun and the search number including both the maximum score noun and the noun;
Correction score calculation means for calculating a correction score based on the correction coefficient and the score calculated by the calculation means;
The keyword extracting device according to (1) or (2), wherein the determining means determines whether or not the noun is a keyword based on the correction score.

  According to the invention according to (3), the maximum score noun and the noun are searched in the character information database, the search number of the maximum score noun, the search number of the noun, and both the maximum score noun and the noun Are searched, and nouns that have become keyword candidates related to (1) are corrected based on the number of searches.

  Thus, according to the invention according to (3), the effect of the correction coefficient (score B) (if the relevance with the maximum score noun with the largest score (score A) value is high, the correction coefficient ( If the score B) is large and the relevance to the largest score noun with the largest value of score A is small, the correction coefficient (score B) is small. The score C) becomes a small value and can be appropriately calculated so as not to be determined as a keyword.

(4) The correction coefficient calculation means performs a division operation on the number of searches including both the maximum score noun and the noun by a square root of a multiplication calculation value of the search number of the maximum score noun and the search number of the noun. The value is the corrected score,
The keyword extraction device according to (3), wherein the determination unit determines whether or not the noun is a keyword based on a multiplication operation value of the correction score and the score.

  According to the invention of (4), the number of searches including both the maximum score noun and the noun is divided by the square root of the multiplication value of the search number of the maximum score noun and the search number of the noun. Based on the value, it is determined whether or not to use the noun as a keyword.

  Thus, according to the invention according to (4), keywords can be extracted based on a calculation method that is a simple calculation such as the number of searches, the number of search characters, the multiplication operation, and the square root operation, and the number of operations is small. Therefore, it is possible to provide a keyword extraction server that can handle free-form (including multilingual) sentences without using knowledge (dictionary, grammar data).

(5) a dividing step of dividing the input text with punctuation marks;
A morpheme extraction step of extracting morphemes of the divided parts divided in the division step;
Determining a part of speech for the morpheme extracted in the morpheme extraction step, and extracting a morpheme determined as a noun;
About the noun extracted in the noun extraction step, the number of characters of the noun, the appearance frequency of the noun in the text, and the total number of sentences in the text and how many sentences the noun appears over A calculation step of calculating a score as a keyword of the noun based on a ratio with an appearance frequency indicating:
And a determination step of determining whether or not to use the noun as a keyword based on the score that is a result of the calculation.

  According to the invention according to (1), a noun is extracted from the input text, the number of characters of the noun, the frequency of appearance of the noun in the text, the total number of sentences in the text and the number of nouns The keyword is determined based on the ratio to the appearance frequency indicating whether it has appeared across sentences (in the text).

  In this way, according to the invention according to (1), it is possible to handle free-form (including multilingual) sentences without using the prerequisite knowledge (dictionary, grammar data), and high-speed processing. Only words that are possible and effective for classification can be extracted.

  That is, it is possible to quickly extract only keywords that are important clues for elucidating the problem and understanding the contents, and that are effective as words used as clues for information retrieval.

  (6) A program that causes a computer to execute the method according to (5).

  According to such a configuration, the same effect as in (5) can be expected by causing the computer to execute the program.

  According to the present invention, premise knowledge (dictionary, grammar data) is not used, it is possible to deal with free-form (including multilingual) sentences, high-speed processing is possible, and effective for classification. Only words can be extracted. That is, it is possible to quickly extract only keywords that are important clues for elucidating the problem and understanding the contents, and that are effective as words used as clues for information retrieval.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Entire system configuration]
FIG. 1 shows an information processing system 1 including a keyword extraction server 10 and a user terminal 30 according to the present embodiment. In FIG. 1, the information processing system 1 shows the keyword extraction server 10 and the user terminal 30 one by one. However, the information processing system 1 is not limited to this, and may be configured by a plurality of units. .

  As shown in FIG. 2, the keyword extraction server 10 includes a CPU (Central Processing Unit) 310 (a plurality of CPUs such as a CPU 320 may be added in a multiprocessor configuration), a bus line 200, and a communication I. / F (I / F: interface) 330, main memory 340, BIOS (Basic Input Output System) 350, I / O controller 360, hard disk 370, optical disk drive 380, and semiconductor memory 390. The hard disk 370, the optical disk drive 380, and the semiconductor memory 390 are collectively referred to as a storage device 410.

  The control unit 300 is a part that controls the keyword extraction server 10 in an integrated manner. By appropriately reading and executing various programs stored in the hard disk 370, the control unit 300 cooperates with the hardware described above, and performs various types according to the present invention. The function is realized.

  The communication I / F 330 is a network adapter when the keyword extraction server 10 transmits / receives information to / from other devices such as the user terminal 30 via the network.

  The BIOS 350 records a boot program executed by the CPU 310 when the keyword extraction server 10 is activated, a program depending on the hardware of the keyword extraction server 10, and the like.

  A storage device 410 such as a hard disk 370, an optical disk drive 380, and a semiconductor memory 390 can be connected to the I / O controller 360.

  The hard disk 370 stores various programs for causing the hardware to function as the keyword extraction server 10, a program for executing the functions of the present invention, a table to be described later, and the like. The keyword extraction server 10 can also use an external hard disk (not shown) as an external storage device.

  As the optical disk drive 380, for example, a DVD-ROM drive, a CD-ROM drive, a DVD-RAM drive, or a CD-RAM drive can be used. In this case, the optical disk 400 corresponding to each drive is used. A program or data can be read from the optical disk 400 by the optical disk drive 380 and provided to the main memory 340 or the hard disk 370 via the I / O controller 360.

  The computer in the present invention refers to an information processing apparatus including a storage device, a control unit, and the like, and the keyword extraction server 10 includes an information processing device including a storage device 410, a control unit 300, and the like.

  In addition, the keyword extraction server 10 according to the present invention has the above-described configuration, so that morphological analysis is performed on text input from the user terminal 30, and nouns are extracted from the input text. It has a function of calculating a score as a keyword and determining whether or not to make a keyword based on the score.

  Here, a configuration for exhibiting the function will be described with reference to a functional block diagram shown in FIG. The keyword extraction server 10 includes a division unit 11, a morpheme extraction unit 12, a noun extraction unit 13, a determination unit 15, a selection unit 16, a search unit 17, a correction coefficient calculation unit 18, and a correction score calculation unit 19. And a character information database (DB) 20.

  The dividing unit 11 is based on text directly input from the user terminal 30, text already stored in the character information database 20, and audio information generated in a program broadcast in a broadcasting station (not shown). Text (English, Japanese, etc.) such as text as created text information, text obtained by voice analysis of information input as voice information, text (including OCR) created from image information, etc. Is not limited to this type), and has a function of dividing by a delimiter such as a punctuation mark.

  The morpheme extraction unit 12 extracts morphemes from the divided parts divided by the division unit 11. A morpheme analysis method is used to extract morphemes. For example, tf.idf (Term Frequency-Inverse Document Frequency, an algorithm for extracting characteristic words (words regarded as important) in a sentence) using a part of an algorithm, a plurality of morphemes are obtained from character information. Has a function to extract.

  The noun extraction unit 13 has a function of determining the part of speech of the morpheme extracted by the morpheme extraction unit 12 and extracting the morpheme determined to be a noun.

  For the noun extracted by the noun extracting means, the determination unit 15 determines the number of characters of the noun, the frequency of appearance of the noun in the text, the total number of sentences in the text, and the number of nouns across the sentences. The score as a keyword of the noun is calculated based on the ratio with the appearance frequency indicating whether or not it has appeared.

で示され、式中、｜ｗ｜は単語ｗの文字数であり、ｔｆ（ｗ）は単語ｗのテキスト中での出現頻度（テキスト中に何回出現したかを示す）、ｓｆ（ｗ）は単語ｗの文中での出現頻度（いくつの文に跨って出現したかを示す）、Ｎはテキスト中の文の総数を示す。
演算部１４は式（１）に基づいてスコアＡを演算する。 Specifically, if the score A for the noun w is the score A (w)

Where | w | is the number of characters of the word w, tf (w) is the frequency of appearance of the word w in the text (shows how many times it appears in the text), and sf (w) is Appearance frequency of the word w in the sentence (indicating how many sentences have appeared), N indicates the total number of sentences in the text.
The calculation unit 14 calculates the score A based on the formula (1).

The determination unit 15 has a function of determining whether or not the noun w is a keyword based on the score A that is the result of the calculation of the expression (1) in the calculation unit 14.
The determination criterion can be set in advance to an arbitrary value. An arbitrary value can be determined by repeating trial and error.

  The keywords extracted here are important clues for elucidating the problem and understanding the contents. Also, in information retrieval, it is a phrase used as a clue for retrieval.

  The selection unit 16 has a function of selecting the maximum score noun having the highest score among the nouns determined by the determination unit 15 as keywords.

  The retrieval unit 17 retrieves the maximum score noun and the noun extracted by the noun extraction unit 13 from a character information database (DB) 20 provided in the keyword extraction server 10 or in an external character information DB (not shown). It has a function of searching and investigating the search number of score nouns, the search number of nouns, and the search number including both the maximum score nouns and nouns.

  The correction coefficient calculation unit 18 has a function of calculating a correction coefficient based on the maximum score noun search number, the noun search number, the maximum score noun, and the search number including both of the nouns.

で示され、式中、ＷｍａｘＡはスコアＡが最も大きい名詞を示し、｜ＷｍａｘＡ＆Ｗ｜は｜ＷｍａｘＡ｜と名詞ｗとのＡＮＤ検索（キーワード抽出サーバ１０のある文字情報データベース（ＤＢ）２０からまたは外部の文字情報ＤＢ（図示せず））でのヒット件数を示し、｜ＷｍａｘＡ｜は名詞ＷｍａｘＡの単独検索（キーワード抽出サーバ１０のある文字情報データベース（ＤＢ）２０からまたは外部の文字情報ＤＢ（図示せず））でのヒット件数を示し、｜Ｗ｜は名詞Ｗの単独検索（キーワード抽出サーバ１０のある文字情報データベース（ＤＢ）２０からまたは外部の文字情報ＤＢ（図示せず））でのヒット件数を示す。 Specifically, when the correction coefficient B for the noun w is a score B (w)

Where WmaxA indicates the noun with the highest score A, and | WmaxA & W | is an AND search between | WmaxA | and the noun w (from the character information database (DB) 20 with the keyword extraction server 10 or externally). Indicates the number of hits in the character information DB (not shown), and | WmaxA | is a single search for the noun WmaxA (from the character information database (DB) 20 with the keyword extraction server 10 or external character information DB (not shown)). )) Indicates the number of hits, and | W | indicates the number of hits in a single search for the noun W (from the character information database (DB) 20 with the keyword extraction server 10 or in an external character information DB (not shown)). Show.

  The correction score calculation unit 19 has a function of calculating a correction score based on the correction coefficient and the score calculated by the calculation unit 14.

で示される。 Specifically, if the score A for the noun w is score A (w), the correction coefficient B for the noun w is score B (w), and the correction score for the noun w is correction score C (w),

Indicated by

  Based on the corrected score, the determination unit 15 determines whether or not the noun is a keyword. The determination criterion can be set in advance to an arbitrary value. An arbitrary value can be determined while repeating trial and error. In the present embodiment described below, a noun having a score A greater than 10 can be determined as a keyword based on 10 as an example. is there.

  According to such a configuration, the keyword extraction server 10 according to the present invention performs morphological analysis on text input from the user terminal 30, extracts nouns from the input text, and scores the extracted nouns as keywords. In addition to having the function of determining whether or not to use as a keyword based on the score, the relationship between the noun as a keyword candidate and the maximum score noun is the number of hits in the database with other articles (search By correcting the relevance based on the number of hits, the nouns that should be truly keywords can be extracted appropriately.

  That is, when compared with tf · idf, which is an example of the prior art, even if a word has high keyword characteristics in tf · idf, if it appears frequently in other articles, the score as a keyword candidate noun will decrease. The present invention has an advantageous effect that the defect of tf · idf can be corrected efficiently.

  In addition, when compared with tf · idf, which is an example of the prior art, even if a noun has low keyword characteristics in tf · idf, the score will increase unless it appears in other articles. In the present invention, there is an advantageous effect that the defect of tf · idf can be corrected efficiently.

[Processing procedure]
Here, a specific processing procedure that can be realized when the present invention is applied will be described with reference to a flowchart shown in FIG. The processing procedure shown below is an example, and there are innumerable processing procedures that can be realized in addition to this.

  In step S <b> 1, the dividing unit 11 of the keyword extraction server 10 includes text directly input from the user terminal 30, text already stored in the character information database 20, text input from other external devices (not shown) ( (It is not limited to language types such as English, Japanese, etc.).

  In step S <b> 2, the morpheme extraction unit 12 of the keyword extraction server 10 extracts a morpheme that is a minimum sound shape having a meaning from the divided parts divided by the division unit 11.

  In step S3, the noun extraction unit 13 of the keyword extraction server 10 determines whether or not the morpheme extracted by the morpheme extraction unit 12 is a noun, and extracts the morpheme determined to be a noun.

  In step S4, the calculation unit 14 of the keyword extraction server 10 calculates a score A indicating a criterion for determining whether or not the morpheme that is the noun extracted in step S3 can be a keyword.

  The score A is calculated based on the above-described equation (1).

  In step S5, when the score A calculated in step S4 is larger than a predetermined value, the determination unit 15 of the keyword extraction server 10 selects a word (noun) that is a morpheme indicating the value of the score A. Judged as a keyword. As an example, when the value of score A is greater than about 10, it is possible to determine a word (noun) that is a morpheme indicating the value of score A as a keyword.

  In step S6, the selection unit 16 of the keyword extraction server 10 selects the maximum score noun having the largest score A among the keywords determined in step S5.

  In step S7, the search unit 17 of the keyword extraction server 10 searches the character information database 20 for the maximum score noun selected in step S6, and the number of hits is the maximum score noun search number (| WmaxA |). To do. In addition, a search is performed in the character information database 20 for other keyword candidate nouns (W), and the number of hits is set as the number of keyword candidate noun searches (| W |).

  Further, the search unit 17 of the keyword extraction server 10 searches the character information database 20 for information including both the maximum score noun and other keyword candidate nouns, and obtains the number of search hits (| WmaxA & W |).

  In step S8, the correction coefficient calculation unit 18 of the keyword extraction server 10 calculates a correction coefficient for score A (score B (w)) based on | WmaxA |, | W | and | WmaxA & W | searched in step S7. To do. The correction coefficient (score B (w)) is calculated based on the above-described equation (2).

  In step S9, the correction score calculation unit 19 of the keyword extraction server 10 determines the noun w based on the correction coefficient (score B (w)) calculated in step S8 and the score A (w) calculated in step S4. A correction score C (w), which is a correction score for, is calculated.

  In step S10, when the correction score C (w) regarding the noun w calculated in step S9 is larger than a predetermined value, the determination unit 15 of the keyword extraction server 10 determines the correction score C (w). A word (noun) indicating a value is determined as a keyword. As an example, when the value of the correction score C (w) is larger than about 10, it is possible to determine a word (noun) indicating the value of the correction score C (w) as a keyword.

  According to such a configuration, the keyword extraction server 10 according to the present invention performs morphological analysis on text input from the user terminal 30, extracts nouns from the input text, and scores the extracted nouns as keywords. In addition to having the function of determining whether or not to use a keyword based on the score, the noun as a keyword candidate is related to the maximum score noun and the number of hits in the database with other articles (search By correcting the relevance based on the number of hits, the nouns that should be truly keywords can be extracted appropriately.

[Keyword identification method]
An example of a keyword specifying method will be described below. For example, when the input text is the following sentence as shown in FIG. 5, the nouns “digital camera”, “camera”, “large copy”, and “Gunyan” extracted by the noun extraction unit 13 are corrected according to this embodiment. FIG. 6 shows the calculation process for the score C (w), the score A (w), the score B (w), and the tf · idf score which is an example of the prior art, and the calculation result is shown in FIG.

  With reference to FIG. 6 showing the calculation process in the case where the nouns determined by the keyword extraction server 10 as keyword candidates from the input text are “digital camera”, “camera”, “large copy”, and “Gunyan”. explain.

  When the noun determined as the keyword candidate is “digital camera”, if the score A (w) is to be obtained, log (| digital camera | +1) * tf (digital camera) * log (N / (sf (digital camera) +1)) must be calculated (from equation (1)).

In the equation, | digital camera | is the number of characters of the word digital camera, so it becomes 4 from the input text, and log (| digital camera | +1) becomes log (5).

  Also, tf (digital camera) is the frequency of appearance of the word digital camera in the text (indicating how many times it appears in the text), so it becomes 4 from the input text, and tf (digital camera) becomes 4.

  Furthermore, since sf (digital camera) indicates the appearance frequency in the sentence of the word digital camera (indicating how many sentences have appeared), sf (digital camera) is 4 from the input text. Since N indicates the total number of sentences in the text, N is 22 from the input text.

  Therefore, log (digital camera / (sf (digital camera) +1)) becomes log (22 / (4) +1)).

  As a result, the value of score A (digital camera) is 12.0502. As an example, if the value of score A is greater than around 6, if a word (noun) that is a morpheme indicating the value of score A is determined as a keyword, the digital camera may be a keyword of the input text. it can.

  Next, in the case where the noun determined as the keyword candidate is “Gunyan” and the score A (w) is to be obtained, log (| Gunyan | +1) * tf (Gunyan) * as in the case of the digital camera. log (N / (sf (Gnyan) +1)) needs to be calculated (from equation (1)).

In the formula, | Gunyan | is the number of characters of the word digital camera, so it becomes 4 from the input text, and log (| Gunyan | +1) becomes log (5).

  Also, tf (Gunyan) is the frequency of appearance of the word digital camera in the text (indicating how many times it appears in the text), so it becomes 2 from the input text, and tf (Gunyan) becomes 2.

  Furthermore, since sf (Gunyan) indicates the appearance frequency in the sentence of the word digital camera (indicating how many sentences have appeared), sf (Gunyan) is 1 from the input text. Since N indicates the total number of sentences in the text, N is 22 from the input text.

  Therefore, log (Gnyan / (sf (Gnyan) +1)) becomes log (22 / (1) +1)).

  As a result, the value of score A (Gunyan) is 10.0928. As an example, if the score A is greater than about 6, and if a word (noun) that is a morpheme indicating the value of the score A is determined as a keyword, Gunyan may be used as a keyword for the input text. it can.

  Similarly, when the noun determined as the keyword candidate is “camera”, the value of the score A (camera) is 6.8896, and when the noun determined as the keyword candidate is “large copy”, the score A ( The value of (large copy) is 4.3467.

  As an example, if the value of score A is greater than about 6, and if a word (noun) that is a morpheme indicating the value of score A is determined as a keyword, “camera” and “large copy” are the texts input above It's hard to be a keyword.

  Even in the case of only score A, the value of score A of “digital camera” is larger than the value of score A of “Gunyan”, and “digital camera” is easily determined as an appropriate keyword. Yes.

  Next, the correction coefficient (score B) of the noun “digital camera” and the noun “Gunyan” is calculated.

The score B (w) is obtained by a division operation of | WmaxA & W | and (| WmaxA | * | W |) ^1/2 (from (Equation 2)).

  In the formula, WmaxA indicates the noun “digital camera” having the highest score A, and | WmaxA & W | indicates an AND search of “digital camera” and the noun “digital camera” or the noun “Gunyan” (character information database (DB ) 20 or the number of hits in the external character information DB (not shown), and | WmaxA | is a single search for the noun “digital camera” (from the character information database (DB) 20 with the keyword extraction server 10 or external) Indicates the number of hits in the character information DB (not shown), and | W | is a single search for the noun “digital camera” or the noun “Gunyan” (from the character information database (DB) 20 with the keyword extraction server 10 or externally) The number of hits in the character information DB (not shown).

In the case of the noun “digital camera”, | WmaxA & W | and (| WmaxA | * | W |) ^1/2 have the same value, so the score B (digital camera) is 1 (see FIGS. 6 and 7). .

  In the case of the noun “Gunyan”, | WmaxA & W |, ie | digital camera & Gunyan |, 24 hits, | WmaxA |, | digital camera |, 113,000,000 hits, | W | Hit.

As a result, the score B (Gunyan) of the noun “Gunyang” is 24 / (113,000,000 * 727) ^1/2 , which is approximately 0.0001 (see FIGS. 6 and 7).

  Similarly, in the case of the noun “camera”, | WmaxA & W |, ie | digital camera & camera |, hits 408,000,000, | WmaxA |, ie | digital camera | hits 113,000,000, and | W | That is, | camera | was hit 310,000,000.

As a result, the score B (camera) of the noun “camera” is 40,800,000 / (113,000,000 * 310,000,000) ^1/2 , which is approximately 0.2141 (see FIGS. 6 and 7). ).

  Similarly, in the case of the noun “large copy”, | WmaxA & W |, ie, | digital camera & large copy | hits 32,800 hits, | WmaxA |, ie | digital camera | hits 113,000,000, and | W | Large photo | has hit 333,000 hits.

As a result, the score B (camera) of the noun “large copy” is 32,800 / (113,000,000 * 333,000) ^1/2 , which is approximately 0.0056 (see FIGS. 6 and 7).

  Next, a corrected score (score C) is calculated from these results.

  Since the correction score (score C) is a value obtained by multiplying the correction coefficient (score B) and the score A as shown in the equation (3), the correction score of the noun “digital camera” (score C (digital camera)) Is 12.0502 * 1 = 12.0502, and the correction score (score C (camera)) of the noun “camera” is 6.8896 * 0.2141 = 1.4751 and the correction score of the noun “large copy” ( The score C (large copy)) is 4.3467 * 0.0056 = 0.0243, and the correction score (score C (Gunyan)) of the noun “Gunyang” is 10.0928 * 0.0001 = 0.001. (See FIGS. 6 and 7).

  As a result, when trying to determine a keyword based on the value of the score A alone, the score A (digital camera) value of the noun “digital camera” and the score A (gunyan) value of the noun “Gunyan” are large values (for example, Therefore, there is a possibility that the noun “digital camera” and the noun “Gunyan” are judged as keywords.

  However, according to the correction coefficient (score B), the score B (Gunyan) value of the noun “Gunyang” is as very small as 0.0001, and the correction coefficient (score B) indicates that it is inappropriate as a keyword. , It became possible to indicate accurately with numbers.

  As a result, the correction score (score C) has a large correction score (score C) value (for example, 10 or more) for the noun “digital camera” that is considered appropriate as a keyword. Therefore, it is possible to provide a keyword extraction server, method, and program capable of easily extracting only nouns (keywords) effective for classification using a correction score (score C) that enables high-speed processing by mathematical formulas and search. it can.

  Further, values of tf (Term Frequency) · idf (Inverse Document Frequency) are shown in FIG. 7 for reference.

  tf is considered to be a word that characterizes the term if a certain term (in this embodiment, the nouns “digital camera”, “camera”, “large copy”, “Gunyan”) appear in the document at a high frequency. Therefore, the value of tf becomes large (indicating the appearance frequency in the document).

  Idf indicates the frequency of the document. If tf is large, the term has an important meaning. For example, a noun such as “ko” appears relatively frequently in a document but does not characterize a specific document. Therefore, by taking the reciprocal (idf) of the appearance frequency df (Document Frequency) of the term in a plurality of documents, the smaller the value of df (the more likely the term characterizing the document), the larger the idf. Extract words that characterize the document.

で示され、式中、ｔｆ（ｗ）は単語ｗのテキスト中での出現頻度（テキスト中に何回出現したかを示す）、ｄｆ（ｗ）は単語ｗの文書集合中での出現頻度（いくつの文書に出現したかを示す）、Ｎは文書集合に含まれる文書の総数を示す。 Specifically, it is calculated by the following equation (4).

Where tf (w) is the appearance frequency of the word w in the text (shows how many times it appears in the text), and df (w) is the appearance frequency of the word w in the document set ( N indicates the total number of documents included in the document set.

  Here, in the case of the noun “digital camera”, tf (digital camera) is 4, df (digital camera) is 97,200,000, and N is 19,200,000,000 (the number considered as the total number of Web documents). Therefore, the tf · idf value is 21.1638.

  In the case of the noun “Gunyang”, tf (Gunyan) is 2, df (Gunyan) is 727, and N is 19,200,000,000 (the number considered as the total number of Web documents)). The tf · idf value is 34.1785.

  Similarly, in the case of the noun “large copy”, the tf · idf value is 10.5224, and in the case of the noun “camera”, the tf · idf value is 8.5419.

  Here, when “camera” and “large copy” are compared, tf · idf is low because even if it is a word with high keyword characteristics such as “camera”, it frequently appears in other articles (documents). It turns out that it is a score (value).

  In addition, in tf · idf, even a word with low keywordity such as “large copy” does not appear so much in other articles (documents), so it can be seen that it has a high score (value).

  However, in this embodiment, when “camera” and “large copy” are compared, the score of “camera” is higher (the correction score of “camera” (score C (camera) is 8.54, “large copy”). The correction score (score C (large copy) is 0.02) is confirmed, and it is confirmed that the keyword candidates are appropriately determined.

  Furthermore, it can be seen that tf · idf has a high score (value) because it does not appear much in other articles (documents) even if it is a low keyword word such as “Gunyan”.

  However, in the present embodiment, the effect of the correction coefficient (score B) (when the relevance with the largest score noun with the highest score A value is high, the correction coefficient (score B) increases and the value of the score A When the relevance to the maximum score noun with the largest is small, the correction coefficient (score B) is small.) As a result, a word with low keywordity such as “Gunyan” has a small correction score (score C). A value is appropriately calculated so as not to be determined as a keyword.

Even in the case of only score A, the value of score A of “digital camera” is larger than the value of score A of “Gunyan”, and “digital camera” is easily determined as an appropriate keyword. Yes.
[How to identify other keywords]

  Furthermore, an example of another keyword specifying method will be described below. For example, in the case where the input text is the sentence shown in FIG. 8, the score A (w) according to the present embodiment for the nouns “earthquake”, “disaster”, “seismic intensity”, and “aftershock” extracted by the noun extraction unit 13. , The score B (w) as the correction coefficient and the score C (w) as the correction score are calculated, and the calculation result will be described.

  When the noun determined as the keyword candidate is “disaster”, the score A (w) is 3.54, the correction coefficient (score B) is 0.29, and the correction score (score C) is 1.03.

  When the noun determined as the keyword candidate is “earthquake”, the score A (w) is 7.24, the correction coefficient (score B) is 1.0, and the correction score (score C) is 7.24. .

  When the noun determined as the keyword candidate is “seismic intensity”, the score A (w) is 3.54, the correction coefficient (score B) is 0.27, and the correction score (score C) is 0.94. .

  When the noun determined as the keyword candidate is “aftershock”, the score A (w) is 4.28, the correction coefficient (score B) is 0.15, and the correction score (score C) is 0.66. .

  When the score A (w) is calculated for the nouns “earthquake”, “disaster”, “seismic intensity”, and “aftershock” as the keyword candidates, “earthquake” has the largest value.

  The term “earthquake” is a highly specialized word because it is often used in special cases, such as when an earthquake occurs. Therefore, “earthquake” is considered as a noun suitable as a keyword candidate.

  The score B (w), which is a correction coefficient, is calculated based on the noun having the largest value of the score A (w), so that “disaster”, “seismic intensity”, “aftershock” co-occurs with the noun “earthquake”. ", The score value of the score B (w) becomes higher.

  Furthermore, since the score C (w) as the correction score uses the calculation result of the score B (w) as the correction coefficient (see Equation (3)), the correction scores for “disaster”, “seismic intensity”, and “aftershock” are used. The value becomes a large value, which indicates that it is possible to appropriately extract keywords from highly specialized words.

  As described above, when this example is analyzed, the word having the highest score A is “earthquake” by the calculation of the score A. The term “earthquake” is highly specialized because it is often used in special cases, such as when an earthquake occurs. Therefore, by calculating the score B, high scores are given to “disaster”, “seismic intensity”, and “aftershock” that often co-occur with earthquakes. In the calculation of score C, since the calculation result of score B is used, it is understood that “disaster”, “seismic intensity”, and “aftershock” have high scores.

[Modification]
There are cases where it is desired to extract keywords from information sources such as newspaper articles, magazine articles, or news information. Also in this case, by extracting information such as newspaper articles, magazine articles, or news information into text, the keyword extraction server 10 according to the present embodiment extracts keywords by calculation using the score A, score B, and score C. can do. The keyword can be selected from a score A or a value having a high score C.

  In some cases, it is desired to extract keywords related to image information such as still images or moving images.

  In this case, the keyword extraction server 10 searches the URL of the target image information, and the keyword extraction server 10 acquires the article information (including the title and snippet) at the top of the search result as text.

  This is because it is considered that related words also appear around the article introducing the URL of the target image information.

  The keyword extraction server 10 searches and acquires the keyword of the image information from the top article information (including title and snippet) of the acquired search result by the calculation using the score A, score B, and score C in the keyword extraction server 10. Can be extracted. The keyword can be selected from a value having a high score A or score C.

  In some cases, it is desired to add a search index to article information. Also in this case, by converting the article information into text, the keyword extraction server 10 according to the present embodiment can extract the keyword by calculation using the score A, score B, and score C.

  In this case, a plurality of keywords can be selected as the search index, and the search index can be set in order from the word having the highest score A or score C value.

  As mentioned above, although embodiment of this invention was described, this invention is not restricted to embodiment mentioned above. The effects described in the embodiments of the present invention are only the most preferable effects resulting from the present invention, and the effects of the present invention are limited to those described in the embodiments of the present invention. is not.

  In the present embodiment, the server 10 has been described as having a hard disk 370 and an optical disk drive 380. However, the present invention is not limited to this, and the server 10 may have a structure without these drive systems, that is, a so-called zero spindle. good. In the case of such a configuration, the contents stored in the hard disk 370 are stored in the large-capacity semiconductor memory 390.

It is a figure which shows the information processing system comprised from the server which concerns on this embodiment, and a user terminal. It is a block diagram which shows the structure of the server which concerns on this invention. It is a functional block diagram which shows the functional structure of the server which concerns on this embodiment. It is a flowchart with which it uses for description about the process sequence by the server which concerns on this embodiment. It is a figure which shows an example of the input text which concerns on this embodiment. It is a figure which shows the calculation example of the score which concerns on this embodiment. It is a figure which shows the example of a calculation result of the score which concerns on this embodiment. It is a figure which shows an example of the other input text which concerns on this embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Information processing system 10 Keyword extraction server 11 Division | segmentation part 12 Morphological extraction part 13 Noun extraction part 14 Operation part 15 Judgment part 16 Selection part 17 Search part 18 Correction coefficient calculation part 19 Correction score calculation part 20 Character information database (DB)
30 User terminal

Claims (6)

A dividing means for dividing the input text by punctuation marks;
Morpheme extraction means for extracting morphemes from the divided parts divided by the dividing means;
Determining a part of speech for the morpheme extracted by the morpheme extraction unit, and extracting a morpheme determined to be a noun;
About the noun extracted by the noun extraction means, the number of characters of the noun, the appearance frequency of the noun in the text, the total number of sentences in the text, and how many sentences the noun appears over Calculating means for calculating a score as a keyword of the noun based on a ratio to the appearance frequency indicating
A keyword extracting device comprising: a determination unit that determines whether or not the noun is a keyword based on the score that is a result of the calculation.   The calculation means includes the number of characters of the noun or a logarithm calculation value around the number of characters of the noun, the appearance frequency of the noun in the text, the total number of sentences in the text, and the number of nouns appearing over the number of the sentences The keyword extraction device according to claim 1, wherein a value obtained by multiplying a ratio with an appearance frequency indicating whether or not or a logarithm calculation value of a number before and after the ratio is used as the score. A character information database storing character information transmitted and received on the Internet;
Selecting means for selecting a maximum score noun with the highest score among the nouns determined to be the keyword by the determining means;
The maximum score noun and the noun are searched in the character information database, and the search number of the maximum score noun, the search number of the noun, and the search number including both the maximum score noun and the noun are searched and investigated. Search means to
A correction coefficient calculating means for calculating a correction coefficient based on the search number of the maximum score noun, the search number of the noun and the search number including both the maximum score noun and the noun;
Correction score calculation means for calculating a correction score based on the correction coefficient and the score calculated by the calculation means;
The keyword extracting apparatus according to claim 1, wherein the determining unit determines whether or not the noun is a keyword based on the correction score. The correction coefficient calculating means divides the number of searches including both the maximum score noun and the noun by the square root of the multiplication calculation value of the search number of the maximum score noun and the search number of the noun. As a correction score,
4. The keyword extracting apparatus according to claim 3, wherein the determining means determines whether or not the noun is a keyword based on a multiplication operation value of the correction score and the score. A splitting process that splits the input text with punctuation marks;
A morpheme extraction step of extracting morphemes of the divided parts divided in the division step;
Determining a part of speech for the morpheme extracted in the morpheme extraction step, and extracting a morpheme determined as a noun;
About the noun extracted in the noun extraction step, the number of characters of the noun, the appearance frequency of the noun in the text, and the total number of sentences in the text and how many sentences the noun appears over A calculation step of calculating a score as a keyword of the noun based on a ratio with an appearance frequency indicating:
And a determination step of determining whether or not to use the noun as a keyword based on the score that is a result of the calculation.   A program for causing a computer to execute the method according to claim 5.

Images