JP2008276561A - Morpheme analysis device, morpheme analysis method, morpheme analysis program, and recording medium with computer program recorded thereon - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a morpheme analysis device for acquiring a proper morpheme analysis result even when any undefined word exists. <P>SOLUTION: This morpheme analysis device is provided with a retrieval result requesting means for, when any undefined word which is not stored in a word dictionary storage means 140 exists in a character string, requesting the retrieval result to an internal or external retrieval device 50 on the basis of the undefined word as retrieval conditions; a document vector calculation means for calculating the whole or a portion of the retrieval result as one document; a similarity calculation means for calculating the similarity of the document vector of the undefined word with the document vector of a known word; a similar word specification means for specifying a similar word as a known word corresponding to the document vector whose similarity is high; and an attribute application means for associating a part of speech and costs of the similar word with the undefined word. The division means is configured to divide the input character string into units by using the part of speech and costs associated with the undefined word by the undefined word attribute application means. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a morpheme analyzer that automatically divides a Japanese sentence into words, a morpheme analysis method, a morpheme analysis program, and a storage medium storing a computer program.

For example, when a character string is input to a Japanese word processor, the character string is divided into appropriate language units, and kanji characters are applied to the language units as necessary. Morphological analysis is performed to divide the string into appropriate language units. In morphological analysis, an input character string may be divided into morphemes that are the smallest meaningful language units, but there are compound words composed of a plurality of morphemes in a word dictionary described later. In some cases, it may be divided into compound words. Therefore, in this specification, “morpheme analysis” is defined as dividing a character string into items (words) of a word dictionary.
Such morphological analysis has an important role as the first stage of processing in machine translation and natural language interface. Hereinafter, “word” is also referred to as “word” as appropriate.
In morphological analysis, a word dictionary and a connectability dictionary are used. The word dictionary designates the part of speech, reading, and utilization type of each word. The concatenation possibility dictionary specifies “types” of two words that can be concatenated. The “type” of a word in the connectability dictionary may be a specific word, a part of speech, or an inflection form. It is assumed that a word that can be at the beginning of a sentence and a word that can be at the end of a sentence can be connected to a special “type” such as “beginning of sentence” and “end of sentence”.
The morphological analysis result is expressed in a graph with words constituting the input character string as nodes (rectangles) and words that can be connected in a positional and grammatical manner as edges (sides). Due to the ambiguity of word division and the ambiguity of isomorphic words, the path from the beginning node to the ending node is enormous. For this reason, using a cost given to a word (hereinafter referred to as “word cost”) and a cost given between two adjacent words (hereinafter referred to as “joint cost”), for example, , N paths are extracted with a path having a low total cost from the beginning of the sentence to the end of the sentence as a priority condition. Here, the “ambiguity of word division” is an ambiguity due to different heading division methods. For example, for the character string “that Japanese”, there may be (a) “that” + “day” + “person” and (b) “that” + “Japanese”. The ambiguity of “isomorphic” is ambiguity due to the presence of words with different readings and parts of speech when the headings are the same. For example, for the heading “ingenuity”, there may be readings “Kuufu” and “Koufu”. The “word cost” is an index indicating how easily the word appears. The “joint cost” is an index indicating how easily adjacent two adjacent words appear. The “word cost” and “joint cost” are set by a statistical method.
In morphological analysis, the path from the beginning of the sentence to the end of the sentence is selected, and a solution expanded as a word string is generated. Therefore, it is difficult to grasp the ambiguity of word division and the ambiguity of related words. On the other hand, the technique which produces | generates the morphological analysis result which isolate | separated the ambiguity of the word division and the ambiguity of the synonym word by grouping a similar word is proposed (for example, patent document 1).
Many of the words that are not described in the word dictionary (hereinafter referred to as “undefined words”) are proper nouns such as personal names, place names, and company names. For this reason, in general, it is assumed that continuous kanji strings, katakana strings, symbol strings, etc. are nouns, and are dealt with by a convenient process such as processing with a uniform cost and part of speech. It has become.
JP2004-30289

  However, when processing an undefined word with a uniform cost and part of speech based on certain assumptions, an appropriate morphological analysis result can be obtained, for example, when the undefined word is not a noun. There is a problem that it may not be possible.

  Therefore, the present invention provides a morpheme analysis apparatus, a morpheme analysis method, a morpheme analysis program, and a storage medium storing a computer program that can obtain an appropriate morpheme analysis result even when an undefined word exists. For the purpose.

  (1) A word dictionary storage unit that stores a plurality of Japanese words in a state in which parts of speech and costs are associated with each other, and a condition that allows the adjacent words to be connected grammatically. Morphological analysis comprising: a concatenation possibility dictionary storage means, and a dividing means for dividing the character string into predetermined units by referring to the word dictionary storage means and the concatenation possibility dictionary storage means for the input character string A known word search result requesting means for requesting a search result from an internal or external search device using a known word that is the word stored in the word dictionary storage means as a search condition, and each known A known word document vector calculating means for calculating a document vector using all or a part of search results for words as one document, and a document vector generated for the known word with respect to the known word. If there is an undefined word that is not stored in the word dictionary storage means in the known word document vector associating means to be attached, and the character string, the internal or external Search result requesting means for requesting a search result from the search apparatus, document vector calculating means for calculating a document vector with all or part of the search results as one document, a document vector of the undefined word, and the known word Similarity calculation means for calculating the similarity of the document vector, similar word specification means for specifying the similar word that is the known word corresponding to the document vector having a high similarity, and the part of speech and cost of the similar word Attribute assigning means for associating with an undefined word, and the dividing means is input using the part of speech and cost associated with the undefined word by the attribute assigning means. The morpheme analyzer is configured to divide the character string into the units.

  According to the invention of (1), the morphological analyzer can give the part-of-speech and cost of a similar word for an undefined word.

  (2) The attribute assigning unit is characterized in that the undefined word is associated with a part of speech and a cost of at least one similar word belonging to the predetermined similarity range ( The morphological analyzer according to 1).

  According to the configuration of (2), multiple types of parts of speech and costs can be associated with undefined words. For this reason, for example, when outputting the morphological analysis results from the beginning of the input character string to the end of the sentence in the order of the lowest total cost, more appropriate multiple paths (paths) are taken into consideration of the ambiguity of undefined words. ) Can be output.

  (3) Further, group the known words into a predetermined group, group document vector generating means for generating a document vector of the group based on the document vector of each known word, the group, and the group Group document vector storage means for associating and storing document vectors to be stored, wherein the attribute assigning means stores the part of speech and cost of the group corresponding to a document vector having a high similarity to the document vector of the undefined word. The morpheme analyzer according to any one of (1) and (2), wherein the morpheme analyzer is configured to be associated with an undefined word.

  According to the configuration of (3), since the part of speech and cost of a group are associated with an undefined word instead of the part of speech and cost of one word, an appropriate part of speech and cost can be associated with an undefined word.

  (4) A word dictionary storage unit that stores a plurality of Japanese words in a state in which parts of speech and costs are associated with each other, and a condition that allows grammatical connection between the adjacent words. Morphological analysis comprising: a concatenation possibility dictionary storage means; and a dividing means for dividing the character string into predetermined units with reference to the word dictionary storage means and the concatenation possibility dictionary storage means for the input character string A known word search result requesting step for requesting a search result from an internal or external search device by using a known word that is the word stored in the word dictionary storage means as a search condition; and a search for each known word A known word document vector calculating step for calculating a document vector with all or part of the result as one document, and associating the document vector generated for the known word with the known word When there is an undefined word that is a word that is not stored in the word dictionary storage means in the intellectual word document vector associating step and the character string, an internal or external word is used as a search condition. A search result requesting step for requesting a search result to the search device, a document vector calculating step for calculating a document vector with all or part of the search results as one document, a document vector of the undefined word, A similarity calculating step for calculating a similarity of a document vector, a similar word specifying step for specifying a similar word that is a known word corresponding to a document vector having a high similarity, a part of speech and a cost of the similar word An attribute assignment step associated with a definition word, and a part of speech and a cost associated with the undefined word in the attribute assignment step. Morphological analysis method characterized by comprising a dividing step of dividing the character string into the unit, the.

  According to the invention of (4), as in the invention of (1), parts of speech and costs of similar words can be assigned to undefined words.

  (5) A word dictionary storage means for storing a plurality of Japanese words in a state in which parts of speech and costs are associated with each other, and a condition for allowing grammatical connection between adjacent words. A connection possibility dictionary storage means, and a dividing means for dividing the character string into predetermined units by referring to the word dictionary storage means and the connection possibility dictionary storage means for the inputted character string. A known word search result requesting step for requesting a search result to an internal or external search device using a known word that is the word stored in the word dictionary storage means as a search condition; and each known word A known word document vector calculating step for calculating a document vector with all or a part of the search results for one document as one document, and the document vector generated for the known word A known word document vector associating step associated with a word, and when an undefined word that is not stored in the word dictionary storage means exists in the character string, the undefined word is used as a search condition. Or a search result requesting step for requesting a search result to an external search device; a document vector calculating step for calculating a document vector with all or a part of the search results as one document; a document vector of the undefined word; A similarity calculation step for calculating the similarity of a document vector of a known word, a similarity word specifying step for specifying a similar word that is the known word corresponding to a document vector having a high similarity, a part of speech and a cost of the similar word Assigning an attribute to the undefined word, and using the part of speech and cost associated with the undefined word in the attribute assigning step. To, morphological analysis program characterized by executing a division step of dividing an input character string to the unit, the.

  According to the present invention, an appropriate morpheme analysis result can be obtained even when an undefined word exists.

  An example of a preferred embodiment according to the present invention will be described below based on the drawings.

[First Embodiment]
(System overview)
FIG. 1 is a schematic diagram showing a morphological analysis system 10 (hereinafter referred to as “system 10”) according to the first embodiment of the present invention.

  As shown in FIG. 1, the system 10 includes a morphological analysis server 20 (hereinafter referred to as “server 20”) and a search server 50. The server 20 and the search server 50 can communicate with each other through a communication line 60 that is, for example, the Internet. The server 20 is a device for morphological analysis of an input character string, and is an example of a morphological analysis device. The server 20 also receives data indicating a character string (data indicating a character string) from an external personal computer (PC) via the communication line 60, performs processing such as translation, and transmits the processed data to the PC or the like. To be sent back to.

  The search server 50 receives a search condition (also referred to as “search word” or “query”) via the communication line 60, and searches for information on a Web site stored using the search condition. This is an apparatus for outputting the URL of a web site related to the search condition, the title of the website, and a snippet (description) as a search result, and is an example of a search apparatus. The search server 50 receives search conditions from the server 20 or an external PC.

  In the present embodiment, the search server 50 is configured as a device external to the server 20, but the search server 50 and the server 20 may be integrated, and the search server 50 may be a search device inside the server 20.

(Main hardware configuration of server 20)
FIG. 2 is a schematic diagram illustrating a main hardware configuration of the server 20. The server 20 is a computer and has a bus 22. The bus 22 includes a CPU (Central Processing Unit) 24, a RAM (Random Access Memory) 26, a ROM (Read Only Memory) 28, an HDD (Hard Disk Drive) 30, a power supply device 32, an input device 34, a communication device 36, and a display. A device 38 is connected. The CPU 24 reads and executes various programs stored in the ROM 28 as appropriate, thereby causing the above-described hardware and the various programs to cooperate with each other to realize various functions according to the present embodiment. The RAM 26 is a local memory used for program execution. The input device 34 accepts input of various data, and may include a keyboard, a pointing device, and the like. The display device 38 displays a screen for accepting data input to the user or displays a calculation result screen by the computer, such as a cathode ray tube display device (CRT) or a liquid crystal display device (LCD). Including a display device.

(Main software configuration of server 20)
FIG. 3 is a schematic diagram illustrating a main software configuration of the server 20. As shown in FIG.
The server 20 includes a known word document vector generation unit 100, a character string reception unit 110, a word division unit 112, a best path search unit 114, an application unit 116, and a word clustering unit 120. The server 20 also has a word dictionary DB 140 and a connectability dictionary DB 142. Each unit described above is realized by various programs stored in the CPU 24 and the ROM 28.

  The known word document vector generation unit 100 is configured to generate a document vector for the words stored in the word dictionary DB 140 as will be described later. The character string accepting unit 110 is configured to accept a character string received from the outside by the communication device 36. The word dividing unit 112 is configured to generate an analysis result by performing a morphological analysis on the character string received by the character string receiving unit 110. The word dividing unit 112 is an example of a dividing unit. The word clustering unit 120 is a configuration for associating parts of speech and costs for undefined words. The best path search unit 114 is configured to specify at least one route (path) based on a predetermined condition for the character string subjected to morphological analysis. The word dictionary DB 140 stores a plurality of Japanese words in a state in which parts of speech and costs are associated with each other, and is an example of a word dictionary storage unit. The connection possibility dictionary DB 142 stores conditions that allow grammatical connection between adjacent words, and is an example of a connection possibility dictionary storage unit.

FIG. 4 is a diagram illustrating an example of the word dictionary DB 140.
As shown in FIG. 4, the word dictionary DB 140 stores a word dictionary.
The word dictionary is configured by associating notation (heading), reading, part of speech, cost, and document vector. The word dictionary is actually converted into a format that can be searched at high speed, such as a Trie structure.

FIG. 5 is a diagram illustrating an example of a connection possibility dictionary. FIG. 6 is a diagram illustrating an example of the connection possibility dictionary DB 142.
First, as a premise for explaining the configuration of the connection possibility dictionary DB 142, the connection possibility dictionary will be described with reference to FIG. The connection possibility dictionary describes information about whether two adjacent words (for example, the left word and the right word) can be connected grammatically. In the concatenation possibility dictionary, for example, in the current Japanese, in horizontal writing, it is usually written from left to right. What type (part of speech or specific word) is shown.

  In the connection possibility dictionary DB 142, a connection possibility dictionary is converted into a connection possibility matrix and stored as shown in FIG. 6 so that the server 20 can perform high-speed processing. In the connectability row example, the row indicates the type of word that appears on the left and the column indicates the type of word that appears on the right. If the word type on the left side and the word type on the right side are connectable, the value of the matrix is set to 1, and if the connection is not possible, the value of the matrix is set to 0.

  A word stored in the word dictionary DB 140 (hereinafter also referred to as “known word”) is associated with a document vector by the known word document vector generation unit 100. The “document vector” is a representation of a document (or sentence) with one vector based on the appearance word. In other words, the “document vector” is a vector whose component is the importance (frequency, etc.) of the appearance word for one document, and is used in the present specification in a normal sense in the field of natural language processing. To do. As illustrated in FIG. 3, the known word document vector generation unit 100 includes a known word search result request unit 102, a known word document vector calculation unit 104, and a known word document vector association unit 106. The known word search result request unit 102 is configured to request a search result from the search server 50 using each known word as a search condition (also referred to as “search word” or “query”). It is an example of a request means. The known word document vector calculation unit 104 is configured to calculate a document vector with a predetermined K (K is a natural number) page of the search results received from the search server 50 as one document. It is an example of a means. For the calculation of the document vector, one set of title and snippet in the search result is set as one page. A snippet is text that is displayed following the title of a search result. The known word document vector associating unit 106 is a configuration for associating each known word with the document vector, and is an example of a known word document vector associating unit.

FIG. 7 is an explanatory diagram of the known word document vector generation unit 100.
The known word search result request unit 102 requests a search result from the search server 50 using known words such as “truck back” and “car” as search conditions. The known word search result request unit 102 receives the search result from the search server 50 as shown in FIG.

式１においては、ＴＦ・ＩＤＦ法によって、図７（ｃ）に示すように、各語の重み付けが行われる。ＴＦ・ＩＤＦ法によって、ある文書における出現頻度が高く、すべての文書のうち特定の文書に偏在する単語が、その文書の特徴を表す単語であると看做されて、重み付けが重くなる。具体的には、例えば、サーバ２０内の図示しない文書記憶ＤＢに記憶した複数の文書データを参照し、各単語の当該文書内における出現回数及び出願文書数を算出し、重要度が算出される。
そして、既知語文書ベクトル関連付け部１０６は、各既知語に対して、その文書ベクトルを関連付ける。 The known word document vector calculation unit 104 specifies the search result of the upper K page from the search result. Then, as shown in FIG. 7B, words other than the query are extracted from the title and snippet of the search result of the upper K page to generate one document. Then, using Equation 1, a document vector is calculated.

In Expression 1, each word is weighted by the TF / IDF method as shown in FIG. According to the TF / IDF method, the occurrence frequency of a certain document is high, and a word that is unevenly distributed in a specific document among all the documents is regarded as a word representing the feature of the document, and the weighting is increased. Specifically, for example, by referring to a plurality of document data stored in a document storage DB (not shown) in the server 20, the number of appearances of each word in the document and the number of application documents are calculated, and the importance is calculated. .
Then, the known word document vector associating unit 106 associates the document vector with each known word.

8, 9, and 10 are explanatory diagrams of the word dividing unit 112.
For example, as illustrated in FIG. 8A, it is assumed that a character string “I am fine with this word” is input to the word division unit 112.
As shown in FIG. 8B, the word dividing unit 112 sets a pointer indicating the position of the character string (between characters, the left side of the character at the beginning of the sentence and the right side of the character at the end of the sentence). As an initial state, the pointer is set to position 0 (the left side of the first character “ko”). In addition, a virtual node called “sentence” is set.
Subsequently, the word division unit 112 searches the word dictionary DB 140 for a word starting from the pointer position (hereinafter referred to as “starting point”). 8 (b), “this (conjunction)” and “ko (suffix: pieces)” are searched for.The word dividing unit 112 shifts the start point backward (right side) one character at a time, A word is exhaustively extracted from the word dictionary DB 140 with reference to the word dictionary DB 140. A process of extracting a word starting from the start point n from the word dictionary DB 140 is referred to as a word extraction process for the start point n (n is 0 and a natural number). "
The word segmentation unit 112 uses a word ending at the start point (in the case of position 0, “sentence”, hereinafter referred to as “preceding word”) and a word starting from the start point (in case of position 0, “this” and “ko”, For each pair of “succeeding words”), the concatenation possibility dictionary DB 142 is referred to, and as shown in FIG. Of the succeeding words, words that are not connectable to any preceding word are excluded. In the example of FIG. 8C, “ko” is excluded.
When the pointer reaches the end of the sentence (position 13 in the example of FIG. 8B), a virtual node “end of sentence” is set, and the word ending at the end of the sentence (in the example of FIG. )) And “end of sentence” are examined, and only those that can be connected are linked to the “end of sentence” node, and the process is terminated.
Finally, the path from the “beginning of sentence” node to the “end of sentence” node becomes the morphological analysis result for the input character string.

The above-described word dividing unit 112 assigns an appropriate cost to each node and link. The cost of each node is stored in the word dictionary DB 140, and the cost of the link is stored in the connection possibility dictionary DB 142 (not shown).
When the antecedent and successor can be concatenated, the antecedent such that the sum of the partial minimum cost from the beginning of the sentence to the antecedent, the concatenation cost between the antecedent and successor, and the word cost of the successor is the smallest As shown in FIG. 9, a special mark is added between the word and the subsequent word. For example, in FIG. 9, the special mark is shown as a thick line.

The best path search unit 114 specifies a predetermined number of paths that are defined in advance, with a low cost as a priority condition. The best path search unit 114 uses, for example, a minimum cost method. Unlike the present embodiment, the best path identification method includes the longest match method, the two-segment longest match method, the minimum morpheme number method, and the minimum clause number method (“Iwanami Laboratory Software Science 15 Natural Language Processing Masao Nagao Iwanami You may use bookstore "etc.).
The application unit 116 is a word processor unit that receives an input of a morphological analysis result (pass) and converts it into, for example, a kanji character as necessary. The application unit 116 includes general word processing software and translation software, and thus description thereof is omitted.

  If there is no undefined word in the character string received by the word dividing unit 112, the morphological analysis can be completed by the above-described processing. On the other hand, when an undefined word exists in the character string, the word clustering unit 120 is activated.

  As shown in FIG. 10A, it is assumed that the character string input to the word dividing unit 112 is, for example, “traversing articles that are interesting”. “Trava” is an undefined word. As shown in FIG. 10C, the word dividing unit 112 extracts words starting from each starting point from the word dictionary DB 140. For convenience of explanation, only one path is described in FIG. 10C and all nodes are linked.

  A word starting from the start point 14 in FIG. 10B cannot be extracted from the word dictionary DB 140. Then, following the word extraction for the start point 14, even if the word extraction process for the start point 15 and the start point 16 is performed, words cannot be extracted from the word dictionary DB 140. Furthermore, when the word extraction process for the starting point 17 is performed, the word “do” can be extracted from the word dictionary DB 140. In this case, the character string “trava” between the starting point 14 where the word cannot be extracted and the starting point 17 where the word can be extracted is an undefined word. The word dividing unit 112 transmits the undefined word to the word clustering unit 120. The word clustering unit 120 is activated by receiving an undefined word.

  As illustrated in FIG. 3, the word clustering unit 120 includes a search result request unit 122, a document vector generation unit 124, a similarity calculation unit 126, a similar word specification unit 128, and an attribute assignment unit 130. The search result request unit 122 is configured to request a search result from the search server 50 using an undefined word as a search condition, and is an example of a search result request unit. The document vector generation unit 124 is a configuration for calculating a document vector using a predetermined K page among search results received from the search server 50 as one document, and is an example of a document vector calculation unit. The similarity calculation unit 126 is a configuration for evaluating the similarity between a document vector of an undefined word and a document vector of a known word, and is an example of a similarity calculation unit. The similar word specifying unit 128 is configured to specify a similar word that is a known word corresponding to the document vector having the highest similarity, and is an example of a similar word specifying unit. The attribute assigning unit 130 is a configuration for associating a similar part of speech and cost with an undefined word, and is an example of an attribute assigning unit.

11 and 12 are explanatory diagrams of the word clustering unit 120. FIG.
The search result request unit 122 requests the search server 50 for a search result by using an undefined word “trava” as a search condition. The search result request unit 122 receives, for example, the search result shown in FIG. 11A from the search server 50.

  The document vector generation unit 124 specifies the search result of the upper K page from the search result. Then, as shown in FIG. 11B, words other than the query are extracted from the title and snippet of the search result of the upper K page, and one document is generated. Then, a document vector is calculated for the one document by using the above-described formula 1.

The similarity calculator 126 uses Equation 2 to calculate the similarity between the “Trava” document vector and the document vector of the known word.

  For example, as shown in FIG. 12A, the similarity between the document vector of “Trava”, which is an undefined word, and the document vector of “Trackback”, which is a known word, is 0.126, which is an undefined word “ The similarity between the document vector of “Traver” and the document vector of “car”, which is a known word, is 0.011.

  The similar word specifying unit 128 sets “track back” to “trava” when the similarity between the document vector of “track back” that is a known word and the document vector of “trava” is the highest among all known words. Identify as a similar term.

The attribute assigning unit 130 assigns the part of speech and cost of the similar term “trackback” to the undefined term “trava”.

  When the word clustering unit 120 assigns the part of speech and the cost to the undefined word in this way, the word clustering unit 120 transmits the undefined word, the part of speech and the cost to the word dividing unit 112. Using the received undefined word, part of speech, and cost, the word dividing unit 112 continues the morphological analysis process starting from the position between the end of the undefined word and the next character.

  The above is the configuration of the server 20. Hereinafter, an operation example of the server 20 will be described mainly using FIGS. 13, 14, and 15. 13, 14, 15, and 16 are mainly flowcharts illustrating an operation example of the server 20.

(Server operation example)
First, the server 20 receives an input of a character string (step S1 in FIG. 13). Subsequently, the server 20 refers to the word dictionary DB 140 and the concatenation possibility dictionary DB 142 and performs morphological analysis of the character string (step S2). Step S2 is an example of a division step. Subsequently, the server 20 outputs a predetermined number of paths under the priority condition that the total cost is small (step S3).
The server 20 also periodically generates and updates document vectors for known words, for example, every 24 hours. Specifically, the server 20 requests a search result from the search server 50 using a known word as a query, and acquires the search result (step S11 in FIG. 14). Step S11 is an example of a known word search result request step. Subsequently, the server 20 extracts terms (words) other than the query from the title and snippet for the top K pages (step S12), weights each term, and generates a document vector (step S13). Steps S12 and S13 are an example of a known word document vector calculation step. Subsequently, the server 20 associates the known word with the document vector and stores it in the word dictionary DB 140 (step S14). Step S14 is an example of a known word document vector association step.

Here, the above-described step S2 will be described with reference to FIGS.
First, the server 20 searches the word dictionary DB 140 for a word (following word) starting from the starting point (step S101 in FIG. 15). Subsequently, it is determined whether or not the start point has come to the end of the sentence (step S102). If it is determined that the start point has not come to the end of the sentence, steps S101 and S102 are repeated. On the other hand, if it is determined in step S102 that the starting point has come to the end of the sentence, it is determined whether or not a word starting from each starting point has been searched from the word dictionary DB 140 (step S103). If it is determined in step S103 that the word has been searched, the part of speech of the known word and the cost are associated with the subsequent word (step S104). On the other hand, if it is determined in step S103 that no word is searched, undefined word processing is performed (step S104A).

Details of the undefined post-processing (step S104A) will be described with reference to FIG.
First, the server 20 requests a search result from the search server 50 using an undefined word as a query, and acquires the search result (step S201 in FIG. 16). Step S201 is an example of a search result request step. Subsequently, for the top K pages of the search results, words other than the query are extracted from the title and snippet (step S202), each word is weighted, and a document vector is generated with the top K page as one document (step S203). . Steps S202 and S203 are an example of a document vector calculation step. Subsequently, the similarity between the document vector of undefined words as a query and the document vectors of all known words stored in the word dictionary DB 140 is calculated (step S204). Step S204 is an example of a similarity calculation step. Subsequently, for the undefined word used as a query, a known word having the highest similarity with the document vector for the undefined word is specified as a similar word (step S205). Step S205 is an example of a similar word specifying step. Subsequently, the server 20 associates the part-of-speech and cost of the similar word with the undefined word as the query (step S206). Step S206 is an example of an attribute assignment step.

  Following step S104 or step S104A, it is determined whether the word ending at the start point (preceding word) and the succeeding word can be connected (step S105). For words ending at the end of the sentence, it is determined whether or not connection with the end of the sentence is possible. If it is determined in step S105 that the connection is possible, a link is established between the preceding word and the succeeding word (step S106). If it is not determined in step S105 that the connection is possible, the subsequent word is excluded (step S106A).

  As described above, the server 20 can give parts of speech and costs of similar words for undefined words. For this reason, even if there is an undefined word, an appropriate morphological analysis result can be obtained.

[Modification]
Next, a modification of the first embodiment will be described.
FIG. 17 is an explanatory diagram of a modification of the first embodiment.

  In the modification, a similarity threshold t for specifying a similar word is set in the similar word specifying unit 128 (see FIG. 3) of the server 20. For example, as shown in FIG. 17, it is assumed that the document vector generation unit 124 (see FIG. 3) generates a document vector a for the word “Stand Bar”. The similarity calculation unit 126 calculates the similarity between the undefined word “Stanbaru” and the known words “Standby”, “Standby”, “Wait”, and the like. The similar word specifying unit 128 specifies a word having a similarity higher than the threshold value t as a similar word. For example, when the “standby” similarity x1, the “waiting” similarity x2, and the “waiting” similarity x3 are larger than the threshold value t, these three words are specified as similar words. Yes.

  Thereby, a plurality of types of parts of speech and costs can be associated with undefined words. For this reason, for example, when outputting the morphological analysis results from the beginning of the input character string to the end of the sentence in the order of the lowest total cost, more appropriate multiple paths (paths) are taken into consideration of the ambiguity of undefined words. ) Can be output.

[Second Embodiment]
Next, the second embodiment will be described focusing on the differences from the first embodiment.
FIG. 18 is an explanatory diagram of the second embodiment.

As shown in FIG. 18A, in the second embodiment, the word dictionary DB 140 of the server 20 stores a document vector of “group” in addition to the word document vector. “Group” means, for example, a set of words having concepts similar to each other such as “standby”, “waiting”, and “waiting”, and is also referred to as “cluster”. The known word document vector calculation unit 104 forms a group of words whose similarity is within a predetermined range in addition to the document vector of each known word, and calculates the document vector of each group (cluster). The document vector of the group is calculated as the sum of the document vectors of the words constituting the group. Specifically, the known word document vector calculation unit 104 calculates the document vector of the group using Equation 3.

  The known word document vector calculation unit 104 is also a group document vector generation unit. The group and the document vector of the group are stored in the word dictionary DB 140. The word dictionary DB 140 is also an example of a group document vector storage unit.

Assume that the document vector generation unit 124 (see FIG. 3) generates the document vector a for the undefined word “stanbaru” as shown in FIG. 18B. The similarity calculation unit 126 uses Equation 4 to calculate the similarity between the undefined word “Stanbaru” and the document vector of the known word group.

  The similar word specifying unit 128 specifies a group having the highest similarity as a similar word.

  As described above, the server 20 according to the second embodiment associates the part of speech and cost of a group with an undefined word instead of the part of speech and cost of one word. Therefore, a more appropriate part of speech and cost for an undefined word. Can be associated.

(About programs and computer-readable recording media)
In the computer, a known word search result request step, a known word document vector calculation step, a known word document vector association step, a search result request step, a document vector calculation step, a similarity calculation step, and a similarity A server control program for executing the word specifying step, the attribute assigning step, the dividing step, and the like can be provided.

  A program storage medium used for installing these server control programs in a computer and making them executable by the computer is, for example, a floppy disk such as a floppy (registered trademark), a CD-ROM (Compact Disc Read Only Memory). ), CD-R (Compact Disc-Recordable), CD-RW (Compact Disc-Rewriterable), DVD (Digital Versatile Disc), etc., as well as semiconductor memory in which programs are temporarily or permanently stored, It can be realized by a magnetic disk or a magneto-optical disk.

  As mentioned above, although embodiment of this invention was described, this invention is not restricted to embodiment mentioned above. The effects described in the embodiments of the present invention are only the most preferable effects resulting from the present invention, and the effects of the present invention are limited to those described in the embodiments of the present invention. is not.

It is the schematic which shows the morphological analysis system which concerns on an example of 1st Embodiment. It is the schematic which shows the main hardware constitutions of a server. It is the schematic which shows the main software structures of a server. It is a figure which shows an example of word dictionary DB. It is a figure which shows an example of a connection possibility dictionary. It is a figure which shows an example of a connection possibility dictionary DB. It is explanatory drawing of a known word document vector production | generation part. It is explanatory drawing of a word division part. It is explanatory drawing of a word division part. It is explanatory drawing of a word division part. It is explanatory drawing of a word clustering part. It is explanatory drawing of a word clustering part. It is a schematic flowchart which shows the operation example of a server. It is a schematic flowchart which shows the operation example of a server. It is a schematic flowchart which shows the operation example of a server. It is a schematic flowchart which shows the operation example of a server. It is explanatory drawing of the modification of 1st Example. It is explanatory drawing of 2nd Example.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Morphological analysis system 20 Morphological analysis server 50 Search server 100 Known word document vector production | generation part 102 Known word search result request part 104 Known word document vector calculation part 106 Known word document vector correlation part 110 Character string reception part 112 Word division part 114 Best Path search unit 116 Application unit 120 Word clustering unit 122 Search result request unit 124 Document vector generation unit 126 Similarity calculation unit 128 Similar word specification unit 130 Attribute addition unit 140 Word dictionary DB
142 Connectability Dictionary DB

Claims (5)

A word dictionary storage means for storing a plurality of Japanese words in a state where parts of speech and costs are associated with each other;
A concatenation possibility dictionary storage means for storing conditions that allow grammatical connection between adjacent words;
A dividing unit that divides the character string into predetermined units with reference to the word dictionary storage unit and the connectability dictionary storage unit for the input character string;
A morphological analyzer having
further,
Known word search result requesting means for requesting a search result to an internal or external search device using a known word that is the word stored in the word dictionary storage means as a search condition;
A known word document vector calculating means for calculating a document vector using all or part of the search results for each of the known words as one document;
A known word document vector associating means for associating a document vector generated for the known word with the known word;
When an undefined word that is not a word stored in the word dictionary storage means exists in the character string, a search result is requested from an internal or external search device using the undefined word as a search condition. Search result request means;
A document vector calculation means for calculating a document vector using all or part of the search results as one document;
Similarity calculation means for calculating the similarity between the document vector of the undefined word and the document vector of the known word;
A similar word specifying means for specifying a similar word that is the known word corresponding to a document vector having a high similarity;
Attribute assigning means for associating the part of speech and cost of the similar word with the undefined word;
Have
The divide unit is configured to divide the input character string into the units using the part of speech and the cost associated with the undefined word by the attribute assigning unit. .   The attribute assigning unit is configured to associate the part-of-speech and cost of at least one similar word belonging to the predetermined similarity range with respect to the undefined word. The morphological analyzer described. The morphological analyzer is
A group document vector generating means for classifying the known words into a predetermined group and generating a document vector of the group based on the document vectors of the known words;
Group document vector storage means for storing the group and a document vector corresponding to the group in association with each other;
Have
2. The attribute assigning unit is configured to associate the part of speech and cost of the group corresponding to a document vector having a high similarity with a document vector of the undefined word with the undefined word. The morphological analyzer according to claim 2. Possibility of connection that stores a word dictionary storage means that stores a plurality of Japanese words in a state in which parts of speech and costs are associated with each other, and a condition that allows grammatical connection between adjacent words A morpheme analyzer comprising: a dictionary storage unit; and a dividing unit that divides the character string into predetermined units with reference to the word dictionary storage unit and the connectability dictionary storage unit with respect to the input character string. A known word search result requesting step for requesting a search result to an internal or external search device using a known word that is the word stored in the word dictionary storage means as a search condition;
A known word document vector calculation step of calculating a document vector by using all or part of the search results for each of the known words as one document;
A known word document vector associating step for associating a document vector generated for the known word with the known word;
When an undefined word that is not a word stored in the word dictionary storage means exists in the character string, a search result is requested from an internal or external search device using the undefined word as a search condition. A search result request step;
A document vector calculation step of calculating a document vector by using all or part of the search results as one document;
A similarity calculation step of calculating a similarity between the document vector of the undefined word and the document vector of the known word;
A similar word specifying step of specifying a similar word that is the known word corresponding to a document vector having a high similarity;
An attribute assignment step for associating the part of speech and cost of the similar word with the undefined word;
A dividing step of dividing the input character string into the units using the part of speech and cost associated with the undefined word in the attribute assigning step;
A morphological analysis method characterized by comprising: On the computer,
Possibility of connection that stores a word dictionary storage means that stores a plurality of Japanese words in a state in which parts of speech and costs are associated with each other, and a condition that allows grammatical connection between adjacent words A morpheme analyzer comprising: a dictionary storage unit; and a dividing unit that divides the character string into predetermined units with reference to the word dictionary storage unit and the connectability dictionary storage unit with respect to the input character string. A known word search result requesting step for requesting a search result to an internal or external search device using a known word that is the word stored in the word dictionary storage means as a search condition;
A known word document vector calculation step of calculating a document vector by using all or part of the search results for each of the known words as one document;
A known word document vector associating step for associating a document vector generated for the known word with the known word;
When an undefined word that is not a word stored in the word dictionary storage means exists in the character string, a search result is requested from an internal or external search device using the undefined word as a search condition. A search result request step;
A document vector calculation step of calculating a document vector by using all or part of the search results as one document;
A similarity calculation step of calculating a similarity between the document vector of the undefined word and the document vector of the known word;
A similar word specifying step of specifying a similar word that is the known word corresponding to a document vector having a high similarity;
An attribute assignment step for associating the part of speech and cost of the similar word with the undefined word;
A dividing step of dividing the input character string into the units using the part of speech and cost associated with the undefined word in the attribute assigning step;
A morpheme analysis program characterized in that is executed.

Images