JP2011232891A - Language analysis program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract an unknown word accurately without using external information and frequency.SOLUTION: A suitability judgment section 14 transmits a word with an unknown word tag to a clause extending section 15 when a morpheme analysis result in a storage section 11 includes the unknown word tag. The clause extending section 15 writes clauses extended to the right side and to the left side or an extended clause into the storage section 11 according to whether the transmitted word includes a delimiter or not. A morpheme analysis section 13 performs a morpheme analysis on the extended clause in the storage section 11 and writes the morpheme analysis result into the storage section 11. The suitability judgment section 14 determines whether the morpheme analysis result of the extended clause and the word with the unknown word tag of the previous morpheme analysis result are matched or not when the morpheme analysis result of the extended clause includes the unknown word tag. When matched, an analysis result output section 16 outputs the previous morpheme analysis result.

Description

  Embodiments described herein relate generally to a language analysis program.

  In general, there are a statistical technique and a linguistic knowledge-based technique for extracting words that are not registered in a dictionary (hereinafter referred to as unknown words or unregistered words) based on Japanese morphological analysis.

  The statistical method is a technique for extracting unknown words based on the word likeness and connection cost of unknown words machine-learned from a large number of corpora. The language knowledge-based technique is a technique for finding an erroneous division of an unknown word and extracting an appropriate unknown word based on large-scale language knowledge such as a dictionary, constraints, and examples.

JP 2003-99426 A JP 200631295 A JP-A-5-67073 JP-A-9-190440

Masanori Ikeya, "Unknown word extraction using probability that character string becomes word", Information Processing Society of Japan report. Natural Language Processing Study Group Report, 2000 (11), pp. 49-54, 2000 Murakiwaki, “Online Unknown Word Acquisition Using Morphological Constraints”, Natural Language Processing, Vol. 17, no. 1,2010

  However, any of the above methods requires the use of detailed external information such as a large corpus, dictionary, language rule, and so on, and accurate unknown word extraction cannot be easily realized. In the case of a statistical method based on frequency, it is possible to recognize frequently used terms (high frequency words), but it is difficult to recognize low frequency words.

  An object of the present invention is to provide a language analysis program capable of easily realizing accurate unknown word extraction without using external information and frequency.

  When the language analysis apparatus according to the embodiment is implemented by a combination of hardware resources and software, a language analysis program for realizing the functions of the language analysis apparatus is installed in a computer in advance.

  The language analysis apparatus according to the embodiment includes a storage unit for storing a target sentence and a morphological analysis result of the target sentence. When a morphological analysis is performed on a target sentence including an unknown word, an unknown word tag attached to the unknown word It is for outputting the morphological analysis result including.

  The language analysis apparatus according to the embodiment includes a storage unit, a morphological analysis unit, an appropriateness determination unit, a phrase expansion unit, and an analysis result output unit.

  The storage unit of the embodiment is for storing the target sentence and the morphological analysis result of the target sentence, and is used as a work memory for various processes by the morphological analysis unit, the appropriateness determination unit, the phrase expansion unit, and the analysis result output unit. Also works.

  The morphological analysis unit of the embodiment writes a morphological analysis result in which a part-of-speech tag is added to each word constituting the target sentence in the storage unit by performing morphological analysis on the target sentence in the storage unit.

  The suitability determination unit of the embodiment determines whether or not an unknown word tag of the part-of-speech tag is included in the morphological analysis result written in the storage unit. If the unknown word tag is included, the unknown word The word with the tag is sent to the phrase extension section.

  The phrase expansion unit of the embodiment determines whether or not the transmitted word includes a delimiter corresponding to a particle, a punctuation mark, a symbol, a sentence head, or a sentence end. As a result of the determination, the word includes a delimiter. In this case, the left extension clause is created, the right extension clause is created, and the right extension clause and the left extension clause are written in the storage unit.

  According to the embodiment, when a word includes a delimiter, the left extension clause is defined as a left delimiter closest to the left of the included delimiter and a left delimiter as a right delimiter. It consists of a character string up to the right delimiter.

  According to the embodiment, when the word includes a delimiter, the right-side extended clause uses the left delimiter as the delimiter closest to the right of the included delimiter and the left delimiter. To the right delimiter.

  On the other hand, when the word expansion part of the embodiment does not include a delimiter, the delimiter closest to the left of the word used for the determination is defined as the left delimiter, and the delimiter closest to the right of the word is delimited as the right Then, an extended clause composed of a character string from the left break to the right break is created, and this extended clause is written in the storage unit.

  The morpheme analysis unit according to the embodiment writes a morpheme analysis result in which a part-of-speech tag is added to a word constituting an extended clause in the storage unit by performing a morphological analysis on the extended clause in the storage unit.

  The appropriateness determination unit of the embodiment determines whether or not an unknown word tag is included in the morphological analysis result of the extended clause, and if the unknown word tag is included, the morphological analysis result of the extended clause and the previous morpheme It is determined whether or not the word to which the unknown word tag is assigned in the analysis result matches.

  If the result of this determination is negative, the appropriateness determination unit of the embodiment sends the word to which the unknown word tag is assigned in the morpheme analysis result of the extension clause to the clause extension unit.

  On the other hand, the analysis result output unit of the embodiment, as a result of the determination by the appropriateness determination unit, the morpheme analysis result of the extended clause and the word to which the unknown word tag of the previous morpheme analysis result is matched, Output previous morphological analysis result.

It is a block diagram which shows the structural example of the language analyzer which concerns on 1st Embodiment. It is a schematic diagram which shows the extended clause production | generation rule in the embodiment. It is a flowchart for demonstrating an example of the operation | movement in the embodiment. It is a schematic diagram for demonstrating the example of a data transition in the embodiment. It is a flowchart for demonstrating an example of operation | movement of the phrase expansion part in the embodiment. It is a schematic diagram for demonstrating the example of a data transition in the embodiment. It is a block diagram which shows the structural example of the language analyzer which concerns on 2nd Embodiment. It is a flowchart for demonstrating an example of the operation | movement in the embodiment. It is a schematic diagram for demonstrating the example of a data transition in the embodiment. It is a schematic diagram for demonstrating the example of a data transition in the embodiment. It is a schematic diagram for demonstrating the example of a data transition in the embodiment. It is a schematic diagram for demonstrating the variation of a general analysis mistake.

  Each embodiment will be described below with reference to the drawings. Note that the language analysis device of each embodiment can be implemented with either a hardware configuration or a combined configuration of hardware resources and software. As the software of the combined configuration, a language analysis program that is installed in advance from a network or a storage medium into a computer serving as a language analysis device and realizes the functions of the language analysis device is used. This is common to each embodiment.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration example of the language analysis apparatus according to the first embodiment. This language analysis apparatus is for outputting a morphological analysis result including an unknown word tag attached to an unknown word when a morphological analysis is performed on a target sentence including the unknown word.

  Specifically, the language analysis apparatus includes a storage unit 11, an input unit 12, a morpheme analysis unit 13, an appropriateness determination unit 14, a phrase expansion unit 15, and an analysis result output unit 16, and each unit 11 to 16 uses a bus 17. Are connected to each other.

The storage unit 11 is a storage medium configured by hardware such as a hard disk drive or a nonvolatile memory device, and stores sentence data to be analyzed (hereinafter referred to as a target sentence) and a morphological analysis result of the target sentence. Is for. The storage unit 11 stores a control program to be executed by the morphological analysis unit 13, the appropriateness determination unit 14, the phrase expansion unit 15, and the analysis result output unit 16. Furthermore, the storage unit 11 stores an analysis dictionary for morphological analysis, a determination rule for determining appropriateness, and a rule for creating an extended phrase. The storage unit 11 also functions as a work memory for various processes performed by the morphological analysis unit 13, the appropriateness determination unit 14, the phrase expansion unit 15, and the analysis result output unit 16.
The input unit 12 is an input interface for inputting a target sentence from the outside and writing it in the storage unit 11. For example, a receiving device that receives a target sentence from a network (not shown) and can write it in the storage unit 11, not shown A storage medium holding device for detachably holding a computer-readable storage medium, reading out the target sentence from the storage medium and writing it in the storage unit 11, and inputting the target sentence in response to a user operation and writing it in the storage unit 11 Therefore, an input device such as a keyboard can be used as appropriate. As the language analysis apparatus, morphological analysis can be performed if the target sentence is stored in the storage unit 11, and any method can be used for writing to the storage unit 11.

  The morphological analysis unit 13 performs a morphological analysis on the target sentence in the storage unit 11, thereby writing a morpheme analysis result obtained by adding a part of speech tag to each word constituting the target sentence to the storage unit 11, and a storage unit 11 has an expanded phrase analysis function for writing a morpheme analysis result in which a part-of-speech tag is added to a word constituting an expanded phrase to the storage unit 11 by performing a morphological analysis on the expanded phrase in 11. Specifically, the morpheme analysis unit 13 performs word division and part-of-speech tagging on the target sentence in the storage unit 11 while referring to the morpheme analysis dictionary stored in the storage unit 11, and displays the analysis result. Store in the storage unit 11. Note that word division may be referred to as morpheme division. Further, “a part of speech tag is assigned to a word” may be read as “a part of speech tag is associated with a word”. The part of speech tag may be called part of speech tag information. In each embodiment, the morpheme analysis method, the analysis dictionary format, and the analysis result format are not limited.

  The appropriateness determination unit 14 determines whether or not there is an analysis failure (for example, the presence or absence of an unknown word) based on the determination rule of the appropriateness determination stored in the storage unit 11 for the result analyzed by the morphological analysis unit 13. Check and determine the appropriateness of the analysis results. The determination result is stored in the storage unit 11. Note that “appropriateness determination” may be read as “determination of presence / absence of unknown word”, and “presence / absence of unknown word” may be read as “whether an unknown word tag is included”. In the present embodiment, the appropriateness determination unit 14 has the following functions (f14-1) to (f14-3).

  (f14-1) For the morphological analysis result written in the storage unit 11 by the target sentence analysis function, it is determined whether or not an unknown word tag is included in the part of speech tag. An unknown word determination function for sending a morphological analysis result to the phrase expansion unit 15. The unknown word tag may be read as unknown word tag information, unknown word label, unknown word label information, or the like.

  (f14-2) It is determined whether or not an unknown word tag is included in the morphological analysis result written in the storage unit 11 by the extended phrase analysis function. If an unknown word tag is included, the morphological analysis result of the extended phrase And an analysis result determination function for determining whether or not a word to which an unknown word tag is assigned in the previous morpheme analysis result matches. Note that “a word with an unknown word tag” may be read as “a word associated with an unknown word tag”.

  (f14-3) A delimiter character determination transmission function for transmitting the morpheme analysis result of the extended clause to the delimiter character determination function of the phrase extension unit 15 when the result of the determination by the analysis result determination function is negative.

  The phrase expansion unit 15 extracts an analysis failure location and extracts an extended phrase including the analysis failure location for the analysis result determined to be inappropriate by the appropriateness determination unit 14, and the extraction result is stored in the storage unit 11. To remember. An analysis failure point means a character string that is dead due to word division or a character string that is not found in the analysis dictionary and that is not given a part-of-speech tag, but can usually be referred to as an unknown word. The extended clause is a character string of a minimum unit including a parsing failure part and delimited by a delimiter defined by an extended rule stored in the storage unit 11 as shown in FIG. The extension rule is not limited to being stored in the storage unit 11 but may be described in advance as a part of a program for realizing the function of the phrase extension unit 15. In this embodiment, the phrase expansion unit 15 has the following functions (f15-1) to (f15-3).

  (f15-1) A word with an unknown word tag is extracted from the morphological analysis result sent from the appropriateness determination unit 14, and a delimiter character corresponding to a particle, a punctuation mark, a symbol, a sentence head, or a sentence end is extracted from this word. Delimiter determination function that determines whether or not it is included. Note that the range of delimiters is not limited to particles, punctuation marks, symbols, sentence heads, and sentence endings. For example, any character can be used as long as it is a character used to delimit sentences, such as conjunctions or numbers. ing.

  (f15-2) As a result of determination by this delimiter determination function, if a delimiter is included in a word, a left extension clause is created, a right extension clause is created, a right extension clause and a left extension clause are created. A first extended phrase creating function for writing the information into the storage unit 11.

  Here, if the word contains a delimiter, the left-side extended clause uses the left delimiter as the delimiter closest to the left of the included delimiter and the left delimiter as the right delimiter. It consists of a character string up to.

  The expanded clause on the right side, when a word includes a delimiter, includes the left delimiter as the included delimiter, and the right delimiter as the delimiter closest to the right of the included delimiter, from the left delimiter to the right delimiter. Consists of a string.

  (f15-3) If the result of the determination by the delimiter determination function is NO, the delimiter closest to the left of the word used for the determination is the left delimiter, and the delimiter closest to the right of the word is the right delimiter A second extended clause creating function for creating an extended clause composed of a character string from the left break to the right break and writing the extended clause in the storage unit 11.

  The analysis result output unit 16 outputs the morphological analysis result of the sentence determined to be appropriate by the appropriateness determination unit 14. For example, as a result of determination by the analysis result determination function of the appropriateness determination unit 14, the analysis result output unit 16 (the morpheme analysis result of the extended clause and the word to which the unknown word tag is assigned in the previous morpheme analysis result) When both match, the analysis result output function for outputting the previous morpheme analysis result is provided.

  Next, the operation of the language analyzer configured as described above will be described with reference to the flowchart of FIG.

  In the language analysis device, the target sentence is input and written in the storage unit 11 by the operation of the input unit 12 by the user (S11). The target sentence may be a sentence directly input by a user from a keyboard or the like, or may be a sentence set (document) read from an existing file. In the case of a sentence set, it is necessary to process the target sentence in sentence units.

  Next, the morpheme analysis unit 13 performs morpheme analysis including word division and part-of-speech tagging on the target sentence in the storage unit 11, and writes the morpheme analysis result in the storage unit 11 (S12). In the present embodiment, as shown in FIG. 4 and the like, the result of word division is indicated by “/”, and the part-of-speech tags are indicated by “<” and “>”. Of the part-of-speech tags, an unknown word tag representing an unknown word is indicated as <not>. The part-of-speech tag may be referred to as a part-of-speech label, and the unknown word tag may be referred to as an unknown word label.

  Next, the appropriateness determination unit 14 determines whether or not the analysis result is appropriate based on the determination rule stored in the storage unit 11 for the morphological analysis result in the storage unit 11. For example, when the determination rule is set to the effect that “determined as appropriate when an unknown word tag is not included in the morphological analysis result, and determined as inappropriate when an unknown word tag is included”, As illustrated in FIG. 4, the appropriateness determination unit 14 determines that the morphological analysis result of Example 11 that does not include the unknown word tag “<un>” is appropriate, and includes the unknown word tag “<un>”. Twelve morphological analysis results are determined to be inappropriate.

  When it is determined to be appropriate, the appropriateness determination unit 14 sends the morphological analysis result to the analysis result output unit 16. If it is determined as inappropriate, the appropriateness determination unit 14 sends the morphological analysis result to the phrase expansion unit 15 (S13).

  Note that the above-described determination rule is just one example. The determination rule may be set by notation and numerical values other than “unknown word” as long as it is a content for determining whether or not there is an unknown word or an unregistered word.

  As shown in FIG. 5, the phrase expansion unit 15 extracts the analysis failure part from the sent morpheme analysis result, and performs the following steps while referring to the creation rule in the storage unit 11 as shown in FIG. S14-2 to S14-5 are executed.

  In other words, the phrase expansion unit 15 determines whether or not a delimiter is included in the analysis failure location when creating the expanded phrase (S14-2).

  As a result of the determination, if a delimiter is included, the extended clauses on both sides are cut out using the character as a delimiter, and the delimiter between the two extended clauses is determined (S14-3).

  For example, in the case of Example 13 shown in FIG. 6, since the analysis failure point “only one side” includes the particle “shika” as the delimiter, the delimiter “sentence” closest to the left of the included delimiter “shika” A separator is used, and the included delimiter character “shika” is a right separator (extended clause on the left). Further, the included delimiter “shika” is the left delimiter, and the particle “ga” that is the delimiter closest to the right of the included delimiter “shika” is the right delimiter (the extended clause on the right).

  On the other hand, as a result of the determination, if the delimiter is not included, the character string including the analysis failure location is cut out from the delimiter closest to both sides of the analysis failure location, and the extension clause delimitation is determined (S14-4). .

  For example, in the case of Example 12 shown in FIG. 6, the analysis failure point “e” does not include any delimiters such as particles, punctuation marks, symbols, sentence heads, and sentence endings. The near delimiter particle “NO” is the left delimiter, and the delimiter “O” closest to the right of the word “E” is the right delimiter.

  The phrase expansion unit 15 creates an extended phrase composed of the character string between the breaks determined in step S14-3 or S14-4 (S15), and writes this extended phrase in the storage unit 11. For example, in the case of Example 13 shown in FIG. 6, an extended phrase on the left side consisting of the character string “one side” from the left delimiter “sentence” to the right delimiter “shika” is created, and the left delimiter “sika” to the right delimiter “ A right expanded phrase consisting of the character string “edit type data” up to “is created” is created. Further, in the case of Example 12 shown in FIG. 6, an extended clause composed of the character string “color change” from the left delimiter “NO” to the right delimiter “NO” is created.

  Next, the morpheme analysis unit 13 again performs morpheme analysis including word division and part-of-speech tagging on the extended clause in the storage unit 11 and writes the morpheme analysis result in the storage unit 11.

  The appropriateness determination unit 14 determines whether or not the morphological analysis result is appropriate based on the determination rule for the morphological analysis result of the extended clause written in the storage unit 11.

  For example, if the decision rule is “If there is no unknown word in the morphological analysis result of the extended clause, it is determined as appropriate. Also, even if there is an unknown word, the morphological analysis result of the extended clause matches the previous morphological analysis result. It is determined that it is determined to be appropriate if there is an unknown word and it does not match the previous morphological analysis result.

  In the case of this setting, the appropriateness determination unit 14 determines that the extended clause of Example 12 shown in FIG. 4 is appropriate, and sends the morpheme analysis result of the extended clause and the previous morpheme analysis result to the analysis result output unit 16. . On the other hand, if it is determined as inappropriate, the appropriateness determination unit 14 sends the morphological analysis result to the phrase expansion unit 15. In this case, the language analysis apparatus repeatedly executes the processing from step S14 to step S13 through step S12.

  Finally, the analysis result output unit 16 compares the morpheme analysis result of the extended clause with the previous morpheme analysis result against the morpheme analysis result determined to be appropriate by the appropriateness determination unit 14. When the extended clause is not created or when the morphological analysis result of the extended clause matches the previous morphological analysis result, the morphological analysis result is output as it is.

  If the morpheme analysis result of the extended clause does not match the previous morpheme analysis result, the analysis result output unit 16 replaces the morpheme analysis result of the extended clause with the previous morpheme analysis result and outputs the result (S15).

  As described above, according to the present embodiment, the unknown word extraction accuracy is improved by performing phrase expansion of the unknown word and recursively optimizing the morphological analysis result for the morphological analysis result including the unknown word. High morphological analysis results can be obtained.

(Second Embodiment)
FIG. 7 is a block diagram illustrating a configuration example of the language analysis apparatus according to the second embodiment. The same reference numerals are given to the functional units that are substantially the same as those in FIG. 1, and detailed descriptions thereof are omitted. Is mainly described.

  That is, the language analysis apparatus 20 according to the present embodiment includes an unknown word extraction unit 21 and a completely unknown word determination unit 22 instead of the appropriateness determination unit 14 as shown by being surrounded by a broken line as compared with the configuration illustrated in FIG. In addition, an unregistered word output unit 23 is provided instead of the analysis result output unit 16. Further, along with the change of each of these units 21 to 23, the language analysis device is a device for outputting the unknown word when the morphological analysis is performed on the target sentence including the unknown word. Similarly, the program and data stored in the storage unit 11 are changed with the change of the units 21 to 23. For example, the storage unit 11 stores a control program to be executed by the morphological analysis unit 13, the phrase expansion unit 15, the unknown word extraction unit 21, the completely unknown word determination unit 22, and the unregistered word output unit 23. Further, the storage unit 11 stores an analysis dictionary for morphological analysis, a completely unknown word determination condition, and an extended phrase creation rule. The storage unit 11 also functions as a work memory for various processes by the morphological analysis unit 13, the phrase expansion unit 15, the unknown word extraction unit 21, the complete unknown word determination unit 22, and the unregistered word output unit 23. In addition, with the change of each unit 21 to 23, the phrase expansion unit 15 changes the data input / output destination.

  Here, the unknown word extraction unit 21 determines whether or not an unknown word tag is included in the morpheme analysis result written in the storage unit 11 by the morpheme analysis unit 13. It has an unknown word tag determination function for writing a word with a word tag (extracted from the morphological analysis result) to the storage unit 11.

  The complete unknown word determination unit 22 has a function of determining whether or not a complete unknown word determination condition is satisfied for a word (= result of morphological analysis to be determined) with an unknown word tag in the storage unit 11. .

  Here, the completely unknown word determination condition is a condition that the following first condition [1] and second condition [2] are simultaneously satisfied. That is, the determination by the completely unknown word determination unit 22 means that it is determined whether or not the following first condition [1] and second condition [2] are simultaneously satisfied.

  Completely unknown word determination condition [1]: Not one-character hiragana or katakana.

  Completely unknown word determination condition [2]: The morpheme analysis result is composed of a single morpheme, and the part of speech is an unknown word.

  In accordance with the complete unknown word determination condition, for example, when the evaluation target is “all inspection <un>”, it is determined as a complete unknown word. A completely unknown word is a word that satisfies all the completely unknown word determination conditions. “<Not>” is an unknown word tag. Further, when the evaluation target is “e <un>”, “a <un>”, “total <name>”, “→ <symbol> special speed <un>”, it is determined that it is not a completely unknown word.

  In addition, as a result of the determination by the complete unknown word determination means, the unknown character tag is determined as a result of determination by the complete unknown word determination unit (f15-1) due to the provision of the complete unknown word determination unit 22. It is a function for determining whether or not a given word includes a delimiter corresponding to a particle, a punctuation mark, a symbol, a sentence head, or a sentence end. The first extended phrase creating function (f15-2) and the second extended phrase creating function (f15-3) of the phrase expanding unit 15 are as described above.

  The unregistered word output unit 23 outputs the morpheme sent from the complete unknown word determination unit 22 as an unregistered word. Further, the types of unregistered words are divided according to the timing and type of determination, and different labels are assigned to each. Specifically, the unregistered word output unit 23 has the following functions (f23-1) to (f23-2).

  (f23-1) As a result of determination by the unknown word extraction unit 21 and the completely unknown word determination unit 22 with respect to the morpheme analysis result written in the storage unit 11 by the extended clause analysis function (f13-2) of the morpheme analysis unit 13, A function of assigning an unregistered word label to the word that is an unknown word and outputting it when the word with the unknown word tag satisfies the first condition [1] and the second condition [2] simultaneously. It should be noted that “... with label” may be read as “... in association with label information”. This is not limited to unregistered word labels, but the same applies to other labels.

  (f23-2) As a result of determination by the unknown word extraction unit 21 and the completely unknown word determination unit 22 with respect to the morpheme analysis result written in the storage unit 11 by the extended clause analysis function (f13-2) of the morpheme analysis unit 13, A function of adding and outputting a label indicating an input error to the word that is an unknown word when the word to which the unknown word tag is assigned does not satisfy the first condition [1] and the second condition [2] at the same time.

  Next, the operation of the language analyzer configured as described above will be described with reference to the flowchart of FIG.

  First, steps S21 and S22 are executed in the same manner as steps S11 and S12 described above. Thus, after the target sentence is input and written in the storage unit 11, the morphological analysis result of the target sentence is written in the storage unit 11.

  Here, the unknown word extraction unit 21 determines whether or not there is a morpheme whose part of speech is an unknown word by determining whether or not the unknown word tag is included in the morphological analysis result in the storage unit 11. To do. If the unknown word tag is not included and the unknown word does not exist, the processing for the morphological analysis result of the target sentence ends, and the language analysis apparatus shifts to input of the next target sentence without outputting anything (shown in FIG. 9). Example 21). If there is an unknown word tag and there is an unknown word, the unknown word extraction unit 21 writes the word with the unknown word tag in the storage unit 11 and sends it to the morpheme analysis unit 13 (S23). For example, in the case of the example 22 shown in FIG.

  Next, the morpheme analysis unit 13 performs morpheme analysis on the sent unknown word, writes the morpheme analysis result in the storage unit 11, and sends it to the completely unknown word determination unit 22 (S24).

  The completely unknown word determination unit 22 determines whether or not the completely unknown word determination condition is satisfied in order to determine whether or not the morphological analysis result of only the transmitted unknown word is a completely unknown word. In the case of (Example 22 shown in FIG. 9), the corresponding unknown word is sent to the unregistered word output unit 23.

  On the other hand, as a result of the determination, if it is not a completely unknown word, the completely unknown word determining unit 22 sends the corresponding unknown word to the phrase expanding unit 15 (S25). For example, in the case of Example 23 and Example 24 shown in FIG. 10 and Example 25 shown in FIG. 11, unknown words that are not completely unknown words are “minute”, “only one face”, and “done”, respectively. Yes.

  The phrase expansion unit 15 creates an expanded phrase based on the sent unknown word and the morphological analysis result of the target sentence in the storage unit 11. The method for creating the extended phrase is the same as that in step S14 described above.

  For example, in the case of Example 23 shown in FIG. 10, the analysis failure point “minute” does not include any delimiters such as particles, punctuation marks, symbols, sentence heads, and sentence endings. The near delimiter particle "beginning" is the left delimiter, the word delimiter "o" closest to the right of the word "minutes" is the right delimiter, and the string from the left delimiter "beginning" to the right delimiter "o" is "slow" Create an extended clause consisting of “hours and minutes”.

  In the case of Example 24 shown in FIG. 10, the left extension phrase “one page” is created and the right extension phrase “Editing type data” is created as in Example 13 described above.

  In the case of Example 25 shown in FIG. 11, since the analysis failure point “deta” does not include any delimiters such as particles, punctuation marks, symbols, sentence heads, and sentence endings, the left side of the word “dere” used in the determination The delimiter particle “ga” that is closest to the left is the left delimiter, the delimiter “ga” that is closest to the right of the word “deta” is the right delimiter, and the left delimiter “ga” to the right delimiter “ga” Create an expanded clause consisting of the string "each saved diamond".

In any case, the phrase expansion unit 15 writes the created expansion phrase in the storage unit 11 and sends it to the morpheme analysis unit 13. (S26)
The morpheme analysis unit 13 performs morpheme analysis on the transmitted extended clause, writes the obtained morpheme analysis result in the storage unit 11, and sends it to the unknown word extraction unit 21 (S27).

  The unknown word extraction unit 21 determines whether or not there is a morpheme whose part of speech is an unknown word by determining whether or not the unknown word tag is included in the morphological analysis result of the transmitted extended clause. . If the unknown word tag is not included and the unknown word does not exist, the language analysis apparatus finishes the processing for the input target sentence, outputs nothing, and proceeds to input of the next target sentence (shown in FIG. 10). Example 23). When there is an unknown word tag and an unknown word exists, the unknown word extraction unit 13 writes the unknown word in the storage unit 11 and sends it to the morpheme analysis unit 13 (example 23 shown in FIG. 9 and shown in FIG. 10). Example 24) (S28). For example, in the case of Example 24 shown in FIG. 10 and Example 25 shown in FIG. 11, the unknown words are “knitting type” and “done”, respectively.

  The morpheme analysis unit 13 performs morphological analysis on the sent unknown word, writes the obtained morpheme analysis result in the storage unit 11, and sends it to the completely unknown word determination unit 22 (S29).

  The completely unknown word determination unit 22 determines whether or not the completely unknown word determination condition is satisfied in order to determine whether or not the morphological analysis result of only the transmitted unknown word is a completely unknown word. In this case (“knitting type” in Example 24 shown in FIG. 10), a label indicating “unregistered word” is assigned to the corresponding unknown word and sent to the unregistered word output unit 23. As shown in Example 24, a label indicating an extended unknown word may be used as a label indicating an unregistered word.

  On the other hand, as a result of the determination, when the completely unknown word determination condition is not satisfied and the word is not a completely unknown word (“Done” in Example 25 shown in FIG. 11), the completely unknown word determining unit 22 sets the corresponding unknown word. A label indicating “input error” is given and sent to the unregistered word output unit 23 (S30).

  The unregistered word output unit 23 outputs the unknown word sent from the complete unknown word determination unit 22 as an unregistered word (S31).

  For example, if the completely unknown word determination unit 22 gives different labels to the unknown word determined to be “unregistered word” and the unknown word determined to be “input mistake”, in step S31, the unregistered word And input mistakes can be output separately.

  As described above, according to the present embodiment, similar to the first embodiment, the morphological analysis is performed recursively by executing the phrase expansion of the unknown word on the morphological analysis result including the unknown word. Unregistered words can be extracted while reducing unknown word analysis errors.

  Further, according to the present embodiment, since the completely unknown word is determined by the determination using the completely unknown word determination condition, the unknown word extraction accuracy can be improved as compared with the first embodiment.

  Further, according to the present embodiment, when a completely unknown word is determined and an unknown word generated due to an input error is distinguished from an unknown word generated due to an unregistered word, the input error is found. Can do.

  Moreover, according to each embodiment as described above, accurate unknown word extraction can be easily realized without using large-scale external information and frequency such as a dictionary and a corpus. Furthermore, the accuracy of morpheme analysis can be improved by incorporating the appropriate morpheme analysis result into the original morpheme analysis result.

Here, the analysis mistake which each embodiment reduces is demonstrated supplementarily.
In general, in the field of language analysis, unregistered words based on morphological analysis are additionally registered in a dictionary from the viewpoint of improving analysis accuracy. In many cases, unregistered words include highly specialized terms, new words, or terms that are uniquely defined by the user. For this reason, additionally registering unregistered words is also useful when extracting specialized knowledge.

  Unregistered words are usually output in the form of “unknown words” by morphological analysis. For this reason, the method of extracting an unknown word as an unregistered word is simple. However, the morphological analysis result may include an unknown word other than an unregistered word due to an analysis error that causes an analysis failure or a dead end somewhere in the sentence. Since the unknown word due to the analysis error is not an unregistered word, the extraction accuracy of the unregistered word is lowered.

  Therefore, in order to improve the extraction accuracy of unregistered words, it is necessary to reduce unknown words due to analysis errors from the morphological analysis results. As shown in FIG. 12, variations in analysis errors are defined as four types: over-judgment, misalignment, over-coupling, and over-division.

  The analysis error due to the fourth excessive division can be reduced by the conventional method, but the analysis error due to excessive determination, positional deviation, and excessive coupling cannot be reduced by the conventional method. As a result of the verification experiment, it is known that about 20% of the analysis errors are overdivision, and the remaining 80% are overdetermination, misalignment, and overcoupling. According to the method of each embodiment, the remaining 80% of excess determination, misalignment, and analysis errors due to excessive coupling can be reduced.

  Note that the method described in the above embodiment includes a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), a magneto-optical disk (MO) as programs that can be executed by a computer. ), And can be distributed in a storage medium such as a semiconductor memory.

  In addition, as long as the storage medium can store a program and can be read by a computer, the storage format may be any form.

  In addition, an OS (operating system) running on a computer based on an instruction of a program installed in the computer from a storage medium, MW (middleware) such as database management software, network software, and the like realize the above-described embodiment. A part of each process may be executed.

  Furthermore, the storage medium in the above embodiment is not limited to a medium independent of a computer, but also includes a storage medium in which a program transmitted via a LAN, the Internet, or the like is downloaded and stored or temporarily stored.

  Further, the number of storage media is not limited to one, and the case where the processing in the above embodiment is executed from a plurality of media is also included in the storage medium in the embodiment, and the medium configuration may be any configuration.

  The computer in the above embodiment executes each process in the above embodiment based on a program stored in a storage medium, and a single device such as a personal computer or a plurality of devices are connected to a network. Any configuration such as a system may be used.

  In addition, the computer in the above embodiment is not limited to a personal computer, but includes an arithmetic processing device, a microcomputer, and the like included in an information processing device. ing.

  In addition, the said embodiment is not limited to a form as it is, In an implementation stage, a component can be deform | transformed and embodied in the range which does not deviate from the summary. Moreover, various modifications can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  DESCRIPTION OF SYMBOLS 11 ... Memory | storage part, 12 ... Input part, 13 ... Morphological analysis part, 14 ... Appropriateness determination part, 15 ... Phrase expansion part, 16 ... Analysis result output part, 17, 28 ... Bus, 21 ... Unknown word extraction part, 22 ... completely unknown word determination unit, 23 ... unregistered word output unit.

Claims (3)

A storage means for storing the target sentence and the morphological analysis result of the target sentence is provided, and when a target sentence including an unknown word is analyzed, a morphological analysis result including an unknown word tag attached to the unknown word is output. A language analysis program used in a language analysis device for
The language analysis device;
A target sentence analysis unit that writes a morpheme analysis result in which a part of speech tag is added to each word constituting the target sentence by performing a morphological analysis on the target sentence in the storage unit;
For the written morpheme analysis result, determine whether or not an unknown word tag of the part of speech tag is included, and if an unknown word tag is included, an unknown word determination means for sending the morpheme analysis result,
A delimiter character that extracts a word with an unknown word tag from the sent morpheme analysis result and determines whether or not a delimiter character corresponding to a particle, punctuation mark, symbol, sentence head, or sentence end is included in this word Determination means,
As a result of this determination, if the word contains a delimiter, the character from the left delimiter to the right delimiter is the delimiter closest to the left of the included delimiter as the left delimiter and the included delimiter as the right delimiter Create a left-side extended clause consisting of a column, and use the character string from the left delimiter to the right delimiter with the included delimiter as the left delimiter and the delimiter closest to the right of the included delimiter as the right delimiter A first extended phrase creating means for creating the right extended phrase and writing the right extended phrase and the left extended phrase to the storage means,
If the result of the determination by the delimiter determining means is negative, the delimiter closest to the left of the word used for the determination is the left delimiter, and the delimiter closest to the right of the word is the right delimiter from the left delimiter A second extended phrase creating means for creating an extended phrase consisting of a character string up to the right delimiter and writing the extended phrase in the storage means;
An expanded phrase analysis unit that writes a morpheme analysis result in which a part-of-speech tag is added to a word constituting the expanded phrase by morphological analysis of the expanded phrase in the storage unit;
It is determined whether or not an unknown word tag is included in the morphological analysis result written in the storage means by the extended phrase analysis means, and if an unknown word tag is included, the morphological analysis result of the extended phrase and the previous time Analysis result determination means for determining whether or not a word to which an unknown word tag is assigned among the morpheme analysis results of
If the result of this determination is NO, delimiter character determination sending means for sending the morpheme analysis result of the extended clause to the delimiter character determination means,
As a result of the determination by the analysis result determination means, if both match, an analysis result output means for outputting the previous morpheme analysis result,
Language analysis program to function as A language analysis used for a language analysis device for outputting the unknown word when a morphological analysis is performed on a target sentence including an unknown word, including storage means for storing the target sentence and a morphological analysis result of the target sentence A program,
The language analysis device;
A morpheme analysis unit that writes a morpheme analysis result in which a part-of-speech tag is assigned to each word constituting the target sentence to the storage unit by performing morphological analysis on the target sentence in the storage unit;
It is determined whether or not an unknown word tag of the part-of-speech tags is included in the written morpheme analysis result, and when an unknown word tag is included, the word to which the unknown word tag is assigned is stored in the storage unit Unknown word tag determination means for writing to
A first condition that a word with an unknown word tag in the storage means is not a single character hiragana or katakana, and a morpheme analysis result is composed of a single morpheme, and the part of speech tag is an unknown word tag A completely unknown word judging means for judging whether or not the second condition is simultaneously satisfied,
As a result of this determination, in the case of No, a delimiter determining means for determining whether or not a delimiter corresponding to a particle, a punctuation mark, a symbol, a sentence head, or a sentence end is included for the word given the unknown word tag,
As a result of this determination, if a delimiter is included in the word, the delimiter closest to the left of the included delimiter is the left delimiter, the delimiter closest to the left of the included delimiter is the left delimiter, and the included The left delimiter is the left delimiter, and the delimiter closest to the right of the included delimiter is created. A first extended phrase creating means for creating a right extended phrase consisting of a character string from the left delimiter to the right delimiter as a right delimiter, and writing the right extended phrase and the left extended phrase into the storage means;
If the result of the determination by the delimiter determining means is negative, the delimiter closest to the left of the word used for the determination is the left delimiter, and the delimiter closest to the right of the word is the right delimiter from the left delimiter A second extended phrase creating means for creating an extended phrase consisting of a character string up to the right delimiter and writing the extended phrase in the storage means;
An expanded phrase analysis unit that writes a morpheme analysis result in which a part-of-speech tag is added to a word constituting the expanded phrase by morphological analysis of the expanded phrase in the storage unit;
As a result of the determination by the unknown word tag determination unit and the complete unknown word determination unit with respect to the morphological analysis result written in the storage unit by the extended phrase analysis unit, the word to which the unknown word tag is assigned is the first condition. And when the second condition is satisfied at the same time, means for outputting an unregistered word label to the word that is the unknown word
Language analysis program to function as The language analysis program according to claim 2,
As a result of the determination by the unknown word tag determination unit and the complete unknown word determination unit with respect to the morphological analysis result written in the storage unit by the extended phrase analysis unit, the word to which the unknown word tag is assigned is the first condition. And when the second condition is not satisfied at the same time, a means for giving a label indicating an input error to the word that is the unknown word and outputting the same,
Language analysis program to function as

Images