JP2009176148A - Unknown word determining system, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an unknown word determining system, method and program to determine unknown words without setting up morphological costs or part-of-speech connection costs. <P>SOLUTION: This unknown word determining system has a character type dividing means which divides Japanese document into character strings per character type, and an unknown word determining means which determines the likeliness of unknown word of the character strings not word-registered among the divided character strings. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an unknown word determination system, method, and program, and more particularly, to an unknown word determination system, method, and program for determining whether a character string is unregistered and valid as a word.

  In recent years, various natural language processing systems such as morphological analysis and speech recognition have been used as actual services.

  Many natural language processing systems require a dictionary.

  On the other hand, with the recent spread of services that allow users to freely post articles, new words, coined words, and words that have been intentionally broken (hereinafter referred to as words that have been unregistered in the dictionary as described above) An increasing number of documents use unknown words.

  In order to process these documents correctly, there are methods of correctly recognizing the position of the unknown word and proceeding thereafter, and registering the unknown word in the dictionary, both of which need to extract the unknown word correctly. There is.

  An example of this type of related unknown word collection system is described in Patent Document 1 and Patent Document 2.

  Patent Document 1 first reads a sentence to be analyzed, and generates a partial character string of several characters or less starting from each position from the beginning to the end of the sentence. Next, it is confirmed whether each partial character string exists in the dictionary for morphological analysis (registered word determination unit). Next, it is examined whether or not the partial character string not registered in the dictionary is an unregistered word that can be a morpheme (unregistered word determination unit). A morpheme cost is calculated for a character string determined to be present in the dictionary by the registered word determination unit (registered word cost calculation unit). The morpheme cost is calculated based on the part of speech, character type, frequency, and priority of the morpheme. The partial character string determined by the unregistered word determination unit as an unregistered word that can be a morpheme is calculated as the unregistered word cost based on the number of characters, character type, pattern, frequency of constituent characters, etc. (unregistered word cost calculation Part). When the morpheme cost and unregistered word cost are calculated for all partial character strings, the connection check of two adjacent morphemes is performed with reference to the grammar rule part, and the morpheme concatenation cost, morpheme cost, A row of morpheme candidates with the lowest cost is set as a morpheme analysis result based on the three registered word costs (morpheme analysis execution unit).

  A specific example will be described. In the string “Wangan War”, (1) Gulf / Noun Kishi / Noun War / Noun, (2) Gulf War / Unknown Word, (3) Gulf / Unknown Word War / Noun are candidates for morphological analysis Assume that you evaluate. If the unregistered word cost calculation unit is adjusted so as to give a high cost, (1) is selected where all are registered words and unknown words do not exist, and conversely (2) and (3) are reduced. It becomes easy to be chosen. In (2) and (3), (3) is selected if the cost for unregistered word costs of 4 characters or more is increased. Although the result obtained according to the cost setting can be adjusted as in the above example, the cost setting is generally a difficult task, and requires expert knowledge and many experiments. For example, if the result is “Tsuchida / Noun Denki / Noun” and “Tsuchida Denki / Unknown Word” for the proper noun “Tsuchida Denki”, the latter is combined rather than two consecutive nouns. Suppose you increase the cost of unknown words. Then, it becomes impossible to separate compound words of nouns such as “information / noun search / noun” of “information search”. On the other hand, it is conceivable that the cost calculation is further refined and realized, but it is difficult to set the cost comprehensively for such an example. In addition, estimation of dictionary attributes (parts of speech) of unknown words is not considered.

  Patent Document 2 is a morphological analysis of an input document, and from the result of division by morphological analysis, a word not included in the dictionary as an unknown word candidate, a value based on the number of appearances in the document, a value based on the presence or absence of a unique suffix, The reliability as an unknown word is calculated from the calculation result using at least two values among the value based on the presence or absence of the ending of the predicate, the value based on the number of characters and the character type, and the unknown word based on the reliability , And the headword, part of speech, and reliability determined by morphological analysis are presented, and the user is asked to determine whether to register in the dictionary.

  In the invention of Patent Document 2, since unknown word extraction is left to morpheme analysis, an unknown word cannot be appropriately extracted when morpheme analysis is partially performed even though it is actually one morpheme.

A specific example is described with the sentence “I saw a very cute girl yesterday”. It can be seen that the correct answer of morphological analysis is “Yesterday / Noun, / Symbol / Adverb Kawai / Adjective Daughter / Noun / Participant / Verb. / Symbol”. However, it is assumed that “kawai” meaning “cute” does not exist in the dictionary for morphological analysis. On the other hand, it is assumed that “kawa (skin) / noun” and “yu / verb” are registered in the dictionary. At that time, “Yui” is valid as a form of utilization of “Yu”, so “Kawa Yui” is analyzed as “Kawa / Noun Yui / Verb” and “Yesterday / Noun, / Symbol Very / Adverb Kawa / Noun Yui / Verb” Daughter / Noun / Participant / Verb. / Symbol ”. This result is a possible part-of-speech concatenation if only the concatenation of two morphemes is seen. In order to prevent such analysis, it is not difficult to imagine that very many settings are required, such as cost setting for morphological analysis and creation of exception rules.
Japanese Patent No. 3880087 Japanese Patent No. 3935374

  The first problem is that it is difficult to set various costs such as a morpheme cost, a part-of-speech concatenation cost, and an unknown word cost so as to correspond to various sentences.

  The reason is that expert knowledge is necessary to adjust the morpheme cost, the part-of-speech concatenation cost, and the unknown word cost in consideration of the balance so as to correspond to various sentences, and trial and error using many sentence examples This is because a large amount of experimentation is required.

  Accordingly, the present invention has been invented in view of the above problems, and an object thereof is to provide an unknown word determination system, method, and program capable of determining an unknown word without setting morpheme costs and part-of-speech concatenation costs. It is in.

  The present invention for solving the above-mentioned problems is characterized by character type dividing means for dividing a Japanese document into character strings by character types, and unknown word determination for determining the unknown word likelihood of character strings that are not registered in the divided character strings. And an unknown word determination system having means.

  The present invention that solves the above-mentioned problem is that an information processing apparatus divides a Japanese document into character strings according to character types, and determines an unknown word likelihood of a character string that is not registered as a word among the divided character strings This is a determination method.

  The present invention for solving the above-mentioned problems is a character type dividing process for dividing a Japanese document into character strings according to character types, and an unknown word determination for determining an unknown word likelihood of a character string that is not registered as a word among the divided character strings. A program that causes an information processing apparatus to execute processing.

  According to the present invention, an unknown word can be extracted without requiring setting of a morpheme cost and a part-of-speech concatenation cost.

  The reason is that the character string is used to divide the character string by determining the word break, and the validity of the character string that does not exist in the dictionary of the divided character string is determined. This is because unknown words can be extracted without requiring the concept of part of speech or morpheme cost.

  An embodiment of the present invention will be described. Hereinafter, “X / Y” indicates that X and Y are divided. Further, Z in “X (Z) / Y” represents additional information for easy understanding of the explanation.

  An unknown word determination system according to an embodiment of the present invention includes a character type dividing unit that divides an input character string according to a character type, an attached word (a word that cannot be composed of a single phrase, and has a meaning attached to an independent word) and a word Division position correction means that corrects the division result by character type by clue expression that divides words centered on the utilization form, and unknown character is judged from the word dictionary and the character type and character length of the extracted character string It has word judgment means.

In the unknown word determination system of the present invention,
1) Japanese word delimiters can be judged to some extent by character type.
2) An unknown word is basically an independent word and an ancillary word is fixed.
Based on these two ideas. Since there are few types of attached words, an exhaustive dictionary can be easily created.

  In the unknown word determination system of the present invention, first, the character string is divided by the character type dividing means at the portion where the character type changes. Here, the character types include hiragana, katakana, kanji, alphabet, numbers, symbols, and the like. However, there may be a restriction that characters such as “-” and “˜” are not separated by a long sound character.

  Next, the division position correcting means corrects the insufficient or excessive division by the character type dividing means by using the attached words and the utilization form as clues. Insufficient division refers to the case where the characters are connected because they are the same character type but can be further divided, such as “Japanese / Happy /.”. In addition, overdivision refers to a case where the character type is different, such as “run / run” in “school / to / run / run /.” But the previous character string is one word.

  Insufficient division and excessive division basically occur in a portion adjacent to a character string whose character type is hiragana. As in the above example, the division position correction for the hiragana character string has the effect of improving the accuracy of unknown word extraction by repelling an inappropriate word as a word in front of the unknown word determination means by correcting the lack of division, By correcting the excessive division, it is possible to correctly extract words that are divided by the character type, and it is possible to extract more appropriate unknown word candidates. As described above, the division position correcting means has a greater effect on the present invention, but is not necessarily a necessary component.

  Specifically, the division position correcting means first checks the attached word and the utilized character string with the longest match from the front of the hiragana character string. When searching for a utilization-type character string, the search is performed not only immediately after but also from the second character. This is because there are many words including one hiragana character immediately after the stem. For example, the stem of “small” is “small”, and the second character such as “small” is used. In the case of an ancillary word, the hiragana is divided into an ancillary word and a non-ancillary word. For example, “Japanese / Happy /.” Changes in hiragana between “word” and “ha”, and as a result of investigating the existence of the adjunct, the particle “ha” was found, and “Japanese / ha ( Particle) / difficult /. " Here, “ha”, which is longer than “ha”, is neither an attached word dictionary nor an inflection form, so it is supplemented that the particle is “ha”. If it matches an inflected form instead of an attached word, the previous character string and the inflected form character string are combined, and the inflected form character string is deleted from the hiragana character string. For example, “school / until / run / run /.” Changes to hiragana between “school” and “until”, “until” matches the particle, and “run” and “ru” "Ru" exists as a practical form, so it finally becomes "School / until (particle) / run /.".

  Next, the adjunct word is examined with the longest match from the rear with respect to the place where the hiragana changes to another character type. If an adjunct word exists, the adjunct word is separated and divided in the same manner as the previous adjunct word processing. It should be noted that it is not necessary to modify the utilization form with backward matching. This is because the inflected form is a single word including that part, and cannot be divided even if it exists.

  The unknown word determination means determines whether each divided character string is an unknown word. In the unknown word determination means, the partial character string, character type, number of characters, word dictionary, extraction frequency, extraction for character strings that do not exist in the word dictionary prepared in advance among the character strings divided in the process up to the previous stage The validity as an unknown word is determined based on the type of utilization and the number of types of utilization extracted. “Extracted utilization types” will be described. The extracted utilization type is the utilization type extracted from the document by processing up to the previous stage with one utilization rule for each unknown word candidate. For example, for an unknown word candidate of “Choroi”, “Choroi (original form)”, “Choroku (continuous form)” and “Choro if (conditional form)” are extracted by the usage rule of “I adjective Auodan”. . In this case, the type of utilization of “Choroi” is “original form of the adjective Audan, continuous form, conditional form”, and the number of kinds is 3. The type and number of utilizations extracted can be used as an index of word-likeness based on the rule of thumb that actual words are used for various purposes if they are words that are useful. Here, a character string that is not valid as an unknown word is a character string that does not make sense due to a word division error, or a character string that has meaning but is not required to be registered because it is a compound word of words in the word dictionary. is there. For example, in the case of a division mistake, a character string with only an unusually short or long hiragana is conceivable. On the other hand, even if a long hiragana includes `` tsu '' or ``-'', there is a possibility that the expression is broken, so if the character string with the registered word is similar, that word It can be regarded as the notation of swaying.

  By adopting such a configuration, an unknown word can be determined without requiring setting of a morpheme cost and a part-of-speech concatenation cost, and the object of the present invention can be achieved.

  Further, dictionary attributes can be estimated by adding dictionary attribute estimation means to the unknown word determination system described above.

  The dictionary attribute estimation means includes a word attribute registered in a dictionary that appears in the same context as a character string determined to be an unknown word from a word segmented document data and a word dictionary in which words and dictionary attributes are registered. Operates based on the idea that they are similar. For example, “Kawaii” of “Ano / Actress / Cuteai / Ne /.” And “Coolie” of “Ano / Actress / Coolie / Ne /.” Have the same word string before and after. It can be regarded as appearing in the same context. If the dictionary attribute “cool” is registered as “adjective” in the word dictionary, the dictionary attribute “cute” can be estimated as “adjective”. Furthermore, if there is a sentence “That actress is good” and the dictionary attribute of “good” is “adjective”, it becomes more probable that the dictionary attribute of “kawaii” is “adjective”. . Thus, the dictionary attribute estimation means can estimate the dictionary attribute by counting the frequency of dictionary attributes of words appearing in the same context and using the most dictionary attribute as an estimation result. The dictionary attribute is not limited to the part of speech, and can be any kind of dictionary attribute that is determined to some extent by the context.

  Hereinafter, embodiments of the present invention will be described in detail.

(First embodiment)
A first embodiment for carrying out the present invention will be described in detail with reference to the drawings.

  In addition, in the following description, when it is described as a word break expression, it is assumed to be a character string of an attached word and a word utilization form. An adjunct in the present invention is a word that cannot constitute a phrase by itself, such as a particle, an auxiliary verb, a prefix, and a suffix.

  FIG. 1 is a block diagram showing the configuration of the first embodiment for carrying out the present invention. Referring to FIG. 1, in the first embodiment of the present invention, document data input means 1 for inputting document data to be subjected to unknown word determination, a storage device 2 for storing document data and dictionary data, and a program It has a data processing device 3 that operates under control, and an unknown word display means 4 for confirming the determination result.

  Each of these means will be described.

  The unknown word display means 4 displays the character string determined as the unknown word by the unknown word determination means 32 together with the document from which the unknown word is extracted.

  The document data input unit 1 causes the document data storage unit 20 to store document data to be determined for unknown words. The document data includes “document data” that represents the main body of the document data and “document ID” that represents the identifier of the document data. “Document data” is an arbitrary Japanese character string. The “document ID” may be specified in advance, or may be automatically assigned, for example, by assigning a document ID so as to be an integer serial number in the input order.

  The storage device 2 includes a document data storage unit 20, a word break expression dictionary storage unit 21, a word dictionary storage unit 22, and an unknown word determination rule storage unit 23.

  The document data storage unit 20 stores the document data input by the document data input unit 1.

  The word segmentation expression dictionary storage unit 21 stores the character string, part of speech, and original form of the attached word, and the character string, the type of the used form, and the basic form of the used form.

  The word dictionary storage unit 22 stores an original character string and a utilization form of words. The word dictionary storage unit 22 may have arbitrary dictionary attributes in addition to the original character string and the utilization form. Part of speech is an example of a dictionary attribute.

  The unknown word determination rule storage unit 23 stores a rule for determining validity as an unknown word. An example of the unknown word determination rule will be described later.

  The data processing device 3 includes a character type division unit 30, a division position correction unit 31, and an unknown word determination unit 32.

  The character type dividing unit 30 reads the document data from the document data storage unit 20, divides the read document data at the portion where the character type changes, and passes the result to the division position correcting unit 31. Here, the character types include hiragana, katakana, kanji, alphabet, numbers, symbols, and the like.

  It may be divided by character type so that it is not separated by characters that do not clearly make sense, such as stuttering (nya, yu, etc ...), prompting sound (tsu, tsu), and long sound (-, ~). Further, it may not be divided at a change point from a specific character type to another character type. For example, words that change from kanji to katakana, such as “Tsuchida Giants”, may be proper nouns, so they are not separated.

  The division position correcting means 31 reads the information of the word break expression from the word break expression dictionary storage unit 21, corrects the result of dividing the divided character string passed from the character type dividing means 30 based on the read data, and is unknown. The corrected divided character string is passed to the word judging means 32. The division position correcting means 31 corrects the insufficient division or excessive division by the character type dividing means 30 using the word delimiter expression as a clue. Here, the shortage of division means a case where the characters can be further divided but are connected because they are the same character type. In addition, overdivision refers to the case where the character type is different but the word is one word including the previous character string.

  The division position correcting means 31 checks the presence of the word break expression with the longest match from the front of the hiragana character string. The utilization type character string is searched from the second character. If the longest matching word delimiter is an ancillary word, the hiragana character string is divided by the ancillary word and other than the ancillary word. Join with a column. Here, when a particle is found, it may be corrected so as to be combined into one unless the front divided character string is hiragana. This is because immediately before the particle, it is basically a noun, and even if it consists of a plurality of character types, it is considered as one noun. In addition, the division position correcting means 31 may virtually handle two characters accompanied by a prompt sound and a long sound, or two characters such as an open sound (kya, ca, etc.) virtually as one character. This is because it is considered that the first character of the two characters does not become a word break.

  Next, the division position correcting means 31 checks the presence of an attached word with the longest match from the back of the hiragana character string. If there is an ancillary word, it is divided by separating the ancillary word in the same manner as the processing of the ancillary word described above. It is supplemented that there is no need for modification of the conjugation type in backward matching. This is because the usage form is a single word including that part, so even if it exists, it cannot be divided, so there is no need to examine it. Here, when the attached word is separated, if the previous divided character string becomes one hiragana character, it is considered that the single hiragana character will not become a word, and therefore, it may not be divided.

  The unknown word determination unit 32 reads the word data from the word dictionary storage unit 22 and the unknown word determination rule from the unknown word determination rule storage unit 23, respectively, and the corrected divided character passed from the division position correction unit 31. For the character string that does not exist in the word read from the word dictionary, the validity of each divided character string as an unknown word is evaluated using the word dictionary data and the rules for determining the unknown word. The determination result is passed to the unknown word display means 4. Further, the unknown word determination unit 32 may create a character string obtained by combining several divided character strings, and may determine an unknown word for the combined character string that is not in the word dictionary. For example, in the case of the division “Tokyo / Tower / Ni / I went”, “Tokyo Tower”, “To Tower”, and “I went” that were connected 2 may be candidates for unknown word determination. When the combined character string is generated, the character string that is known as an attached word by the division position correcting means 31 may be excluded from the target of combination. In this case, if “ni” is known as an attached word in the previous example, a character string including “ni” is not a candidate, and only “Tokyo Tower” is a combined character string.

  Specifically, in the unknown word determination means 32, the validity of the character string that does not exist in the word dictionary prepared in advance among the divided character strings is determined as the unknown word determination rule in the unknown word determination storage unit 23. Use to evaluate. The unknown word determination rule has validity as an unknown word based on at least one of partial character string, character type, number of characters, word dictionary, number of extractions, extracted utilization type, and extracted utilization type number. Is defined. Each rule can be scored. When the score is positive, the rule represents a character string that seems to be an unknown word, and when the score is negative, the rule represents a character string that does not look like an unknown word. Even if this score is a score adjusted by a human according to the purpose, an unknown word and a non-unknown word may be prepared in advance and a score for identifying them may be set by automatic learning. In addition, an unknown word and a non-unknown word may be learned by a general-purpose machine learning method based on a partial character string, character type, number of characters, word dictionary, and number of extractions. Examples of general-purpose machine learning include decision trees, neural networks, inductive logic programming and support vector machines. When using machine learning, some unknown words and non-unknown words are classified in advance by hand. Then, a partial character string, a character type, the number of characters, a word dictionary, the number of extractions, an extracted utilization type, and an extracted utilization type number are created as the feature amounts constituting the classification rule. For example, it is assumed that “choroi (original form)”, “choroku (continuous form)” and “choroi (conditional form)” with “choroi” as the original form are extracted and the total frequency is 20. Examples of feature quantities that can be extracted from the information include “including katakana”, “katakana to hiragana”, “frequency = 20”, “frequency 10 or more”, “partial character string =“ choro ””, “partial character string =“ i ””, “ Number of types of utilization = 4 "Type of utilization =" original form "" "Type of utilization =" conditional form "" "Type of utilization =" continuous form "" "Number of characters = 4" "Number of characters 3 or more" "Including open sound Or the like. With respect to unknown words and non-unknown words classified manually, the above-described feature amount is generated, and a classification rule can be obtained by a general-purpose machine learning algorithm. A case where an unknown word candidate is classified as an “unknown word” by the learned classification rule can be determined as an unknown word. Hereinafter, an example set by a human will be described.

  In the unknown word determination rule storage unit 23, rule 1) “contains N katakana characters = N”, rule 2) “composed of a plurality of words in the word dictionary = −5”, rule 3) “consists of a plurality of character types. Suppose that three rules of “= 4” are stored. Then, the unknown word determination unit 32 applies each rule to the unknown word candidate (a character string that does not exist in the word read from the word dictionary among the corrected divided character strings passed from the division position correction unit 31) and conforms. When the sum of the scores of the determined rules exceeds 0, it is determined as an unknown word.

  At this time, assuming that “Tokyo” and “Tower” exist in the word dictionary, an example of performing the determination of “Tokyo Tower” is considered. Here, it is assumed that “Tokyo Tower” is extracted as an unknown word candidate when it is set not to divide by a change from kanji to katakana, or when divided character strings are combined. “Tokyo Tower” matches all the rules, with rule 1 having a score of 2, rule 2 having a score of -5, rule 3 having a score of 4, and the total being “1”. Therefore, since the threshold value 0 is exceeded, “Tokyo Tower” is determined as an unknown word.

  Here, if the policy is to make it difficult for a compound word to be determined as an unknown word, the score of rule 2 may be a large negative value. In addition, when a plurality of specific rules are matched at the same time, a score at the time of matching may be set, such as giving a score larger than the sum of the scores.

  Examples of unknown word determination rules based on word dictionaries include character similarity with registered words, such as when words registered in the word dictionary have words whose edit distance from the divided character string is less than a certain value. A rule that uses can be considered. Here, the edit distance is defined as the minimum number of procedures to make the same as the other character string by replacing, inserting, and deleting one character from the other character string. Is done. For example, the edit distance between “Cute” and “Cute” is 1 because it is the same only by inserting “T”. If the edit distance is sufficiently small, it can be considered appropriate because it can be considered as a fluctuation of the word. When emphasis is placed on the notation fluctuation, a special editing distance that eliminates the cost of deleting characters such as prompt sounds and long sounds may be used. Further, as an example of the rule based on the number of extractions, “the number of extracted words before and after N = N” is raised. This is because the fact that the preceding and following words are different means that they are extracted in various contexts, and therefore it is possible to determine that there is a low possibility of a break error.

  Next, the overall operation of this embodiment will be described in detail with reference to the functional block diagram of FIG. 1 and the flowchart of FIG.

  First, document data from which unknown words are to be extracted is stored in the document data storage unit 20 by the document data input means 1 (S1 in FIG. 2). FIG. 3 is an example of data stored in the document data storage unit 20. FIG. 3 includes two pieces of document data, the document ID is 1, and the document data is “I went to Tokyo Tower yesterday.” The document ID is 2, “That actress is cool! Document data exists.

  Next, the character type dividing means 30 reads the document data from the document data storage unit 20, divides each document data at a point where the character type changes, and generates each document ID and a divided character string list corresponding to the document ID (FIG. 2). S2). The character types are “Hiragana”, “Katakana”, “Kanji”, “Symbol”, “Alphabet”, and “Number”. In addition, “−” and “˜” are regarded as long sounds and are not separated by the above characters. FIG. 4 is an example of the generated divided character string list. The divided character string list includes a “position” that indicates the order from the beginning of the divided character string, a “split character string” that indicates the divided character string, and a “corrected” character that indicates whether each character string has been corrected. ? "Field. The initial value of “corrected?” Is “0” indicating that the correction has not been made. The first data in the list of “document ID = 1” in FIG. 4 stores “0” because the first divided character string of document ID = 1 is “Yesterday” and “corrected?” Has not been corrected yet. Has been.

  Next, the division position correcting means 31 acquires one position of the character string that has not been corrected from the top of the divided character string list (S3 in FIG. 2). Hereinafter, the position is referred to as a correction position.

  Next, if the character type of the obtained divided character string at the correction position is hiragana, the division position correction means 31 reads the word break expression from the word break expression dictionary storage unit 21 and proceeds to S5 in FIG. It moves to S7 of FIG. 2 (S4 of FIG. 2). FIG. 5 is an example of word break dictionary data stored in the word break expression dictionary storage unit 21. The word delimiter dictionary is mainly composed of an attached word and a character string representation of a utilization form. For the types of the auxiliary words such as auxiliary verbs, the usage forms are expanded and registered in the dictionary in advance, but only the basic forms may be stored and the division position correcting means 31 may execute the expansion of the usage forms. . The dictionary of ancillary words includes an “adjunct” that represents the character string of the ancillary word, a “type” that represents the type of the adjunct, a “type of utilization” that represents the type of the inflected form, and the original character string It consists of "original form" to represent. The utilization-type dictionary is composed of a “utilization form” representing a utilization form character string, a “type” representing the type, a “use type” representing the utilization form type, and a “original form” representing the original character string.

  Next, when the character type of the divided character string is hiragana, the division position correcting means 31 corrects the division from the front of the character string using the word delimiter expression (S5 in FIG. 2). The operation will be described by taking as an example a case where the current correction position is document ID = 1 in FIG. First, in the word delimiter expression, the longest matching character string is searched from the front of the divided character string. The utilization type character string is searched from the second character.

  In this example, it is assumed that the particle “ni” has the longest match. Here, the second character “I” coincides with the use of an adjective, but when “I” is separated, “I” immediately follows, so that it is not divided. If it matches any of the attached words, the attached word is separated from the split character string, the attached word string is inserted at the current correction position, and “corrected?” Is updated to 1, and thereafter The divided character string is corrected by adding one by one, and the correction position is shifted backward by one. Specifically, the list of document ID = 1 in FIG. 4 is shown in FIG. In FIG. 6, since “ni” in position 5 is divided into “ni” and “ni”, “ni” is inserted to correct the subsequent positions. In FIG. 6, in the flowchart of the present invention, it is guaranteed that the divided character string at the position before the corrected “ni” has been corrected, so that “corrected?” Is 1. Let me supplement. The correction position is 5 to 6. Here, an example will be described in the case where it matches not the attached word but the utilization form. As an example sentence, think of “I made it short”. The divided character string list based on the character type in the previous example is “short / hung”. Here, it is assumed that “ku” of “Kuhashita”, which is hiragana, exists in the utilization form. Then, since the utilization form is the ending of the previous word, the utilization form character string is separated and integrated into the previous divided character string as “short / hashed”. In this case, the previous character string is corrected again. If both an ancillary word and an inflected form are matched, the ancillary word takes precedence. In addition, in the division position correcting means 31 of this example, two characters accompanied by a prompt sound and a long sound and two characters of open sound (kya, ca, etc ...) are virtually treated as one character. For example, “KA” of “Kawaii” at position 3 in document ID = 2 in FIG. 3 matches the particle, but “KA” is treated as a single character, and therefore does not match, “KA” and “WAII”. It is not divided into “i”.

  Subsequent to S5, for the character string at the current correction position, the division position correcting means 31 corrects the division using only the attached words in the word delimiter expression from the rear of the character string (S6 in FIG. 2). The correction operation in S6 will be described by taking as an example the case where the current correction position is 3 of document ID = 2 in FIG. First, the longest matching character string is searched from the back of the divided character string from the attached words of the word delimiter expression. In this example, it is assumed that the particle “Ne” has the longest match. In that case, the longest matching adjunct is separated from the divided character string, the current correction position is updated to “corrected?”, The adjunct character string is inserted after the current correction position, and thereafter Shift the position of 1 backward, and shift the correction position 1 backward.

  Specifically, the list of document ID = 2 in FIG. 4 is shown in FIG. In FIG. 7, in the flowchart of the present invention, it is guaranteed that the divided character string at the position before the corrected “Ne” has been corrected, so that “corrected?” Is 1. Let me supplement. Since the correction position is shifted from 3 by 1, it becomes 4. Further, in the division position correcting means 31 of this example, if the character string before the division becomes a single hiragana character as a result of the division by the word delimiter, it is considered that there is almost no possibility that the single hiragana character is a word. do not do. For example, “no” of “that” in position 1 of document ID = 2 in FIG. 3 matches the particle, but only “a” precedes it, so it is not divided.

  Next, the division position correcting means 31 updates “corrected?” In the list of the current correction positions to 1 without distinction between the cases coming from S4 and the cases coming from S6 (S7 in FIG. 2).

  Next, the division position correcting means 31 moves to S9 if “corrected?” Of all the divided character strings in the divided character string list is 1, and moves to S3 if not. (S8 in FIG. 2).

  Next, the unknown word determination means 32 reads the word data from the word dictionary storage unit 22 and the unknown word determination rule from the unknown word determination rule storage unit 23, and determines whether each divided character string has been registered as a word. By referring to the read word data, it is determined whether or not the unregistered divided character string is an unknown word using the read unknown word determination rule (S9 in FIG. 2).

  FIG. 8 is an example of data registered in the word dictionary storage unit 22. The word dictionary includes a “character string” in which the original character string of the word is stored, and, if there is a utilization form, a “utilization type” in which the utilization form type is stored. For example, the second line in FIG. 8 indicates that “yesterday” is a word that is not utilized.

  When the utilization type is designated, the unknown word determination unit 32 converts the utilization type into the original form by making the utilization a basic form according to the utilization rules. For example, “Ike (instruction type)”, “Tone (Ta type)”, etc. are converted to “Iku”. This is a process for acquiring frequency information appropriately by absorbing differences in the surface layer due to utilization. FIG. 9 is an example of rules registered in the unknown word determination rule storage unit 23, and ten rules are registered. In FIG. 9, the rules are described in a natural language for easy understanding. However, in actuality, the rules are deterministically processed by a program, and the rules written in the natural language are not interpreted. Absent. The unknown word determination rule storage unit 23 includes a “rule ID” that represents a rule identifier, a “rule” that represents a rule content, and a “score” that represents a rule score. A rule with a positive “score” represents a rule that seems to be an unknown word, and a rule that has a negative value represents a rule that does not look like an unknown word. For example, a rule of “2 characters or less” with a rule ID of 2 indicates that if it is 2 characters or less, it does not look like an unknown word and the penalty is “−2”.

  The unknown word determination means 32 applies all the rules to the divided character string for determining whether it is an unknown word, calculates the sum of the scores of the matching rules, and if the sum of the scores is larger than the threshold, the divided character string is unknown. Judged as a word. The threshold value in this example is 0. Assuming that all the “corrected?” In the divided character string lists in FIGS. 6 and 7 are 1, details of the operation will be described for both divided character string lists. The positions 1, 2, 3, 5, 6, and 7 in FIG. 6 are all determined to be registered in the word dictionary of FIG. When the rule of FIG. 9 is applied to the “tower”, the rule IDs 1, 6, and 7 are met. A score of 1 is score 3, a score of 6 is score 2, and a score of 7 is score 2. The total is 7, which exceeds the threshold 0 and is determined as an unknown word. Similarly, in FIG. 7, only “Kawaii ~ i” of the divided character string at position 3 is an unknown word candidate, conforms to rule IDs 1, 3, 7, and 8 in FIG. From score 2 to 7, score 2 from 7, and score -6 from 8 to score 1, exceeding threshold 0 and becoming an unknown word. For example, a sentence written in hiragana such as “I want to read this book” cannot be divided well in the present invention, but a large penalty is imposed by the rule of rule ID 8, so that it is determined as an unknown word. Absent. An example of determining unknown word candidates that will be utilized will also be described. For unknown words that are used, all information of unknown word candidates that have the same basic form is used. For example, the unknown word judging means 32 may select “Choroi (basic form)”, “Choroku (continuous form)”, “Corokan (ta form)”, “Chorokute (ta form joint form)”, etc. According to the steps, it is converted to the basic form “Choroi”. The frequency of all the candidates is the sum of their use. In addition, there are four types of utilization of all the candidates: “basic type”, “continuous type”, “t-type”, and “t-type continuous type”. When the rule of FIG. 9 is applied, the rule IDs 1, 4, 6, 9, and 10 are met. The score is 3 + 2 + 6 + 3 + 3 = 17, and is determined as an unknown word.

  Finally, the unknown word display means 4 displays the divided character string determined as an unknown word in S9 together with the document ID from which the unknown word is extracted and its document data (S10 in FIG. 2). FIG. 10 shows an example of display. The document ID and document data from which the unknown word is extracted are displayed as “document ID” and “input document”, respectively, and the result of the division by the process is “division result”, and is determined as an unknown word in the divided character string. The divided character string is displayed as “unknown word”. This display method is merely an example, and an unknown word may be displayed but a specific character string may be embedded before and after the character string. For example, it is conceivable to enclose an unknown word with a tag "<unknown>" such as "That actress <unknown> is so cute!"

  In this embodiment, the character type dividing unit 30 divides the character type, and the division position correcting unit 31 corrects the word break position by using the attached word and the word utilization form, thereby dividing the word. Therefore, unlike general morpheme analysis, word division can be performed without the need to set morpheme costs and part-of-speech concatenation costs, and unknown word determination means 32 can determine the character strings that are not registered in the word dictionary. Among them, since it is configured to determine a character string that is valid as a word as an unknown word, it is only necessary to regularize the word likeness, and it is necessary to set morpheme costs and part-of-speech concatenation costs that require specialized knowledge for setting. Without extracting unknown words.

  Here, it will be explained that the creation of rules for determining unknown words according to the present invention is easier than the creation of rules for morphological analysis and does not require specialized knowledge. First, settings necessary for performing morphological analysis will be described with reference to a typical morphological analysis tool, Juman (Non-Patent Document 3).

  <Non-patent document 3> Japanese morphological analysis system JUMAN version 5.1 (http://nlp.kuee.kyoto-u.ac.jp/nl-resource/juman.html)

  The item to be set is the part-of-speech concatenation cost that takes into account the morpheme cost and the utilization form. There are 50 types of parts of speech, excluding “undefined words”, including subcategories that should be distinguished grammatically. The developer needs to evaluate the cost by judging the concatenation possibility of at least 2500 (= 50 × 50) parts of speech. For cost evaluation, it is determined that the morphological analysis result of the document is correct, and if it is incorrect, what can be corrected to avoid the error, and whether the sentence that has been analyzed correctly will not be affected. The ability to judge is necessary. Also, since part of speech must be given to the word dictionary, it is necessary to have expertise to judge the part of speech including the initial dictionary construction and the subcategory of words. Furthermore, it is necessary to set a morpheme cost, which is a cost for appearance of a morpheme in a sentence. Explain the necessity of morpheme costs. For example, if you compare a noun and a particle and look at the string “company is”, “company (noun) / ha (particle)” and “company (noun) / ha (noun, meaning of leaf)” Suppose that it is determined which is appropriate. When the part of speech connection cost from a noun to a noun and the part of speech connection cost from a noun to a particle cannot be determined, it can be determined by using the morpheme costs of the noun and the particle. When the morpheme cost of a particle is lower than that of a noun, “company (noun) / ha (particle)” is output. Thus, setting the part-of-speech concatenation cost and the morpheme cost so that the morpheme analysis is successful assuming various sentences requires high expertise and effort.

  On the other hand, in the unknown word determination rule, a string with an unusually large number of characters in hiragana is considered to be insufficiently divided, so a negative score, katakana is a positive score because there is a high possibility of an unknown word, etc. Rules can be created intuitively. In morphological analysis, it is necessary to set a cost for the relationship between words. However, in unknown word determination, since only one word needs to be considered, it can be said that the setting is not complicated.

(Second Embodiment)
A second embodiment for carrying out the present invention will be described in detail with reference to the drawings.

  FIG. 11 is a block diagram showing the configuration of the second exemplary embodiment for carrying out the present invention. Referring to FIG. 11, the second embodiment of the present invention has a non-delimited expression dictionary storage unit 24 in addition to the configuration of the storage device 2 in the first embodiment shown in FIG. The first embodiment differs from the first embodiment in that it has a non-separated expression considering division position correcting means 33 instead of the division position correcting means 31 of one data processing device 3.

  Hereinafter, the non-delimited expression dictionary storage unit 24 and the non-delimited expression-considered division position correcting unit 33 different from the first embodiment will be described. Other configurations are the same as those in the first embodiment, and thus are omitted.

  The non-delimited expression dictionary storage unit 24 stores typical hiragana character strings as non-delimited expressions. Examples include “connectives” and “indicators” as non-delimited expressions.

  The non-delimiter expression considering division position correcting unit 33 reads the information of the word delimiter expression from the word delimiter expression dictionary storage unit 21, reads the information of the non-delimiter expression from the non-delimiter expression dictionary storage unit 24, and is passed from the character type dividing unit 30. The division result of the divided character string is corrected, and the corrected divided character string is passed to the unknown word determination unit 32.

  The non-delimited expression consideration division position correcting means 33 searches the expression registered in the non-delimited expression dictionary with the longest match from the front and rear before applying the word delimiter expression to each divided character string. It differs from the division position correcting means 31 in that the non-delimited expression is separated by the same process as the process of the attached word of the delimited expression. In the division position correcting means 31 of the first embodiment, for example, there is an excessive division in which a character string “such as” is separated from “such as / to (particle)”. The delimiter expression considering division position correcting means 33 uses the non-delimiter expression that is not a word delimiter, and therefore has the effect of eliminating excessive division as in the above example.

  Next, the overall operation of the present embodiment will be described in detail with reference to the functional block diagram of FIG. 11 and the flowchart of FIG.

  Since S1 and S2 in FIG. 12 are the same as S1 and S2 in FIG. 2 which is the flowchart of the first embodiment, description thereof is omitted.

  Next, the non-separated expression considering division position correcting means 33 acquires one position of the character string that has not been corrected from the top of the divided character string list (S3 in FIG. 12). This is called a correction position.

  Next, the non-separation expression considering division position correcting means 33 confirms whether or not the character type of the obtained divided character string at the correction position is hiragana, and if it is hiragana, the word delimiter expression dictionary storage unit 21 reads the word. The delimiter expression is read, and the non-delimiter expression is read from the non-delimiter expression dictionary storage unit 24, and the process proceeds to S5 in FIG. 12, otherwise the process proceeds to S7 in FIG. 12 (S4 in FIG. 12). FIG. 13 is an example of non-delimited expressions stored in the non-delimited expression dictionary storage unit 24. The non-delimited expression dictionary includes a “character string” field representing the character string itself. Typical non-delimited strings are directives and conjunctions.

  In S4, when the character type of the divided character string acquired in S3 is hiragana, the non-delimited expression-considered division position correcting means 33 uses the non-delimited expression from the front of the divided character string at the current correction position. Is further corrected by word break expression (S5 in FIG. 12).

  As an example, consider the sentence “Please ask me that way.” The character type dividing means 30 divides “That's how / request / please /.”. When the non-delimited expression in FIG. 13 is applied to “that is like that” at position 1, “that” is the longest match, “that” is position 1, “is so” is position 2, and “request” is the position. 3. “Please” is position 4, and “.” Is position 5. Since “that” has been inserted at position 1, the non-separated expression considering division position correcting means 33 updates “corrected?” At position 1 to 1 and advances the current correction position to 2. When the longest match of the word segmentation expression is searched for “ha so” at position 2, it matches the particle “ha”, so “ha” is position 2, “so” is position 3, "Request" is position 4, "Please" is position 5, "." Is position 6, "Correction completed" at position 2 is updated to 1, and the current correction position is advanced to 3.

  Next, the non-delimiter expression considering division position correcting means 33 corrects the division using the non-delimiter expression from the rear of the divided character string at the current correction position, and further corrects it with the word delimiter expression (S6 in FIG. 12). Since the current correction position is 3, when the delimiter character to be drawn with the longest match from the back is examined, it matches “that way”. In this case, since it is a complete match, the “corrected?” At position 3 is updated to 1 without dividing, and the process proceeds to S7. In the first embodiment, there is a case where excessive division is caused by not using a non-delimitation expression such as “so // (particle)”, but the excessive division is eliminated.

  Since S7 to S10 in FIG. 12 are the same as S7 to S10 in FIG. 2 which is the flowchart of the first embodiment, description thereof is omitted.

  In this embodiment, in addition to the first embodiment, there is a non-delimited expression dictionary storage unit 24 that stores a character string that does not become a word break, and non-delimited expression-considered division position correcting means 33 Since the expression is structured so that the character string is not divided, there is no excessive division of the hiragana character string, and the words are extracted more accurately, thereby reducing the number of non-word character strings. The effect is improved.

(Third embodiment)
A third embodiment for carrying out the present invention will be described in detail with reference to the drawings.

  Conventionally, in morphological analysis, the part-of-speech estimation and morphological analysis are not considered at the same time, because the part-of-speech decision and the part-of-speech concatenation cost are used to evaluate the segmentation result and determine the output result. The dictionary attribute of unknown words could not be estimated. Therefore, the third embodiment is characterized in that a configuration for estimating dictionary attributes of unknown words is added to the features of the first embodiment.

  FIG. 14 is a block diagram showing the configuration of the third exemplary embodiment for carrying out the present invention.

  Referring to FIG. 14, in the third embodiment of the present invention, in addition to the configuration of the first embodiment shown in FIG. 1, word segmented document data input means 5 and unknown word information output means 6 are provided. First, in that it has a word-division-completed document data storage unit 25 in addition to the configuration of the storage device 2 in FIG. 1, and has a dictionary attribute estimation means 34 in addition to the configuration of the data processing device 3 in FIG. The form is different.

  The word segmented document data input unit 5, the unknown word information output unit 6, the word segmented document data storage unit 25, and the dictionary attribute estimation unit 34, which are different from the first embodiment, will be described below. Other configurations are the same as those in the first embodiment, and thus are omitted.

  The unknown word information output means 6 displays the unknown words and the dictionary attributes of the unknown words estimated by the dictionary attribute estimation means 34 together with the document data from which the unknown words are extracted.

  The word-divided document data input unit 5 stores the word-divided document data in the word-divided document data storage unit 25. The data input by the word-divided document data input means may insert the result of automatic division by an arbitrary morpheme analysis system, may be data manually divided, or character type division according to the first embodiment The result of automatic division by means 30 and division position correction means 31 may be used.

  The dictionary attribute estimation unit 34 includes a character string determined as an unknown word by the unknown word determination unit, word data read from the word dictionary storage unit 22, and word-segmented document data read from the word-segmented document data storage unit 25. Then, the dictionary attribute of the unknown word is estimated from the dictionary attributes defined in the word dictionary storage unit 22. Specifically, from the idea that the dictionary attribute of an unknown word is similar to the dictionary attribute of a word that appears in the same context as the unknown word, the document data from which the unknown word is extracted and the word string around the unknown word are the same If document data is searched from word-divided document data and a character string sandwiched between the same patterns is a word registered in the word dictionary, the frequency of the dictionary attribute is counted, and the most frequent dictionary attribute is Use dictionary attributes. For example, it is defined as a part of speech as a dictionary attribute of a word dictionary, and the word dictionary storage unit 22 has at least two data items of “noun” for “school” and “noun” for “pool”. The stored document data storage unit 25 stores “yesterday / ha / school / to / had /.” And “yesterday / ha / pool / to / had /.”. Suppose that “Gukkou” of “/ to / had /.” Is determined as an unknown word. In that case, assuming that the character patterns of the two words before and after are considered to be the same context, “school” and “pool” are words that appear in the same context, and the dictionary attribute is “noun” for both. Therefore, the part of speech that is the dictionary attribute of “Gakkou” is estimated as “noun”. Here, the number of words before and after may be changed.

  Further, when the similarity between a word that appears in the same context and a character string of an unknown word is high, the frequency may be counted by weighting the dictionary attribute of the word. As the similarity between character strings, the edit distance, the number of characters that match the front, and the number of characters that match the back are conceivable. In particular, backward matching is effective because compound words in Japanese often have meanings in the backward words, and the utilization form is considered to be effective in estimating dictionary attributes. In addition, in the case of a document whose notation is intentionally broken, it is preferable to use an edit distance in which the deletion cost of characters representing prompt sounds and long sounds is set low. When measuring the similarity of character strings, a character string obtained by converting an unknown word katakana character string into hiragana may be used. For example, when katakana is converted to hiragana, the unknown word “kawaii” matches “cute” in the word dictionary. The dictionary attribute is not the best one, and all attributes whose frequency exceeds the threshold may be assigned in order of frequency. When there is no document data with the same context, or when the frequency cannot be counted because there is no word with dictionary attribute among words that appear in the same context as the unknown word in the same context document data The dictionary attribute is “unknown”. Here, since the reliability is low when the frequency is below a certain level, “unknown” may be output.

  Next, the overall operation of the present embodiment will be described in detail with reference to the functional block diagram of FIG. 14 and the flowchart of FIG.

  First, the document data from which unknown words are to be extracted is stored in the document data storage unit 20 by the document data input unit 1, and the word-segmented document data is stored by the word-segmented document data input unit 5. (S1 in FIG. 15). An example of data stored in the document data storage unit 20 is the same as that in FIG. 3 described in the first embodiment. FIG. 16 shows an example of document data stored in the word-divided document data storage unit 25. According to FIG. 16, the word-division-completed data storage unit 25 is composed of “division-division document ID” and “word-division-completed document data” containing the result of word division.

  S2 to S9 in FIG. 15 are the same as S2 to S9 in the flowchart in FIG.

  Next, the dictionary attribute estimation unit 34 uses the word data read from the word dictionary storage unit 22 and the word segmented document data storage unit 25 for all the divided character strings determined as unknown words by the unknown word determination unit 32. The dictionary attribute is estimated using the word-divided document data read from (S10 in FIG. 15).

  FIG. 17 shows data stored in the word dictionary storage unit 22 in the example of the present embodiment. Part of speech and polarity are defined as dictionary attributes. The polarity is “positive” if the word includes a positive meaning and “negative” if the word includes a negative meaning. Here, as in the case of the description of the first embodiment, the description will be made on the assumption that “cute” is determined as an unknown word in the sentence of document ID 2 in FIG. 2 by S9. In addition, as shown in FIG. 7, the result of dividing the sentence with document ID 2 up to S9 is “that / actress / cool ~ i / ne /!”. In this example, the same context is assumed when the first word and the last word of the unknown word match. At that time, when a sentence having a pattern of “actress / <word> / ne” is acquired from the word-division document data of FIG. 16, three sentences of the divided document IDs 1 to 3 are extracted, and “Cute-ai” is obtained. The words appearing in the same context are “cute”, “good”, and “cool” from the document ID 1. In this case, referring to attribute 1 in FIG. 17, the part of speech is adjective = 3, and there is no other attribute candidate. Therefore, attribute 1 (part of speech) of “Kawaii ~ i” can be estimated as “adjective”. Regarding polarity, it can be estimated that “cute”, “good”, “positive” is 2, “cool”, negative is 1, and attribute 2 (polarity) of “cute” is positive. Here, if there is no document data of the divided document ID2, neither positive nor negative can be determined as it is, but it can be determined by counting the frequency of dictionary attributes by weighting according to the character string similarity. In some cases. Specifically, because “Kawaii-I” is similar in character string to “Cute” rather than “Coolie”, the frequency of “Cute” is positively weighted and the frequency is counted. That's fine. An example in which the edit distance is used as the character string similarity will be described. The edit distance is a value that can be considered to be similar as the edit distance is small, and is 0 for a completely matched character string. The edit distance between “cute” and “cute” is 2, and the edit distance between “cute” and “cute” is 3, and “cute” and “cute” are more similar. You can see that As an example of weighting, when the reciprocal of the edit distance is considered, “positive” of “cute” is ½, “negative” of “cool” is １ ／, and there is no document data of document ID 2 However, it can be presumed to be positive by increasing the weight of the dictionary attribute of words with similar character strings.

  Finally, the unknown word information display means 6 displays the unknown word determined by the unknown word determination means 32 and the dictionary attribute estimated by the dictionary attribute estimation means 34 together with the extracted document data (S11 in FIG. 15). FIG. 18 is a display example of the unknown word information display means 6. The document ID and document data from which the unknown word is extracted are displayed as “document ID” and “input document”, respectively. The divided character string is displayed as “unknown word”. In each unknown word, an estimation result of an arbitrary dictionary attribute defined in the word dictionary is displayed.

  In this embodiment, in addition to the first embodiment, it has a word segmented document data storage unit 25 and a dictionary attribute estimation means 34, and it is said that the dictionary attributes of words and unknown words that appear in the same context are similar. Since the dictionary attribute is estimated based on the idea, it is possible to estimate an arbitrary dictionary attribute determined to some extent by the context.

(Fourth embodiment)
A fourth embodiment for carrying out the present invention will be described in detail with reference to the drawings.

  FIG. 19 is a block diagram showing the configuration of the fourth exemplary embodiment for carrying out the present invention. Referring to FIG. 19, in the fourth embodiment of the present invention, in addition to the configuration of the second embodiment shown in FIG. 11, word segmented document data input means 5 and unknown word information output means 6 are provided. The second embodiment is that it has a word-divided document data storage unit 25 in addition to the configuration of the storage device 2 in FIG. 11 and a dictionary attribute estimation means 34 in addition to the configuration of the data processing device 3 in FIG. The form is different.

  The word segmented document data input unit 5, the unknown word information output unit 6, the word segmented document data storage unit 25, and the dictionary attribute estimation unit 34 of the fourth embodiment are the same as those described in the third embodiment. I will omit it.

  Next, the overall operation of this embodiment will be described in detail with reference to the functional block diagram of FIG. 19 and the flowchart of FIG.

  Since S1 of FIG. 20 is the same as S1 of FIG. 15, description thereof is omitted.

  Since S2 to S9 in FIG. 20 are the same as S2 to S9 in FIG.

  S10 to S11 in FIG. 20 are the same as S10 to S12 in FIG.

  In the present embodiment, since it has the configuration of the second embodiment, it is possible to segment words with high accuracy, so that the accuracy of unknown word extraction is increased, and the word segmented document data storage unit 25 and dictionary attribute estimation means 34 are further increased. Since the dictionary attribute is estimated based on the idea that the dictionary attribute of the word that appears in the same context and the unknown word are similar, it is possible to estimate any dictionary attribute that is determined to some extent by the context .

(Fifth embodiment)
A fifth embodiment for carrying out the present invention will be described in detail with reference to the drawings.

  FIG. 21 is a block diagram showing the configuration of the fifth embodiment for carrying out the present invention. Referring to FIG. 21, the fifth embodiment of the present invention is the third point in that it has original shape estimation means 35 in addition to the configuration of the data processing apparatus 3 of the third embodiment shown in FIG. Different from the embodiment.

  The original shape estimation means 35 reads data from the word delimiter expression dictionary storage unit 21 and the word dictionary storage unit 22, respectively, receives the dictionary attribute estimation result of an unknown word including at least part of speech by the dictionary attribute estimation unit 34, and determines the original form of the unknown word. Estimate its usage. If the unknown word is a part-of-speech with no use form, the character string of the unknown word is used as it is.

  The original form estimation means 35 first uses the utilization form expression read from the word segmentation expression dictionary 21 to examine the utilization form character string possible with the part of speech of the unknown word by backward matching, and narrows down the utilization form candidates.

  As an example, suppose that the extracted unknown words include “cool”, “cool” and “cool”, and the estimation result of those parts of speech is “adjective”. Of the inflected forms of adjectives, "i", "ku", and "kata" that match backwards are three of the adjective rules for adjectives: "I adjective Aodan", "I adjective Idan special", and "I adjective Idan". Suppose that If it is decided so far, the utilization form of the unknown word is determined, and the original form is the basic form of the utilization form. If it is not determined, next, the original shape estimation means 35 reads the word that is the utilization form of the narrowed candidates from the data read from the word dictionary storage unit 22.

  Next, each unknown word is converted into a basic form with each usage word candidate, and the degree of similarity with a word in the word dictionary of the same usage form is measured. At that time, the usage form of the most similar word is regarded as the usage form of the unknown word, and the basic form is used as the original form of the unknown word. As the degree of similarity of character strings, the number of front-matching characters, the number of back-matching characters, the editing distance, and the like can be considered. Taking the editing distance as an example, if the basic form of the unknown word is all “cool” and the word “cute” is the most similar word, the usage form is “a” The original form becomes “cool” which is its basic form. If the similarity of the most similar word appears in a different usage, it may be determined uniquely, such as by comparing the second word, or a plurality of candidates may be presented. If the unknown word is katakana, it may be converted to hiragana to check the character string similarity and usage. When measuring the similarity of character strings, if a high similarity is recognized and it can be regarded as a fluctuation of a word in the word dictionary, the original form may be used as a word in the word dictionary. In the case of the previous example, the original form of “cool” is “cute”.

  Next, the overall operation of the present embodiment will be described in detail with reference to the functional block diagram of FIG. 21 and the flowchart of FIG.

  Since S1 to S10 in FIG. 22 are the same as S1 to S10 in FIG. 15, description thereof is omitted.

  Next, the original shape estimation means 35 reads data from the word delimiter expression dictionary storage unit 21 and the word dictionary storage unit 22 respectively, receives the dictionary attribute estimation result of an unknown word including at least part of speech by the dictionary attribute estimation unit 34, and the unknown word Is estimated (S11 in FIG. 22).

  As in the example up to S10 in the third embodiment, the case where the unknown word extraction and the part-of-speech estimation results are “tower (noun)” and “cute (adjective)” will be described as an example. To do. First, because “tower” is a noun, there is no use form, so the original form is “tower”. On the other hand, “Kawaii ~ i” is an “adjective”, so there is a utilization form. The original form estimation means 35 uses the adjective usage form from the word segmentation expression dictionary storage unit 21 to check the adjective usage form by backward matching. The conjugation forms of adjectives that match back and match "I" are "I adjective Audan", "I adjective Idan", and "I adjective Idan special". Next, the basic form is acquired from the word dictionary data among the three candidate usage words, and the similarity between all the basic forms and the unknown word basic form is measured among the same usage forms. As an example of the character string similarity measure, an example will be described in which an edit distance that eliminates the cost of deleting a long sound (-, ~) is used. As a result, it is assumed that the edit distance between “Kawaii ~ i” and “Kawaii” is 1, which is the most similar word. Eventually, the usage form of “Kawaii-i” becomes the same “adjective aodan” as “cute”, and the original form becomes “Kawaii-i” which is its basic form.

  Next, the unknown word information display means 6 receives the result from the original form estimation means 35, and displays the extracted unknown word character string, dictionary attribute, original form and its utilization form, and document data from which the unknown word is extracted (FIG. 22 S12). FIG. 23 shows a display example. In FIG. 13, the document ID and the unknown word extracted document, the result after the division process, and the extracted unknown word are displayed, and the dictionary attribute and original form of the unknown word and its utilization form are displayed.

  In this embodiment, in addition to the third embodiment, the original form estimation means 35 is provided, and the utilization form candidates are narrowed down by backward matching of the part-of-speech of the unknown word and the utilization form character string, and further utilized when not uniquely determined. The utilization form is determined based on the similarity of the character string with the word of the candidate form, and the basic form of the utilization form is used as the original form. Therefore, the original form can be estimated even when the unknown form has the utilization form. According to the present embodiment, it is possible to provide means for recognizing that the same word is used even if there are unknown words having different character strings depending on the utilization form.

(Sixth embodiment)
A sixth embodiment for carrying out the present invention will be described in detail with reference to the drawings.

  FIG. 24 is a block diagram showing the configuration of the sixth embodiment for carrying out the present invention. Referring to FIG. 24, the sixth embodiment of the present invention is the third point in that it has original shape estimation means 35 in addition to the configuration of the data processing device 3 of the fourth embodiment shown in FIG. Different from the embodiment.

  Since the original shape estimation means 35 is the same as that described in the fifth embodiment, the description thereof is omitted.

  Next, the overall operation of the present embodiment will be described in detail with reference to the functional block diagram of FIG. 24 and the flowchart of FIG.

  25 are the same as S1 to S10 of FIG. 20 described in the fourth embodiment, and thus description thereof is omitted.

  25 are the same as S11 to S12 of FIG. 22 described in the fifth embodiment, and thus description thereof is omitted.

  In the present embodiment, since it has the configuration of the fourth embodiment, it is possible to divide the word with high accuracy, so that the accuracy of unknown word extraction is improved, and further, the original shape estimation means 35 is provided, and the part-of-speech and utilization form of the unknown word are provided. Narrow down usage candidates by backward matching of character strings.If it is not uniquely determined, determine the usage shape based on the similarity of the character string with the word of the usage shape candidate, and use the basic shape of the usage shape as the original form. Therefore, unknown words can be extracted with high accuracy, and the original form can be estimated even if the unknown words have a utilization form.

  Although the first to sixth embodiments of the present invention have been described above, the present invention can also be realized as computer software.

  FIG. 26 is a general block diagram of an information processing system that implements an unknown word determination system according to the present invention.

  The information processing system shown in FIG. 26 includes a processor 300, a program memory 301, and a storage medium 302. As the storage medium, a RAM or a magnetic storage medium such as a hard disk can be used.

  The program memory 301 stores a program for executing processing steps performed by the data processing apparatuses according to the first to sixth embodiments, and the processor 300 operates according to this program. The storage medium 302 is used as the storage device in the first to sixth embodiments.

  The present invention can be applied to an unknown word extraction function as preprocessing for morphological analysis. In addition, it can be applied to applications such as a program that realizes automatic dictionary multiplication by acquiring unknown words and dictionary attributes from document data.

It is a block diagram which shows the structure of 1st Embodiment. It is a flowchart which shows operation | movement of 1st Embodiment. It is explanatory drawing which shows an example of the document data to input. It is explanatory drawing which shows an example of the division | segmentation character string list produced | generated by the character type division | segmentation means. It is explanatory drawing which shows an example of a word break expression dictionary. It is explanatory drawing for demonstrating the process of a division position correction means. It is explanatory drawing for demonstrating the process of a division position correction means. It is explanatory drawing which shows an example of a word dictionary. It is explanatory drawing which shows an example of an unknown word determination rule. It is explanatory drawing which shows the example of a display of an unknown word display means. It is a block diagram which shows the structure of 2nd Embodiment. It is a flowchart which shows operation | movement of 2nd Embodiment. It is explanatory drawing which shows an example of a non-separation expression dictionary. It is a block diagram which shows the structure of 3rd Embodiment. It is a flowchart which shows operation | movement of 3rd Embodiment. It is explanatory drawing which shows an example of document data after word division. It is explanatory drawing which shows an example of a word dictionary. It is explanatory drawing which shows the example of a display of an unknown word information display means. It is a block diagram which shows the structure of 4th Embodiment. It is a flowchart which shows operation | movement of 4th Embodiment. It is a block diagram which shows the structure of 5th Embodiment. It is a flowchart which shows operation | movement of 5th Embodiment. It is explanatory drawing which shows the example of a display of unknown word information. It is a block diagram which shows the structure of 6th Embodiment. It is a flowchart which shows operation | movement of 6th Embodiment. It is a general block block diagram of the information processing system which implemented the unknown word determination system by this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Document data input means 2 Memory | storage device 3 Data processing device 20 Document data memory | storage part 21 Word delimiter expression dictionary memory | storage part 22 Word dictionary memory | storage part 23 Unknown word judgment rule memory | storage part 24 Non-delimiter expression dictionary memory | storage part 25 Word-division completed document data memory | storage Section 30 Character type division means 31 Division position correction means 32 Unknown word determination means 33 Non-delimitation expression consideration division position correction means 34 Dictionary attribute estimation means 35 Original form estimation means 4 Unknown word display means 5 Word divided document data input means 6 Unknown word information Display 300 processor 301 program memory 302 storage medium

Claims (51)

Character type dividing means for dividing a Japanese document into character strings according to character types;
An unknown word determination system comprising: an unknown word determination unit that determines the likelihood of an unknown word of a character string that is not registered as a word among the divided character strings. The unknown word determination system according to claim 1, wherein the character type dividing unit does not divide by a specific character type change or character. The unknown word according to claim 1, further comprising: a division position correcting unit that corrects a division position of the hiragana character string divided by the character type dividing unit based on an attached word or a utilization form of the word. Judgment system. The division position correcting means searches the hiragana character string with the longest match from at least one of the front and rear, and if there is an appendage, divides the hiragana character string into an appendix and other than the appendix The unknown word determination system according to claim 3. The division position correcting means searches for a utilization form expression of a word with the longest match from the front with respect to a hiragana character string, and if there is a utilization form expression, combines the utilization form character string with the previous character string. The unknown word determination system according to claim 4. 6. The division position correcting unit divides a hiragana character string by a longest match of a non-delimited expression from the front and the rear, and corrects the division position by an attached word and a utilization form. The unknown word determination system described in 1. The unknown word determination means determines whether the unknown character of the character string is based on whether or not a character string that is not registered as a word among the divided character strings is an attached word. The unknown word determination system according to any one of the above. The unknown word determination means includes the divided character string based on at least one of a partial character string, a character type, the number of characters, a word dictionary, the number of extractions, an extracted utilization type, and an extracted utilization type number. The unknown word determination system in any one of Claims 1-7 which determines the likelihood of the unknown word of the character string among which the word registration is not carried out. 9. The unknown according to any one of claims 1 to 8, wherein the unknown word determination means determines the likelihood of an unknown word of the character string based on a score of a rule that matches the character string in advance by assigning a score to the rule. Word determination system. The unknown word judging means is a rule for identifying an unknown word and a non-unknown word based on a partial character string, a character type, the number of characters, a word dictionary, the number of extractions, an extracted utilization type, and an extracted utilization type number. The unknown word determination system according to claim 1, which is acquired by machine learning. A dictionary attribute estimating means for estimating a dictionary attribute of an unknown word based on the number of dictionary attributes of a word that appears in the same context as the unknown word from a word dictionary in which the word-divided document data and the dictionary attribute are registered The unknown word determination system in any one of Claims 1-10. 12. The unknown word determination system according to claim 11, wherein the dictionary attribute estimation unit uses the same context when several words before and after match. The unknown word determination system according to claim 11 or 12, wherein the dictionary attribute estimation means counts the dictionary attributes of words that appear in the same context as the unknown word, and uses the dictionary attribute with the highest frequency as an estimation result. The dictionary attribute estimation means counts a dictionary attribute of a word that appears in the same context as an unknown word with a weight according to a character string similarity with the unknown word, and sets the most frequent dictionary attribute as an estimation result. The unknown word determination system according to any one of claims 11 to 13. Claims 1 to claim 2 having original form estimation means for estimating the original form of an unknown word and its use form by examining the use form of the unknown word part-of-speech with the longest match from the back and using the basic form of the use form as the original form. The unknown word determination system according to any one of 14. The original shape estimation means converts all unknown words into basic forms using possible usage forms. As a result, a plurality of originally different unknown words are matched as basic forms, and there is only one type of usage form that matches. The unknown word determination system according to claim 15, wherein the original form of those unknown words is used as the basic form, and the utilization form and the original form are estimated. The original shape estimation means converts all unknown words into basic forms with possible utilization forms, and as a result, a plurality of unknown words that were originally different match as basic forms, and if there are multiple types of utilization forms that coincide, By acquiring words of the same usage form as multiple matching usage forms, measuring the similarity of character strings, and using the basic form of the usage form of the word with the most similar character strings as the original form of the unknown word, The unknown word determination system according to claim 15 or 16, wherein the original form is estimated. Information processing device
A Japanese document is divided into character strings according to character types.
An unknown word determination method for determining the likelihood of an unknown word of a character string not registered as a word among the divided character strings. 19. The unknown word determination method according to claim 18, wherein when a Japanese document is divided into character strings, a specific character type change or character is not divided. 20. The unknown word determination method according to claim 18 or 19, wherein the information processing device corrects the division position of the divided hiragana character string based on an attached word or a utilization form of the word. The modification of the division position is performed by searching for an adjunct word with the longest match from at least one of the front and rear of the hiragana character string, and if there is an adjunct, the hiragana character string is divided into an adjunct word and a non-adjunct word The unknown word determination method according to claim 20. 21. The modification of the division position is performed by searching a utilization form expression of a word with the longest match from the front with respect to a hiragana character string, and combining the utilization form character string with a previous character string if there is a utilization form expression. The unknown word determination method according to claim 21. The correction of the division position is performed by dividing the hiragana character string by the longest match of the non-delimited expression from the front and rear, and correcting the division position by the attached word and the usage form. The unknown word determination method described in 1. The determination of the unknown word is performed by determining whether the character string is an unknown word based on whether a character string that is not registered as a word among the divided character strings is an attached word. The unknown word determination method according to any one of the above. The unknown word is determined based on at least one of a partial character string, character type, number of characters, word dictionary, number of extractions, extracted utilization type, and extracted utilization type number. The unknown word determination method according to any one of claims 18 to 24, wherein an unknown word likelihood of a character string that is not registered as a word is determined. The unknown word is determined according to any one of claims 18 to 25, in which the unknown word is scored in advance, and the unknown word likelihood of the character string is determined based on a score of a rule that matches the character string. Word determination method. The determination of the unknown word is a rule for identifying an unknown word and a non-unknown word based on a partial character string, a character type, the number of characters, a word dictionary, the number of extractions, an extracted usage type, and an extracted usage type number. 27. The unknown word determination method according to any one of claims 18 to 26, wherein the word is acquired by machine learning. The information processing device estimates a dictionary attribute of an unknown word based on the number of dictionary attributes of a word that appears in the same context as the unknown word from a word dictionary in which word-divided document data and dictionary attributes are registered The unknown word determination method according to any one of claims 18 to 27. 29. The unknown word determination method according to claim 28, wherein the estimation of the dictionary attribute of the unknown word uses the same context when several words before and after are matched. 30. The unknown word determination method according to claim 28 or 29, wherein the estimation of the dictionary attribute of the unknown word is performed by counting dictionary attributes of words appearing in the same context as the unknown word and using the dictionary attribute having the highest frequency as an estimation result. . The estimation of the dictionary attribute of the unknown word is performed by counting the dictionary attribute of the word appearing in the same context as the unknown word with a weight according to the character string similarity with the unknown word, and determining the most frequent dictionary attribute as the estimation result. The unknown word determination method according to any one of claims 28 to 30. The information processing apparatus estimates the original form of an unknown word and its utilization form by examining the utilization form of the unknown word part-of-speech with the longest match from the rear and using the basic form of the utilization form as the original form. 31. The unknown word determination method according to any one of 31. The original form of the unknown word and its utilization form are estimated by converting all unknown words into a basic form with possible utilization forms. 33. The unknown word determination method according to claim 32, wherein if there is only one kind of word, the original form of those unknown words is used as its basic form, and the inflected form and the sentence reduction are estimated. The estimation of the original form of the unknown word and its utilization form is the result of converting all unknown words to the basic form with possible utilization forms. If there are multiple types of words, acquire words that have the same utilization form as the matched utilization forms, measure the similarity of the character strings, and use the basic form of the utilization form of the word that has the most similar character strings as the original form of the unknown word The unknown word determination method according to claim 32 or claim 33, wherein the utilization form and the original form are estimated. Character type division processing that divides a Japanese document into character strings according to character types;
A program that causes an information processing apparatus to execute an unknown word determination process that determines an unknown word likelihood of a character string that is not registered as a word among the divided character strings. 37. The program according to claim 36, wherein the character type dividing process does not divide by a specific character type change or character. 37. The information processing apparatus according to claim 35 or 36, wherein the information processing apparatus executes a division position correction process for correcting a division position of the hiragana character string divided by the character type division process based on an attached word or a word utilization form. The program described in. The division position correction process searches the hiragana character string for the longest match from at least one of the front and the back, and if there is an appendage, divides the hiragana character string into the appendix and other than the appendix The program according to claim 37. The division position correcting process searches the hiragana character string with the longest match from the front for the word utilization form expression, and if there is a utilization form expression, combines the utilization form character string with the previous character string. The program according to claim 38. 40. The division position correction process divides a hiragana character string from the front and the rear by the longest match of the non-delimited expression, and corrects the division position by the attached word and the utilization form. The program described in. The unknown word determination process determines whether the unknown character of the character string is based on whether a character string that is not registered as a word among the divided character strings is an attached word. The program according to any one. The unknown word determination processing is based on at least one of the partial character string, character type, number of characters, word dictionary, number of extractions, extracted utilization type, extracted utilization type number, and the divided character string. 42. The program according to any one of claims 35 to 41, wherein the likelihood of an unknown word of a character string not registered as a word is determined. 43. The program according to any one of claims 35 to 42, wherein in the unknown word determination process, a rule is scored in advance, and an unknown word likelihood of the character string is determined based on a score of a rule that matches the character string. . The unknown word determination processing is a rule for identifying an unknown word and a non-unknown word based on a partial character string, character type, number of characters, word dictionary, number of extractions, extracted usage type, extracted usage type number 43. The program according to claim 35, wherein the program is acquired by machine learning. Performs dictionary attribute estimation processing to estimate the dictionary attribute of unknown words based on the number of dictionary attributes of words that appear in the same context as the unknown word from a word dictionary in which word-divided document data and dictionary attributes are registered 45. The program according to claim 35, which is executed by an apparatus. 46. The program according to claim 45, wherein the dictionary attribute estimation processing is performed in the same context when the preceding and following words match. 47. The program according to claim 45 or claim 46, wherein the dictionary attribute estimation process counts dictionary attributes of words appearing in the same context as the unknown word, and uses the dictionary attribute having the highest frequency as an estimation result. The dictionary attribute estimation processing counts a dictionary attribute of a word that appears in the same context as an unknown word with a weight according to a character string similarity with the unknown word, and uses the most frequent dictionary attribute as an estimation result. The program according to any one of claims 45 to 47. A request that causes the information processing device to execute the original form estimation process for estimating the original form of the unknown word and its use form by examining the use form of the unknown word part-of-speech with the longest match from the back and using the basic form of the use form as the original form Item 50. The program according to any one of Items 35 to 48. In the original shape estimation process, as a result of converting all unknown words into basic forms with possible usage forms, a plurality of unknown words that were originally different are matched as basic forms, and there is only one type of usage form that matches. 50. The unknown word determination system according to claim 49, wherein the original form of those unknown words is used as its basic form, and the inflected form and the sentence reduction are estimated. The original shape estimation process is a result of converting all unknown words into basic forms with possible utilization forms.If a plurality of unknown words that were originally different match as basic forms, and there are multiple types of utilization forms that coincide, By acquiring words of the same usage form as multiple matching usage forms, measuring the similarity of character strings, and using the basic form of the usage form of the word with the most similar character strings as the original form of the unknown word, The program according to claim 49 or 50, wherein the original form is estimated.

Images