JP3143906B2 - Device for determining the presence of unknown words - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION [Industrial Application Field] The present invention relates to a system that requires processing of a character string such as a word processor, machine translation, and inquiry to a database. The present invention relates to a device for determining the presence of an unknown word that is used to determine whether or not there is an unknown word.

[Conventional technology]

The conventional determination device that determines the presence of an unknown word compares the character string of a sentence with the character string registered in the dictionary, attempts to divide the character string in various ways, and registers it in the dictionary no matter how it is divided Is a device that determines that there is an unknown word if a portion that does not match the given character string is found. Here, the unknown word means a character string that is not registered in the dictionary.

The following example is given as an example of a case that can be solved by a conventional device.

In the dictionary, "Kyoto" (Tokyo) and "Tokyo" (Tokyo) are registered. If "East" (Higashi), "Miyako" (Miyako) and "Tokyo" are not registered, the character string ""Tokyo" means "Tokyo""Tokyo" or "East""Kyoto"
Is divided. The character strings “East” and “To” are not registered in the dictionary, and “East” and “To” are parts that do not match the character strings in the dictionary. Therefore, it is determined that an unknown word exists in the character string “Tokyo”. In this case, the unknown word is any of "east", "city", and "Tokyo".

Further, as an example in which the conventional apparatus cannot solve the problem, there is the following example.

The dictionary includes "East" (Higashi), "Kyoto" (Tokyo),
If character strings "Tokyo" (Tokyo) and "Miyako" (Miyako) are registered and "Tokyo" is not registered, the unknown word "Tokyo" is not an unknown word in the conventional device. "East""Kyoto" (Today) or "Tokyo"
It is said to be "Miyako" (Tokyo), and it is not clear that there is an unknown word "Tokyo" (Tokyo).

[Problems to be solved by the invention]

In the conventional device described above, unknown words can be detected by:
Only when a part that does not match the character string registered in the dictionary is found. Therefore, in the conventional apparatus, if there is no portion that does not match the character string registered in the dictionary, it is not possible to determine that there is an unknown word even if an unknown word exists, and in particular, the number of character strings registered in the dictionary increases. There is a problem in that the presence of an unknown word cannot be determined as the number of words increases.

[Means for solving the problem]

The apparatus for determining the presence of an unknown word according to the present invention includes: (A) the number of characters constituting a first character string input for determining the presence or absence of an unknown word, the number being included in the first character string; And is used to determine whether or not the unknown word is present in the first character string, which is expressed using the number of kanji characters and the number of consecutive kanji characters. Code generation means for generating a first unknown word determination code,
(B) generating a plurality of second character strings by dividing the first character string, and converting each of the second character strings in a character string group composed of the plurality of second character strings into: The third character string previously added to the third character string with respect to the second character string that matches with the third character string by collating with a third character string registered in advance Acquiring the corresponding part of speech of the sequence, and determining the score given to the part of speech determined in advance for a predetermined evaluation of the part of speech as an independent word, the second character that matches the third character string If the second character string that does not match the third character string is assigned to the first character string, the first character string is cut again by a different cutting method from the previous cutting method, and a plurality of second character strings are cut. Regenerate the character string and generate the second
When there is the second character string that does not match the third character string, such as comparing each character string with the third character string, a different dividing method is used each time. The first character string is cut again to generate a plurality of second character strings, and the generated second character strings are repeatedly compared with the third character strings. The first character string is divided according to all possible division methods for the character string of the above, and even if the second character string is compared with the third character string, the first character string still does not match the third character string. If there is a second character string, it is determined that there is an unknown word in the first character string, and the dividing operation is stopped, and the first character string is divided to generate a plurality of second character strings. Then, the first character string is cut and generated. When said plurality of second string against the each of the third string, if the third of the second character string that does not match the string runs out,
When outputting as a result of analysis a plurality of the character string groups obtained by segmentation by all different segmentation methods performed until there is no second character string that does not match the third character string, A morphological analysis unit that adds all the scores of the character string group given to the second character string, and outputs the obtained score as the score of the character string group, together with the plurality of character string groups; (C) a dictionary in which the third character string, the part of speech corresponding to the third character string, and the score assigned to the part of speech are registered, (D) the morphological analysis unit outputs Among the plurality of character string groups indicating the analysis result, the number of the character string groups to which the smallest score assigned to each group is assigned is counted, and the obtained number is output as a first numerical value. Analysis number counting means to perform (E) input Created using various combinations of the predetermined number of characters, the number of kanji characters, and the number of consecutive kanji characters, which are referred to when determining the presence or absence of an unknown word in the first character string. A second unknown word determination code, a numerical value that is given to the second unknown word determination code and is a numerical value that becomes a reference for evaluation when determining whether or not the unknown word exists, and is based on the first numerical value. An unknown word determination code table in which, when the value is small or equal to the first numerical value, a second numerical value, which is a numerical value that can determine that an unknown word exists in the first character string, is registered; F)
In order to determine the presence / absence of an unknown word in the first character string, the second unknown word coincident with the first unknown word determination code given to the first character string by the code generation means. Refer to the word determination code in the unknown word determination code table,
The second numerical value given to the second unknown word determination code is obtained, and the first numerical value output from the analysis number counting means and the second numerical value obtained from the unknown word determination code table are obtained. Comparing the second numerical value with the first numerical value, if the second numerical value is smaller than the first numerical value or equal to the first numerical value, determining that an unknown word exists in the first character string;
When the numerical value of the first character string is larger than the first numerical value, a comparison determining unit that determines that there is no unknown word in the first character string is provided.

〔Example〕

Next, embodiments of the present invention will be described with reference to the drawings.

 FIG. 1 is a configuration diagram of one embodiment of the present invention.

As shown in FIG. 1, the apparatus for determining the presence of an unknown word according to the present invention comprises a morphological analyzer 1, a dictionary 2, an analysis counter 3, an unknown word determination table 4, and a comparator 5.

 FIG. 2 shows the operation flow.

FIG. 3 is a diagram showing an outline of the processing of the morphological analyzer 1, and an example of a dictionary used at that time is shown in FIG.

Note that morphological analysis is to determine the presence of a word in a character string. For example, to determine a word in a sentence, such as "I've finished talking" to "I'm done talking". It is to judge that "the story gas is seen" or "the story is more charcoal" is not the correct answer. An apparatus that performs such morphological analysis is called a morphological analyzer.

Also, in the above example, "talk is over"
I have seen it, "and" the story has benefited. " A correct answer and a group expected to be a correct answer among the analysis results are referred to as analysis candidates. The number of analysis results is called an analysis number, and the number of analysis candidates is called a candidate number. The dictionary describes a certain character string and its grammatical attribute.

 The morphological analyzer 1 performs the following operation.

First, the input character string is separated. Next, the divided character string is compared with the character string registered in the dictionary. If there is a portion that does not match the character string registered in the dictionary, the portion is cut again. If there is a portion that does not match the character string registered in the dictionary, no matter how it is cut, it is determined that there is an unknown word, and the operation is stopped. If there is no part that does not match the character string registered in the dictionary, the following operation is performed. The character string data that matches the cut portion of the input character string is extracted from the dictionary, and the points in the data extracted from the dictionary are added up. Then, the divided character string and the score are output.

Next, the dictionary 2 is composed of a character string and data, and the data includes a score. The higher the part of speech is the independent word, the higher the score is given.

 The outline of the above is shown in FIG.

The unknown word determination table 4 includes a search code a and a numerical value b. The search code is created based on the number of characters of the analysis character string and the like. The numerical value b is set to be a value proportional to the number of characters. The outline is shown in FIG.

In the morphological analysis, the judgment device attaches a score and selects the one with the smallest score, and if the number of selected analyzes is large, it is assumed that an unknown word exists in the sentence to be analyzed, and the number of analysis is determined. If there are few unknown words, the unknown word does not exist.

In general, the number of characters in one word of a natural language used by humans falls within a certain value. The number of letters and the number of words in the language have a certain distribution. The distribution has the property that "the number of characters tends to decrease as the number of words increases". This property is common not only to Japanese but also to other languages. Few words contain more than 100 characters in one word, but words with about three characters exist in large quantities. If an unknown word exists and the unknown word is decomposed into small units, the number of analysis candidates for the number of characters in the small unit increases,
The number of sentence parsing increases. For example, if the word “mercury column” is unknown, “water”, “silver” and “pillar” are both “mercury”
It can be separated from "pillar", "water" and "silver pillar", and the number of analysis increases. The number of words and the number of analyzes are predicted from the number of characters in one sentence, and if more analysis results are obtained than the predicted values, it can be determined that there is an unknown word.

 The unknown word determination device operates as follows.

First, an input character string is obtained. Then, a code a is obtained by using the number of characters of the input character string, and a result and a score obtained by performing a morphological analysis and separating the input character string are obtained. The lowest score obtained in this way is defined as the first candidate, and the number of the first candidate is set as the numerical value c, and the numerical value c is obtained. Next, the unknown word determination table is drawn using the code a to obtain a numerical value b. Then, the numerical value b and the numerical value c are compared. If number b is number c
If it is smaller or equal, it is determined that there is an unknown word. If the numerical value b is larger than the numerical value c, it is determined that there is no unknown word.

 Next, the flow of the above processing will be described using an example.

If "Tokyo" is an unknown word (not registered in the dictionary), the result is as follows.

First, the morphological analyzer 1 obtains an input character string “living in Tokyo”. Next, a code a is obtained as follows using the number of characters of the input character string.

Number of sentence characters 6 Number of kanji 4 Number of continuous kanji 3 Code a = (6 4 3) Next, morphological analysis is performed, and the result and score of the input character string are obtained. FIG. 7 shows an example of the dictionary used at this time. An example of the result obtained by this morphological analysis is shown below.

Next, the analysis number counting device 3 sets the one with the lowest score among the above as the first candidate. The first candidate has 32 points. Thus:

Next, the number of first candidates is set to a numerical value c, and the numerical value c
Get. That is, since the number of first candidates is two, the numerical value c
= 2. Further, the comparing device 5 obtains a numerical value b by using the code a to consult the unknown word determination table. FIG. 8 shows an example of the unknown word determination table used at this time. The code a is (64 3)
Therefore, the numerical value b = 2. Then, the numerical value b and the numerical value c are compared. Since the numerical value b = 2 and the numerical value c = 2, the numerical value b = the numerical value c. If the numerical value b is smaller than or equal to the numerical value c, it is determined that there is an unknown word. If the numerical value b is larger than the numerical value c, it is determined that there is no unknown word. Here, the numerical value b =
Since it is the numerical value c, it is determined that there is an unknown word.

If "Tokyo" is not an unknown word (registered in the dictionary), the result is as follows.

 First, an input character string "living in Tokyo" is obtained.

Next, a code a is obtained as follows using the number of characters of the input character string.

Number of characters 6 Number of kanji 4 Number of continuous kanji 3 1 Code a = (6 4 3) Next, a morphological analysis is performed to obtain a result and a score obtained by separating the input character string. An example of the result obtained by this morphological analysis is shown below.

Next, among the above, the one with the lowest score is the first candidate. The first candidate has 32 points. Thus:

Next, the number of first candidates is set to a numerical value c, and the numerical value c
Get. That is, since the number of first candidates is one, the numerical value c
= 1. Further, the unknown word determination table is subtracted using the code a to obtain a numerical value b. Here, the code a is (64 3)
Therefore, the numerical value b = 2. Then, the numerical value b and the numerical value c
Compare. Numerical value b = 2 Since numerical value c = 1, numerical value b> numerical value c. Here, since the numerical value b is larger than the numerical value c, it is determined that there is no unknown word.

〔The invention's effect〕

As described above, the present invention makes use of the phenomenon that when the unknown word is present in the sentence, the number of analyzes of the sentence is increased, thereby determining the presence or absence of the unknown word that could not be determined by the conventional device. This has the effect that errors in processing the character string by machine can be reduced.

Also, when a character string is processed by a machine, by notifying humans that there is an unknown word, correction of character string division,
There is an effect that a character string to be registered in the dictionary can be easily found.

[Brief description of the drawings]

FIG. 1 is a block diagram of one embodiment of the present invention, FIG. 2 is a diagram showing the operation of a device for determining the presence of an unknown word, FIG. 3 is a diagram showing an outline of the processing of the morphological analyzer 1, and FIG. Is a morphological analyzer 1
FIG. 5 shows an example of a dictionary used in FIG. 5, FIG. 5 shows a format of a dictionary, FIG. 6 shows a format of an unknown word determination table, and FIG. 7 shows an example of a dictionary used for morphological analysis. FIG. 8 is a diagram showing an example of the unknown word determination table. 1 morphological analysis device, 2 dictionary, 3 analysis number counting device, 4 unknown word judgment table, 5 comparison device.

Claims (1)

(57) [Claims] (A) The number of characters, the number of kanji included in the first character string, and consecutive kanji constituting a first character string input to determine the presence / absence of an unknown word Is determined using the number of characters, and is assigned to the first character string and used to determine whether or not the unknown word exists in the first character string. Code generating means for generating a code, (B) generating a plurality of second character strings by dividing the first character string, and selecting each of the character strings in a character string group composed of the plurality of second character strings. Each of the second character strings is compared with a third character string registered in advance, and the second character string that matches the third character string is assigned to the third character string in advance. The obtained part-of-speech corresponding to the third character string is obtained, and the part-of-speech A predetermined score assigned to the part of speech with respect to a predetermined evaluation is assigned to the second character string that matches the third character string, and does not match the third character string. If there is the second character string, the first character string is cut again by a different cutting method from the previous cutting method to generate a plurality of second character strings again.
When there is the second character string that does not match the third character string, such as comparing each character string with the third character string, a different dividing method is used each time. The first character string is cut again to generate a plurality of second character strings, and the generated second character strings are repeatedly compared with the third character strings. The first character string is divided according to all possible division methods for the character string of the above, and even if the second character string is compared with the third character string, the first character string still does not match the third character string. If there is a second character string, it is determined that there is an unknown word in the first character string, and the dividing operation is stopped, and the first character string is divided to generate a plurality of second character strings. Then, the first character string is cut and generated. When said plurality of second string against the each of the third string, if the third of the second character string that does not match the string runs out,
When outputting as a result of analysis a plurality of the character string groups obtained by segmentation by all different segmentation methods performed until there is no second character string that does not match the third character string, A morphological analysis unit that adds all the scores of the character string group given to the second character string, and outputs the obtained score as the score of the character string group, together with the plurality of character string groups; (C) a dictionary in which the third character string, the part of speech corresponding to the third character string, and the score assigned to the part of speech are registered. (D) the dictionary output from the morphological analysis unit. Among the plurality of character string groups indicating the analysis result, the number of the character string groups to which the smallest score assigned to each group is assigned is counted, and the obtained number is output as a first numerical value. (E) input Created using various combinations of the predetermined number of characters, the number of kanji characters, and the number of consecutive kanji characters, which are referred to when determining the presence or absence of an unknown word in the first character string. A second unknown word determination code and the second unknown word determination code,
A numerical value serving as a criterion for evaluation when judging the presence or absence of the unknown word. If the value is smaller than or equal to the first numerical value, the unknown character is included in the first character string. An unknown word determination code table in which a second numerical value, which is a numerical value that can determine that a word exists, is registered; (F) in order to determine the presence or absence of an unknown word in the first character string, The second unknown word determination code table refers to the second unknown word determination code that matches the first unknown word determination code given to the first character string by the code generation means, and Obtaining the second numerical value given to the determination code, comparing the first numerical value output from the analysis number counting means and the second numerical value obtained from the unknown word determination code table, The second numerical value is smaller than the first numerical value or the second numerical value When it is equal to the numerical value of 1, it is determined that an unknown word exists in the first character string, and when the second numerical value is larger than the first numerical value, it is determined in the first character string. An apparatus for determining the presence of an unknown word, comprising: comparison determination means for determining that an unknown word does not exist.