JPH03263260A - Japanese language analyzer - Google Patents

Abstract

PURPOSE:To infer the attribute of a proper noun and to decide an unregistered word range by performing unregistered word processing centering the detection of the proper noun, and detecting the attribute by the information of a word cogenerated with the proper noun. CONSTITUTION:Character string data comprising a Japanese language text is inputted to an input means 1, and is stored in an input character string storage part 2. Also, the word and the attribute data of the word are stored in a word dictionary part 3, and the word to be connected to the proper noun and the attribute of the proper noun are stored in a cogenerative word table 4. In such a case, a morphemic analysis control part 14 performs analysis by referring the character string data stored in the storage part 2 to the dictionary part 3. When no word equivalent to a character string targeted to be analyzed exists or no connection by the word in a connection table 6 is performed, the word is judged as an unregistered word, and the unregistered word processing is started from the word in the character string in which the word equivalent to the character string targeted to be analyzed exists. Here, an unregistered word detection control part 11 decides the temporary range of the unregistered word at an unregistered word temporary detecting part 8, and the attribute of a detected proper noun is inferred.

Description

DETAILED DESCRIPTION OF THE INVENTION [Industrial Field of Application J] The present invention relates to a Japanese language analysis device that analyzes the part of speech and structure of each word constituting a Japanese sentence.

[Background Art] In recent years, there has been an increasing demand for natural language processing technology represented by machine translation. Among these, morphological analysis of sentences containing kanji and kana is the first process performed when analyzing Japanese, and its accuracy affects the overall analysis accuracy. By the way, processing of unregistered words poses a major problem during morphological analysis.

There are two ways to extract unregistered words from a sentence: analyze from the beginning of the sentence and define the range where the same character type continues from the point where dictionary lookup fails as unregistered words, or skip the failed character and move on to the next one. Generally, a dictionary is searched from a character, and this process is repeated until a word can be extracted, and at the point when a word can be extracted, the characters up to the character immediately before that word are treated as unregistered words.

[Problem to be solved by the invention] In the method described above, in which the range of consecutive characters of the same type is treated as unregistered words, for example, if there is a character string "Prime Minister Nakasone", "Dictionary lookup at Nakako" If you fail, "Prime Minister"
Even though there is a registered word, the entire word is unregistered. Also, in the same example, if you think about how to look up a dictionary by shifting - characters, on the third try,
When the word ``Prime Minister'' is drawn, the word ``Nakasone'' before it will be recognized as an unregistered word, but depending on the way the letters are arranged, there is a possibility that the word may be extracted incorrectly.

Measures to Solve Problem 1] Looking at the words detected as unregistered words, most of them are nouns, and among these, proper nouns (names of people, places, organizations) seem to account for a large proportion. Furthermore, these proper nouns rarely appear alone in a sentence, but often appear with co-occurring words (eg, prime minister, president, city, bank, etc.).

Focusing on this, and in order to solve this problem, the present invention includes a co-occurrence word table that stores words connected to proper nouns and attributes of proper nouns to which these words connect, and the co-occurrence word table is stored in a storage means. An analysis means for analyzing character string data by referring to a word dictionary; and a structure configured to refer to the co-occurrence table to obtain attributes of character strings connected to words that cannot be analyzed by the analysis means and are stored in the co-occurrence word table. be done.

[Operation j] By performing unregistered word processing that focuses on detecting proper nouns using the above method, the possibility of incorrect analysis of unregistered words in a sentence is reduced, and the information possessed by words that co-occur with proper nouns is reduced. It is possible to estimate the attributes (person name, place name, organization name) of the proper noun detected by this method.

Embodiment] A Japanese language analysis device according to an embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a functional block diagram of a Japanese language analysis device according to an embodiment of the present invention. In the figure, ] is a means for inputting Japanese character strings. 2 is an input character string storage unit that stores Japanese character strings input from the input means 1; 3 is a word dictionary unit that holds morpheme information including word readings, 2 spellings, 2 part-of-speech information; and 4, which will be described later. , the spelling and pronunciation of words that are likely to co-occur with proper nouns such as people's names, place names, and organization names, as well as co-occurring words that have the attributes of co-occurring proper nouns (person names, place names, and organization names mentioned above). In the table, 5 is a prefix table that registers words that are often used as prefixes of co-occurring words.

Reference numeral 6 represents a connection table having information on whether or not connections between words are possible. 7 is a dictionary that sequentially compares the character string to be analyzed in the input character string with the prefix table 5, the co-occurrence word table 4, and the word dictionary 3 from the beginning to search for the word that matches the character string to be analyzed the longest. This is a table search section. Reference numeral 8 denotes an unregistered word provisional range detection unit that detects a provisional unregistered character string range using the method described later when the dictionary search fails, and 9 represents a proper noun that detects a proper noun from the unregistered character string using the method described later. The noun detection unit and further 10 are a proper noun attribute determination unit that estimates the attributes of the proper noun from the information of the co-occurring words detected by the proper noun detection unit 9. Reference numeral 11 denotes an unregistered word detection control section that determines the range of unregistered words using the unregistered word provisional range detection section 8 and the proper noun attribute determination section 10. Further, 12 is an analysis result storage unit that stores the results of morphological analysis, 13 is an output display unit, and 14 is a morphological analysis control unit that controls Japanese morphological analysis.

The operation of the apparatus configured as described above will be described below. First, the input Japanese character string is analyzed from the beginning. The dictionary search section of the dictionary/table search section 7 searches the word dictionary 3 from the beginning of the character string to be analyzed, and searches for the longest matching word. Once a word has been extracted, the search is performed again starting with the next character of the word to extract the word. Then, when the next word is extracted, connection table 6
is used to check whether the previous $g14 and the following word can be connected, and if they are connectable, the analysis of that part is correct and the next word is extracted. here,
If the word corresponding to the character string to be analyzed cannot be found in the dictionary, or if the connection table indicates that it cannot be connected to the previous word, it is determined that there is an unregistered word. Then, from the point where it is determined that there is an unregistered word,
Enters unregistered word processing.

The following description will be given with reference to the flowchart shown in FIG. The unregistered word detection control section 11 determines a tentative unregistered word range using the unregistered word tentative range detection section 8 according to the criteria shown in FIG. Furthermore, step s 11 is performed by the proper noun detection unit 9 for the tentative range of unregistered words set according to the above criteria.
1, the dictionary/table search unit 7 searches for words registered in the co-occurrence word table shown in FIG. 4 from the end of the tentative unregistered word range. If a matching word is found in step s1, the process moves to step sla and the character immediately before it is looked at to see if there is a word registered in the prefix table of FIG. 5. If there is a corresponding character, the part before that character is set as an unregistered word in step slb.

If there is no matching prefix, in step slc, the words from the beginning to before the registered word in the co-occurrence word table are treated as unregistered words.

In step sld, the proper noun attribute determination unit 10 determines the attribute of the detected proper noun from the “attribute” information in the co-occurrence word table, and outputs the information to the analysis result storage unit 12.

In step s2, it is determined whether or not there is a co-occurring word. If there is no co-occurring word, in step S2a, the previous word in the tentative unregistered word range is extracted and compared with the co-occurring word table. If there is a match, the entire temporary range is set as an unregistered word in step s2a, and the process moves to step sld. This process takes into consideration the case where a co-occurring word comes before a proper noun. If no co-occurring word is found before the proper noun, the entire tentative range is set as an unregistered word in step s26.

Furthermore, actual processing will be explained in detail by giving a specific example.

Consider the following example sentences. ``...is the former Prime Minister Nakabonene.
In Beijing on February 25th...'' In this example, if the dictionary lookup fails at ``中'', the unregistered word provisional range detection unit 8
Based on the criteria shown in Figure 3, we determine that ``the prime minister of the middle bone root'' is the provisional range. For this part, the dictionary/table search unit searches the co-occurrence word table in Figure 4 from the end of the character string, and
"Prime Minister" is found (step 51) 0 there "Prime Minister"
If you search the prefix table in Figure 5 for the character ``Gen'' before ``, a match will be found. Therefore, the unregistered word detection control unit 11 determines that the unregistered word in this case is the part of "Nakasone" (step 5lb). Furthermore, since the attribute of "Prime Minister" in the table is "Person's name", this result is output to the analysis result storage unit 12 as shown in Fig. 6 (step s1d). Consider the example of "...is former Prime Minister Nakasone..."
In the example sentence above, if the dictionary look-up fails at the middle letter r as before, the tentative range is ``Nakasone'', and a search for co-occurring words starts after this, but in this case it is not found. ! (Step s2). Therefore, if you search for the word "Prime Minister J" in front of this range in the co-occurrence word table, you will find a match.Therefore, "Nakasone" is treated as an unregistered word, and the attribute "Person's name" of "Prime Minister" is added to the analysis result. This result is output to the storage unit 12.

[Effects of the Invention] As described above, the present invention provides a co-occurrence word table that stores words connected to proper nouns and attributes of proper nouns to which the words are connected, so that when there is a word that is not registered in the dictionary, , the co-occurrence word table is searched for words connected to that word and the attribute information of the unregistered word is obtained, making it possible to accurately determine the range of unregistered words and estimate the attributes of the detected proper noun. I can do it.

[Brief explanation of drawings]

Fig. 1 is a block diagram of a Japanese language analysis device according to an embodiment of the present invention, Fig. 2 is a flowchart showing a control procedure in an embodiment of the present invention (Fig. 3 is a flowchart showing the control procedure in this embodiment), and Fig. 3 is a block diagram of a Japanese language analysis device according to an embodiment of the present invention. 4 is a diagram showing a co-occurrence word table, FIG. 5 is a diagram showing a prefix table, and FIG. 6 is a diagram showing an example of an output result in an embodiment of the present invention.1 ...Input means, 2...Input character string storage section, 3...
・Word dictionary part, 4...Co-occurrence word table, 5...Prefix table, 6...Connection table, 7...Dictionary・
Table search unit, 8... Unregistered word temporary range detection unit, 9.
... Proper noun detection unit, 10... Proper noun attribute determination unit,
11... Unregistered word detection control unit 1 2... Analysis result storage unit, Figure 3... Output display unit, 4... Morphological analysis control unit.

Claims (1)

[Scope of Claims] An input means for inputting character string data constituting a Japanese sentence, a storage means for storing the character string data input from the input means, and a word dictionary storing words and attribute data of the words. a co-occurrence word table storing words connected to proper nouns and attributes of proper nouns connected to the words; and morphological analysis for analyzing character string data stored in the storage means with reference to the word dictionary. and means for obtaining attributes of character strings connected to words that cannot be analyzed by the morphological analysis means and are stored in the co-occurrence word table by referring to the co-occurrence table. Analysis device.