JPH05135096A - Morpheme analyzing system - Google Patents

Abstract

PURPOSE:To also extract the wrong characters or missed characters of a sen tence while enabling high-speed retrieval and reducing memory capacity. CONSTITUTION:A character index is generated to sort all the characters contained in the respective words of a word dictionary one by one and to define preserving positions in the respective words of the word dictionary as an index list and for the morpheme analysis of an input text, the index list of each character is extracted from the character index concerning the character string of the input text. When the index list is continued, the character string is judged as one word and concerning the character string for which the word is not judged, it is judged that wrong characters, missed characters or any frequent character exists at the discontinuous part.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a morphological analysis method for natural language processing.

[0002]

2. Description of the Related Art Natural language processing has been applied to word processors and machine translations. As a processing process, first, morphological analysis is performed to divide input text into words to give part-of-speech information and semantic information, followed by syntactic processing ( Parsing) is performed, and semantic processing and context processing are performed to remove the ambiguity and vagueness that remain in these processes.

Conventional morphological analysis methods include a method using a finite automaton, a method using associative memory, and a method using a neuro technique.

In the finite automaton method, an automaton is generated from a state transition diagram of character matching, and a word of a character string is understood by software or wired logic.

In the associative memory system, a full text search is performed by providing a search function to the memory body and comparing character strings cell by cell.

In the method using the neuro technology, full text search is performed while learning is performed by a neuro computer.

[0007]

In the conventional morphological analysis method, the finite automaton method is capable of missing characters in a character string and fuzzy search by using a world card.
There was a problem that it was difficult to handle when the first character of the character string became ambiguous. Further, in the case of a hardware configuration, various search requests of application software cannot be handled, and in the case of a software configuration, the search speed may be inferior to other methods.

Next, the associative memory system requires a large-capacity memory body, resulting in high cost. It is technically very difficult to use a mass storage device such as a disk for the memory body, and the memory body is currently 1 MB.
Less than Further, it is difficult to combine with software, especially with an application, so that a program on the application side also becomes complicated for searching a complicated sentence.

Next, in the neurocomputer method, although it takes time to search until learning, after learning, high-speed searching can be performed without being much influenced by the amount of documents, and fuzzy searching is also possible. However, the problem is that if the search is for a solid text image, it will be a fast search, but it will be difficult to deal with a database such as a structured electronic dictionary.

An object of the present invention is to provide a morphological analysis method which enables extraction of typographical errors or omissions of sentences while achieving high-speed retrieval and a small-capacity memory configuration.

[0011]

In order to solve the above-mentioned problems, the present invention provides a word dictionary for storing words of Japanese characters in a notation character and a structure of an attribute list, and a notation character in the word dictionary. A character index of a structure having a character in which all the included characters are sorted for each character and an index list which is a storage position in the notation character, and a processing device which analyzes a morpheme of an input text to be a morpheme analysis target. The processing device extracts the character of the character index that matches each character of the input text and the index list, extracts the character string having the consecutive storage positions as one word, and extracts the character string that fails to be extracted. When the storage positions are not consecutive by one character or a predetermined number of characters, it is determined that any one of the one character or the predetermined number of characters includes an erroneous character, a missing character, or a multi-character.

[0012]

According to the present invention, a character index is generated for each character stored in the word dictionary, and the continuity of the storage position is determined from the index list of the character index for each character forming the character string of the input text. By determining the presence or absence, it is analyzed whether or not the character string is one word. Also, when one character or a predetermined number of characters that are not continuous are included, it is determined that the character or the character string becomes an erroneous character.

[0013]

1 is a block diagram showing an embodiment of the present invention. The file-structured word dictionary 1 is stored with a notation of words including hiragana, katakana, alphanumeric characters, single kanji, and idioms and a structure of its attribute list. For example, in the structure pointer K, as shown by the structure 1k, the idiom "Akiyu" is stored in the notation, "noun" is stored in the part of speech of the attribute list, and "akime" is stored in the reading.

The character index 2 of the file structure is generated by the processing device 3 using the word dictionary 1 as a database, and an index list for each character of all the written characters of the word dictionary 1 is stored. For example, the pointer H has a member character of "autumn" as shown in the configuration 2n.
However, the index list stores the address of the character "autumn" in a word having the character "autumn". This index list is the storage position of the letters "autumn" that compose the words of the word dictionary 1 that include the letters "autumn". For example, the words "autumn rain,""midautumn," and "autumn moon" are stored in the word dictionary 1. When stored, the address "A ₁ " of the character "autumn" in each word, because each contains the character "autumn",
“A ₈ ” and “A ₂₁ ” are generated and saved as save position data.

The procedure for generating the character index 2 is to extract the position (address) from the beginning of each word in the dictionary file for all the notation characters in the word dictionary 1, collect the information for each character, and sort it. Then, the index pair indicating the character and the address is collected and stored in the character index file. For this storage allocation, for example, a hash function that avoids collision of storage positions is used, and it is also constructed in a high-speed search environment.

After the processing device 3 has generated the character index 2 from the word dictionary 1, the interface 4
Morphological analysis processing is performed on the input text (texts mixed with kana and kanji) given through.

This process is executed according to the procedure shown in FIG.
First, the input text is subjected to morphological analysis by the longest match method or the like (step S1). For this processing, character index 2 is used for matching with the dictionary, and a character string consecutive from the first character of the input text is extracted character by character from character index 2 and the index list is taken out in order. If the distances of the list strings are all 1 and there is the longest one, it is determined that the character string exists in the word dictionary 1, and the character string is determined as the morphological analysis list.

This process will be described with reference to the example shown in FIG. In the figure, (a) shows a case where the written portion of the word “Southeast Asia” stored in the word dictionary 1 is stored at addresses “a” to “a + 4”. For this word, in the character index 2, as shown in FIG. 2B, the address "a + 2" and "a +" are added to the index list for the character "a".
"4" is written with other addresses, and the character "Ji"
For, the address “a + 3” is written in the index list together with other addresses, the address “a” is written in the character “east”, and the address “a + 1” is written in the character “south”.

Here, in the morphological analysis, when the input text includes the character string "Southeast Asia", the index list of the character string "Southeast Asia" is read from the character index 2 and the address "a +" included therein is read.
It is recognized that the distances between adjacent characters are all 1 from "2", "a + 4", "a + 3", "a", "a + 1".
For example, the distance between the letters "East" and "South" is "a + 1"-"a".
= 1. Therefore, it can be determined that the character string "Southeast Asia" exists in the word dictionary 1, and it is listed in the morpheme list as one word.

Returning to FIG. 2, the input text is extracted as a morpheme list for each word by the process of step S1. However, there are some characters that fail in this morpheme analysis. Character strings and characters that have failed in this analysis are extracted as a single Kanji character string list (step S2).

It is detected / corrected whether or not there are erroneous characters, omissions, and multiple characters in the extracted character strings and characters (step S).
3). Among them, the typographical error is detected by referring to the character index 2 for each character of the detection target character string, reading the index list, and obtaining the distance between the addresses of the preceding character and the succeeding character. When there is a character other than 1, the character is skipped to obtain the address distance to the next character, and when the distance from the previous character becomes 2, the skipped character is determined to be a typographical error.

For example, if the character string for which the morphological analysis has failed is "Tohkoh Asia", the distance between addresses in the index list of the characters "Higashi" and "Soft" is not 1. At this time, when the character "soft" is skipped and the distance between the next character "a" and the previous character "east" is checked, it becomes 2. Therefore, the character "soft" is determined to be a typographical error.

Next, the detection of missing characters is performed by using 1 in the character string to be detected.
The character index 2 is referred to for each character, the index list is read, the distance between addresses with the previous character is calculated, and when there is a distance that becomes 2, it is determined that there is a caret between both characters. ..

For example, if the character string for which the morpheme analysis has failed is "southeast aa", the inter-address distance between the third character "a" and the fourth character "a" is 2. , It is determined that there is a missing character between the two characters "A" and "A".

Next, the multi-character detection is performed by using 1 of the character string to be detected.
When the index list for each character is read and the distance between the previous character and the address is not 1, skip that character and find the distance to the next character, and if this distance becomes 1. The skipped characters are judged to be multi-characters.

For example, if the character string for which the morphological analysis has failed is "Southeast Soft Asia", the distance between the characters "South" and "Soft" is not 1, so the character "Soft" is skipped and the character "A" is skipped. ", And the character" south "is calculated, and this distance is 1
Therefore, the character "soft" is determined to be a multi-character.

Returning to FIG. 2 again, the character string in which erroneous characters, omissions and multi-characters have been detected / corrected in step S3 is returned to the morpheme list obtained by the analysis in step S1 and correctly morpheme analyzed. It is taken out as a candidate list of character strings. This candidate list is subjected to a connection check process with another character string, and the morphological analysis ends (step S4).
In this connection check process, for example, the part of speech for the previous word is checked.

As described above, in this embodiment, the character index is generated from the word dictionary 1, and the morpheme analysis is performed from the continuity of the distance between the addresses of the analysis target character string, and the erroneous character, the missing character, and the multiple character are detected. ..

Therefore, as compared with the character string collation with the word dictionary, the word containing the character can be directly searched from the character index for high-speed analysis, and when the first character of the text becomes ambiguous. Typographical errors, omissions, including analysis of
Make it easy to check multi-characters.

The memory capacity may be any as long as the character index can be secured, and a relatively small capacity such as the internal memory of the computer is sufficient, and the program on the application side is not complicated.

Further, it is possible to easily deal with analysis of a database such as an electronic dictionary.

Although the embodiment shows the case of detecting one erroneous character, omission, and multi-character, it can be applied to detection of n characters (two or three characters) erroneous character, omission, and multi-character. ..

For example, in detecting erroneous characters of n characters, when there is a character whose inter-address distance is not 1, the inter-address distance is obtained by skipping n characters, and when this becomes n + 1, the skipped n characters are erroneous characters. judge.

Similarly, when detecting a multi-character of n characters, when there is a character whose distance is not 1, the n characters are skipped to obtain the distance, and when the distance becomes 1, the n characters between them are determined to be multi-characters. .. In addition, in the multi-character detection of n characters, it is determined that there is a character whose distance is not 1 and there is a missing character between the characters when the distance is n + 1.

Regarding the above-mentioned n-character erroneous characters and omissions, it is possible to detect erroneous characters and omissions at the end of a word by checking the total number of characters. To this end,
As shown in, the number of characters of the word containing the character and the character position in the word are added as members to the character index list in addition to the address data, and the nth character matches the number of word characters in the dictionary and the character position. Whether or not it is added to the judgment.

[0036]

As described above, according to the present invention, morphological analysis and typographical errors are detected from the continuity of the storage position of the word dictionary extracted from the character index of the character string of the input text.
Since the detection of punctuation and multi-characters is performed, the search speed is higher than that of the analysis which is a collation of a word dictionary and a character string. Also, a small memory can be secured for the character index. Multi-character verification can be performed.

[Brief description of drawings]

FIG. 1 is a configuration diagram showing an embodiment of the present invention.

FIG. 2 is an analysis processing procedure diagram of the embodiment.

FIG. 3 is a mode diagram of morphological analysis according to an embodiment.

FIG. 4 is a diagram illustrating another embodiment.

[Explanation of symbols]

 1 ... Word dictionary, 2 ... Character index, 3 ... Processing device.

Claims (1)

[Claims] 1. A word dictionary that stores words of Japanese characters as a notation character and a structure of an attribute list, a character in which all the characters included in the notation character in the word dictionary are sorted, and the notation character A character index of a structure having an index list that is a storage position in the inside, and a processing device that analyzes a morpheme of an input text that is a morpheme analysis target, the processing device including the character that matches each character of the input text When the character string of the index and the index list are extracted, the character string in which the storage positions are continuous is extracted as one word, and when the storage position is not continuous by one character or a predetermined number of characters in the character string that has failed to be extracted, A morphological analysis method for determining whether any one of the one character or a predetermined number of characters is erroneous, missing, or polymorphic.