JPH0816910B2 - Language analyzer - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to a language analysis device, and more particularly to a language analysis device that analyzes a natural language useful for, for example, an automatic translation device.

2. Description of the Related Art When a corresponding Japanese sentence is created from a foreign language sentence such as English, for example, the morpheme of the input English sentence is analyzed, its syntax is analyzed, and its sentence structure is converted. Generate a translation of

When analyzing a sentence morpheme, whether a certain word is used alone or is used as a compound word or a compound word combined with other words is an important issue that influences the analysis result. In the conventional method, dictionary information for a single word and dictionary information for a compound word or a compound word are provided in parallel, and a compound word or compound word that matches a specific condition such as the longest match method is always prioritized. There is a method to select.

In the former conventional method, the number of combinations of candidates to be selected becomes extremely large, and it is necessary to analyze all of them. Therefore, it takes a lot of time for processing, and there is a risk that an incorrect combination can be regarded as a correct answer, which is wasteful.

Also, in the latter conventional method, the usage as an idiom is always selected with priority over the other. Therefore, the usage as an idiom is always selected even in the case of collocations where the degree of connection between words is not high, and there is a high risk of deriving an incorrect analysis result.

In this way, if all possible candidates in morphological analysis are set as solutions, the number of candidates increases, and the number of solutions generated by the subsequent structural conversion and translation generation processing becomes enormous. Such a large number of solution candidates results in a significant reduction in the speed of subsequent processing.

Therefore, in order to improve the efficiency of the entire automatic translation process, it is required to reduce the number of such useless solutions and increase the efficiency of analysis.

Aim The present invention has been made in view of such requirements, and an object thereof is to provide a language analysis apparatus capable of efficiently performing morphological analysis.

Structure In order to achieve the above object, the present invention has a dictionary means for storing dictionary data including morpheme data for words, collocations and idioms, and analysis means for performing morphological analysis on an input sentence by referring to the dictionary means. In the language analysis device having and, the dictionary means, for collocations and idioms,
The analysis unit includes connection degree data indicating a connection degree between words forming a compound word or a compound word, and the analysis unit refers to the dictionary unit for each word included in the input sentence and refers to one word with another word. When a plurality of dictionary data are searched for in the combination, the language analysis device is characterized by referring to the connectivity data and selecting a combination of words with high connectivity. Hereinafter, a specific description will be given based on an embodiment of the present invention.

Referring to FIG. 2, there is shown an overall configuration of an embodiment in which the language analysis device according to the present invention is applied to an English-Japanese automatic translation device. The present invention is effective not only for an English-Japanese automatic translation device that translates English into Japanese, but also for an analysis device for any language that analyzes a sentence of an input language, mainly when translating one language into another language. Needless to say, it is applied to each other.

This embodiment has an input unit 10 for inputting an English text 12 to be translated into Japanese. The input unit 10 is, for example, a keyboard having character keys such as alphanumeric keys and function keys, an optical character reader (OCR) for reading English text recorded on paper, and / or a storage medium such as a magnetic disk. It may include a file storage device or the like for reading the recorded English text.

The English text input by the input unit 10 is
It is read into 14 and pre-processing for translation is performed. Here, recognition of sentences and processing of unknown words are mainly performed. This functions as part of the morphological analysis.

The pre-edited English sentence data is transferred to the morphological analysis unit 16 together with the information obtained by the pre-editing. The morphological analysis unit 16 indexes the word dictionary 18 and divides the sentence into sentences, analyzes the morphemes of the English sentence, performs various processing such as processing of unknown words, proper nouns, expressions of time, expressions of numbers, etc. , And performs the processing of the entire sentence such as recognition of the same rank. The morphological analysis rule is stored in the analysis rule file 36.

The morphologically analyzed English sentence data is transferred to the syntax analysis unit 20 together with the dictionary information obtained by the morphological analysis. The syntactic analysis I unit 20 is a functional unit that applies grammatical rules to English sentence data to analyze the surface structure of a sentence and find all syntactic possibilities.

The English text data that has been parsed by the syntax analysis I unit 20 is sent to the syntax analysis II unit 22 together with the analysis information. Here, a solution is selected by applying a structural description from the surface analysis result obtained by the analysis I. This creates a probable parse tree of the English sentence and creates its structure. These parsing rules are still stored in the parsing rule file 36.

The parsed English sentence data is transferred to the structure conversion unit 24 as parse tree data. The structure conversion unit 24 creates a syntax tree of a corresponding Japanese sentence from a syntax tree that is an intermediate structure of the English sentence, and converts the Japanese sentence into a Japanese base structure that is easy to translate.

The syntax tree data indicating the basic structure of Japanese thus structurally transformed is sent to the translated sentence generation unit 26, and the translated sentence is generated in the latter. This is a functional unit that generates a Japanese sentence from the tree structure of the Japanese syntax tree.

The translated Japanese data, that is, the translated data, is sent to the post-editing unit 30. The post-editing unit 30 uses the information used for the translation process to index the dictionary 18 to correct the translated sentence data, thereby completing a more natural Japanese sentence. This Japanese sentence data is transferred to the output unit 32 and the translated Japanese sentence 34
Is output from the output unit 32. The output unit 32 includes a file storage device such as a printer, a display, and / or a magnetic disk.

The flow of these series of translation processes is controlled by the control unit 38 that controls the entire control of the present apparatus. In the present embodiment, the word dictionary 18 stores dictionary data for English and Japanese words, and describes not only vocabulary but also various information such as relations, that is, co-occurrence relations, meanings, singularity, and parts of speech. ing. The analysis rule file 36 stores rule data for morphological analysis and syntax analysis.

The operation display unit 40 is connected to the control unit 38. The operation display unit 40 gives various instructions to the apparatus from an operator,
For example, it has an operation key such as a translation instruction key and a cursor key, a display and an indicator for visually displaying an input English sentence text, a Japanese sentence as a translation result, intermediate data such as dictionary information, and various instructions to an operator. Note that many of these operation display functions are provided on the keyboard when the input unit 10 has a keyboard, and on the output unit.
If the display 32 is provided with a display, the display may be included in the display.

Referring to FIG. 1, a detailed configuration of the morphological analysis unit 16 is illustrated. The morphological analysis unit 16 has an input interface 104 that interfaces with the input device 100 such as the keyboard of the input unit 10 and the input document file 102. The input interface 104 is provided with an input character string buffer for temporarily storing the English character string data in the form of code data such as ASCII from the input device 100 and the input document file 102, and temporarily storing the character string data. There is.
These input character strings may have been pre-edited by the pre-editing unit 14.

The morpheme analysis unit 16 has a processing unit 106, a dictionary search unit 108, a contradiction resolution rule processing unit 110, and a control unit 112, as illustrated. The processing unit 106 is an analysis processing function unit that performs morphological analysis, and has a searched dictionary information buffer, that is, a dictionary information storage table 120 (FIG. 9). The morphological analysis instructs a dictionary search in accordance with the character string of the search key in order from the beginning of the input character string, stores the dictionary information obtained from the dictionary search unit 108 in the searched dictionary information buffer 120 in accordance therewith, and sets the highest priority flag described later. It is performed by executing a process of priority according to the.

The dictionary search unit 108 is a functional unit that searches the word dictionary 18 based on the search key character string designated by the processing unit 106, extracts dictionary information, and transfers the dictionary information to the processing unit 106.

As shown in the example of the entry information in FIG. 3, the word dictionary 18 stores the highest priority flag in addition to the grammatical information such as the part of speech and conjugation for each word entry. This dictionary is referred to as a priority flag added dictionary file here.
The “highest priority flag” is a flag that indicates the strength of the connection between the words included in the collocations or idioms that make up the dictionary entry, where “0” indicates a weak connection or no connection, and “1” indicates a strong connection. Indicates. by this,
In a sentence, a compound word or a compound word that is determined to be strongly connected is estimated to be a compound word, and if not, the possibility of using the word alone is considered in parallel.

As illustrated in FIG. 3, each entry in the word dictionary 18 is arranged for each of the collocations and idioms, and each of the words constituting each of them, without distinguishing between the singular words and the collocations or idioms. Also, each inflection type is 1
Make up one entry. If there are multiple usage forms, they are registered as separate entries, and the usage form is displayed in the usage section. The same applies to the part-of-speech, which allows registration of a plurality of parts-of-speech and has the respective part-of-speech information. Other information includes countable nouns,
Uncountable distinctions, intransitive verbs, transitive verb distinctions, translations, etc. are registered.

For example, "get" is the original form of the verb, and the highest priority flag is "0". The idiom "get up" is a prototypical verb phrase, and its highest priority flag is "1". Also, the prepositional phrase “up to”
Indicates that the highest priority flag is "1", but the noun phrase "white house" as a collocation has the highest priority flag "0", and the latter indicates that the degree of connection between words is low. In the figure, symbols Indicates a space character.

In this way, the dictionary information searched by the dictionary search unit 108 includes the highest priority flag. When the highest priority flag is "1" for the same character string or overlapping character strings, the contradiction must be resolved. The contradiction resolution rule processing unit 110 performs this contradiction resolution, and this processing is performed by referring to the highest priority flag contradiction resolution rule stored in the analysis rule file 36.

In the present embodiment, the contradiction resolution rule is the following (1) to (3).
Are applied in the order of, and the priority selection is performed by this.

(1) Idioms or words whose parts of speech are verbs, (2) collocations, idioms or words with a large number of constituent characters, (3) collocations, idioms or words whose position in the sentence is earlier.

The usage of the word thus selected, in other words, the analysis unit is displayed in the searched dictionary information buffer 120 of the processing unit 106 as active information. In the activity information, "1" indicates that the analysis unit is valid, and "0" indicates that the possibility is not adopted.

The control unit 112 is a functional unit that controls and controls the operation and processing of each functional unit of the morphological analysis unit 16. This may be included in the control unit 38 that controls the entire apparatus.

The result of the morphological analysis is transferred to the syntax analysis I unit 20 through the output interface 114. When not directly transferred to the syntactic analysis I section 20, the syntactic analysis input file 116 and the syntactic analysis dictionary information file 118 are temporarily stored.

In the present embodiment, during morphological analysis, all words, collocations or idioms starting from the cut-out position of the dictionary lookup unit are extracted, but for collocations or idioms recognized as a group of "units" according to the highest priority flag, it is extracted. The dictionary information obtained for each of the constituent words is discarded. That is, the strength of the connection between the words in the sentence is determined by referring to the highest priority flag of the dictionary information obtained by the morphological analysis. It is presumed that a compound word or a compound word that has a strong connection is considered to be a compound word in the sentence, and if not, the possibility of using the word alone is considered in parallel.

The processing with such a highest priority flag is performed in the sequence as shown in FIG. Receives the input character string data from the input unit 10 (200) and the dictionary file with the highest priority flag 1
In order to index 8, the input character string is cut out in dictionary lookup units (201), the dictionary 18 is searched accordingly (203), and this is performed up to the final position of the sentence indicated by the input character string data (202), the maximum The contradiction of the priority flag is resolved (204), and the morphological analysis result is output to the syntax analysis I unit 20 (205).

In the input process 200, first, the input document file 102 or the input device 100 is read into the input character string buffer of the input interface 104 (210, FIG. 5). Input string data is
For example, a code such as ASCII is input, but when the code EOF is read, the processing unit 106 writes a NULL code in the input character string buffer and sets it as the final position.

Next, the processing unit 106 shapes the input character string (211). For example, when two or more characters belonging to the space character type such as a space character are consecutive, they are combined into a single space character. White space character Tabs, line breaks Etc. are included. In addition, the characters corresponding to the space from the beginning of the input character string buffer to characters other than the character corresponding to the first space appearing are deleted.

For example If you format the input string as follows, it becomes, as shown in Figure 6, The position of the code [NULL] indicates the final position of the buffer.

The dictionary lookup delimiter used in the dictionary lookup unit cutout process 201 is placed at a position of an apostrophe following an alphabetic character, a digit, an apostrophe, a hyphen and a period, and a space character. The processing unit 106 has a dictionary lookup head pointer, which is initially set at the head of the buffer.

Then, the dictionary retrieval unit 108 indexes the dictionary file 18 with the highest priority flag using the character string from the character pointed by the leading pointer to the character immediately before the next delimiter as the search key character string. The dictionary entry is compared with the search key character string, and if both match, the dictionary information is fetched (203). If the entire character string of the dictionary entry matches at least the part starting from the beginning of the search key character string, and the part immediately after that part is the dictionary lookup delimiter or the apostrophe or period, it is determined as a match. For example, as shown in Fig. 7, search key character strings When the leading pointer points to the leading character "g" of "," the dictionary entries "get" and "get upon" match this.

The retrieved dictionary information is stored in the searched dictionary information buffer 120 of the processing unit 106. Along with this reading, the start position and end position of the matched character string are stored. This specifies the character positions in the input buffer in order from the beginning. The searched dictionary information buffer 120 is provided with an active information storage area, which is information indicating whether or not the retrieved dictionary information is effective for subsequent processing. Leave it at "1".

Thereafter, the head pointer is updated each time the dictionary is looked up, and is set to the character immediately after the delimiter that appears next to the current head pointer when the character string is viewed from left to right. Thus, the dictionary search is sequentially performed. In the above example, first "I" for "I", then "w" for "will", then "g" for "get",
The first character of the dictionary lookup unit is designated in this order. When the leading pointer passes the NULL code, it is determined to be the final position (202). FIG. 9 shows an example of the dictionary information found in this way for the example of the English input character string.

Referring to FIGS. 8A to 8D, the conflict resolution rule processing unit 110
A description will be given of the contradiction resolution processing 204 of the highest priority flag performed by referring to the highest priority flag contradiction resolution rule file 36. 8th A
The flow of FIG. 8 and FIG. 8B is the processing when the positions of the words for which the highest priority flag is set overlap, and the flow of FIG. 8C and FIG.
The flow in the figure is a process of deleting an analysis unit, that is, an element by the highest priority flag, that is, a process of setting active information to "0". In these flow charts, the symbol "<
“=” Indicates substitution, “→” indicates reference, and “p → x” indicates the content of x held by the entry of the pointer p.

First, a word set whose top priority flag is "1" and whose position in the sentence overlaps is detected (steps 220 to 223).
Next, the highest priority flag elimination rule is applied to each of the detected pairs, and the valid one is selected (steps 224 to 235).

For example, in the above example, as shown in FIG. For the "get u" at the start position "8" and the end position "13"
“1” is set in the highest priority flag between p ”and“ up to ”at the start position“ 12 ”and the end position“ 16 ”, and the character positions thereof overlap. First, the rule (1) described above is applied, and it is determined whether or not they are verbs by referring to the part of speech of the save pointer psave and the part of speech of the pointer p (224).
In this example, the combination of "get up" is selected because it corresponds to a verb.

When the rule (1) is not satisfied, the rule (2) is applied (228), the entry of the save pointer psave is referred to, the length of the character string and the entry of the pointer p are referred to, and the character is referred to. Compare column length len. Further, when the rule (2) is not satisfied, the rule (3) is applied (229), the start position of the save pointer psave is referred to and its position starts and the start position of the pointer p are referred to, and the position start is started. Compare with.

When the conflict resolution rules (1) to (3) are applied in this order and any one of them is satisfied, the liveness information of the entry that is not satisfied, that is, the invalid entry is set to "0" (23
2), leave that of other, ie valid entries as "1" (231). The application of such a contradiction resolution rule is sequentially executed for each entry up to the final position while moving the pointer p in steps (234, 235), and the active information is set to "1" only for a valid entry. The state of the above example is shown in FIG. For example, the entry “up
The activity information of "to" was set to "0".

Next, both the liveness information and the highest priority flag are "1".
The combination and the position that are partially overlapped are detected (236 to 241), and their activity information is set to "0" (242,
249). The application of such a contradiction resolution rule is sequentially executed for each entry up to the final position while moving the pointer p in steps (243, 248), and the liveness information of the invalid entry is set to "0". This allows, for example, the entry “ge
For "t up", the activity information of "get" and "up" is "0"
(Fig. 10). The entry "white""white ho
For "use" and "house", since the highest priority flags are all "0" even if there are overlapping positions, their liveness information is maintained at "1".

When the processing is performed just before the final position [NULL], the contents of the input buffer of the input interface 104 and the searched dictionary information buffer 120 are output from the output interface 114 to the syntax analysis I unit 16. The contents of the searched dictionary information buffer 120 are output only for the entries for which "1" is displayed in the active information. For example, the contents of the input buffer are
6 may be written, and the contents of the searched dictionary information buffer 120 may be written to the syntactic analysis dictionary information file 118. At this time, since the activity information and the highest priority flag are also output, the syntactic analysis dictionary information file 118 has the same structure as the searched dictionary information buffer 120. The activity information and the highest priority flag may not be output.

EFFECT According to the present invention, during sentence morphological analysis, all words, collocations or idioms starting from the cut-out position of the dictionary lookup unit are extracted, but the collocations or collocations recognized as a group according to the highest priority flag included in the dictionary information For idioms,
The dictionary information obtained for each of the words that compose it is discarded. That is, in a sentence, the strength of the connection between words is determined, and it is presumed that the compound or the idiom with strong connection is the usage as the idiom in the sentence, and if not, the usage of the word alone is determined. Possibility is also considered in parallel. Thereby, the morphological analysis is efficiently performed.

[Brief description of drawings]

FIG. 1 is a functional block diagram showing a detailed configuration example of a morpheme analysis unit of the embodiment shown in FIG. 2, and FIG. 2 shows an embodiment in which the language analysis device according to the present invention is applied to an English-Japanese automatic translation device. FIG. 3 is a functional block diagram showing the overall configuration, FIG. 3 is an explanatory diagram showing a configuration example of a dictionary file with a top priority flag in the embodiment shown in FIG. 1, and FIG. 4 shows an example of morpheme analysis processing in the same embodiment. Flow chart, FIG. 5 is a flow chart showing an example of input processing in the morphological analysis processing, FIG. 6 is an explanatory diagram showing an example of shaping an input character string in the same embodiment, and FIG. 7 is a dictionary search in the same embodiment. FIG. 8A to FIG. 8D are flow charts showing an example of resolution processing of the contradiction of the highest priority flag in morphological analysis, and FIG. 9 is an example of the contents of the searched dictionary information buffer obtained by dictionary dictionary. Fig. 10 shows the highest priority flag Is an explanatory diagram showing an example of the contents of a retrieved dictionary information buffer result of performing conflict resolution process grayed. Explanation of code of main part 16 …… Morphological analysis section 18 …… Dictionary file with top priority flag 36 …… Top priority flag conflict resolution rule file 106 …… Processing section 108 …… Dictionary search section 110 …… Contradiction resolution rule processing section 112 ... Control unit

Claims (2)

[Claims] 1. A linguistic analysis apparatus comprising: dictionary means for storing dictionary data containing morpheme data for words, collocations and idioms; and analysis means for performing morpheme analysis on an input sentence by referring to the dictionary means. , The dictionary means includes a priority flag for establishing a compound word and a compound word as a compound word or a compound word, and the analysis means refers to the dictionary means for each word included in the input sentence, and 1 When multiple dictionary data are searched for in combination with other words for one word,
Judge whether the priority flag is set, when the priority flag is set, presume that it is a usage as a compound word or idiom, discard the dictionary data for each word constituting the compound word or idiom, A language analysis device characterized by performing morphological analysis as a compound word or a compound word. 2. The apparatus according to claim 1, wherein the analyzing means sets a priority flag for a plurality of combinations of a word searched by the dictionary means with another word, When a contradiction occurs, a linguistic analysis apparatus is characterized in that one of the plurality of combinations is prioritized according to a predetermined contradiction resolution rule to resolve the contradiction.