JPH11167574A - Natural language processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a natural language processor which can properly process even an input text that includes an unknown word part. SOLUTION: This processor adds the extension information including at least the word separation information for each character of an input text to form the extension characters, generates an extension character string related to the character string of the input text by means of those extension characters, decides the chaining probability for each extension character string based on the paths of all partial extension character strings covering the head through the end of the input text and also on the partial chaining probability of preliminarily prepared various partial extension character strings, and selects an extension character string that can secure the optimum chaining probability among those obtained chaining probabilities. If the partial extension character strings of the extension character string are not prepared beforehand, an extension character estimation part 5 estimates the partial chaining probability of the partial extension character strings based on the partial chaining probability of other partial extension character strings having the extension characters which are common to some extension characters of the relevant partial extension character string.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a natural language processing apparatus, and more particularly to an apparatus for processing digitized natural language text without using a dictionary (for example, morphological analysis) by using a stochastic method. Can be done.

[0002]

2. Description of the Related Art A large number of computerized natural language texts have been easily available due to an increase in opportunities for text creation by word processors and the spread of Internet-compatible devices. For various application systems that apply a large amount of natural language text, such as character recognition systems, machine translation systems, information retrieval systems, and information extraction systems, and apply various types of natural language processing, the morphological analysis process performs specialized processing for various applications. This is a very important process that is performed in common before performing a process, and determines a semantic unit in a sentence such as a word or phrase, that is, a morpheme.

[0003] If a morphological analysis positioned at the first stage of various application systems is erroneously performed, the error has an influence on processes such as recognition, translation, search, and extraction at a subsequent stage, and greatly affects the processing accuracy. In general, in the subsequent processing, it is assumed that morphological analysis is performed correctly, and it is very difficult to repair the error. Further, even if the restoration is possible, the restoration process becomes complicated, so that a large amount of natural language texts cannot be processed in an expected time.

Thus, in the morphological analysis processing,
High precision of word segmentation (morpheme segmentation) is required, and a processing speed of processing a large amount of natural language text at high speed is also required.

[0005] Words are separated by word delimiters (spaces), and it is only necessary to give a tag such as part of speech to a word. A method has been established in which a probability model of a tag sequence is estimated and error correction based on an example is added.

On the other hand, even in a language such as Japanese, for example, in which words are not separated, some examples have been proposed in which a method for English using a probability model is applied. As an example of morphological analysis using a stochastic model,
There is a method disclosed in the following document.

References: Mikio Yamamoto and Masakazu Masuyama, "Japanese Morphological Analysis Using Chain Probability of Extended Characters Including Part-of-Speech and Delimiter Information," Proc. Of the 3rd Annual Meeting of the Association for Language Processing, 1
This document describes that there are the following problems when applying a morphological analysis method using a probabilistic model to Japanese that does not clearly have word breaks.

(Problem A) In the case of English, even if there is an unknown word, the word division is not affected, but in the case of Japanese, the unknown word affects the word division. Become serious.

(Problem B) In the case of Japanese, since there is ambiguity of the delimiter, there is a problem in applying the English probability model as it is on the assumption that the number of word divisions is constant.

In order to solve the above problem, the above-mentioned document proposes a morphological analysis method based on characters. Japanese characters are generally used in about 3000 kinds, and the average word length is also about 2 characters, so based on the property that one Japanese character has information close to a word Proposed a chain probability model using extended characters in which information on morphological analysis was added to each Japanese character. According to this method, since it is based on characters, it is not necessary to use a word dictionary in which a plurality of character strings are registered as words. Without using a word dictionary, the concept of unknown words is eliminated and (Problem A) Will be resolved. In addition, the length of a character is always 1 and is constant in one sentence, the number of characters per sentence corresponding to the number of word divisions in English is also constant, and an English probability model can be applied. B) is solved.

According to the method disclosed in the above document, basically, when performing a morphological analysis based on a character, when a natural language text is given as an input sentence, a word constituting the input sentence is used. As a sequence, the most probable word sequence is output from all combinations of whether or not each character immediately follows a word boundary. As means for realizing this method, a chain probability (hereinafter also referred to as a partial chain probability) p (W, W) of the extended character ei defined by the expression (1) and the extended character defined by the expression (2)
A chain probability model of an extended character string using T) is used. Here, the extended character ei is different from ordinary characters such as "I" and "ha", and is obtained by adding extended information including at least word-separated (morpheme-separated) information to a character.

[0012]

(Equation 1) Here, ci is the character at the position i of the input character string (input text string), and di is after (or before) the character ci.
It is assumed that the information is delimiter information.

[0013]

(Equation 2) Here, n is the length of the input character string, and N is N-gra
N of m, that is, the length of the character set referred to for finding the optimal solution (the number of characters constituting the character set), ei is the morpheme sequence W
And the extended character determined from the information of the tag T.

An apparatus for implementing the morphological analysis method described in the above-mentioned document is as follows (for example, Japanese Patent Application No. Hei 9
-68300 and drawings).

That is, (a) a text is read as an input sentence, and for each character of an input character string of the input sentence,
An extended character string generation unit that forms an extended character by adding extended information including at least word delimiter information, and that generates all extended character strings related to the input character string as candidates using the formed extended character; A) a chain probability calculation unit that calculates the chain probabilities of all the generated extended character strings as candidates;
(C) Find the maximum value of the chain probability from the obtained chain probability candidates, select an extended character string that gives the maximum chain probability as the optimal extended character string, and select a word sequence corresponding to the optimal extended character string. Is provided as a configuration requirement.

The above constituent elements perform morphological analysis by performing the following operations.

That is, (S1) the extended character string generation unit
A text is read as an input sentence, and for each character of the input character string of the input sentence, extended information including at least word delimiter information is added to form an extended character, and the input character is formed using the formed extended character. Paths of all extended character strings from the beginning to the end of all input sentences related to the column are generated as candidates and stored in the score table.

(S2) Next, the chain probability calculating section calculates the partial chain probability corresponding to the partial extended character string having a fixed number of characters stored in the extended character table created by training (learning) in advance. Calculating a chain probability p (W, T) of the extended character string corresponding to the path of the extended character string,
Store it in the score table.

(S3) Thereafter, the optimum route searching unit
Reference the chain probability candidate of the extended character string in the score table, find the maximum value of the chain probability from the candidates, select the extended character string that gives the maximum chain probability as the optimal extended character string, and select this optimal extended character. An analysis result including an arrangement of word strings corresponding to the columns is output as a morphological analysis result.

[0020]

For example, a phenomenon in which an arbitrary character in the input text is replaced by another unexpected character due to the performance limit of an input device such as a character recognition device (hereinafter, referred to as garbled character). What happens is what happens normally in normal usage. Further, even when the input text is digitized, an unexpected word spelling (hereinafter, referred to as a miss type) due to an input error of the operator is often included in the input text in the process of digitizing the text. Occur.

Generally, a character string portion containing such an erroneous character mixed in an input text is treated as an unknown word. However, this kind of unknown word which should not exist as a natural language is a natural language. Should be distinguished from words that exist but are not known by the morphological analyzer.

However, in the conventional morphological analyzer, if a character string that was not originally an unknown word in the input text is to be treated as an unknown word due to garbled characters or typos, the unknown word is called an unknown word. Just treating it as a category, there is no concept of restoring the unknown word part to a correct word, and there is no means for restoring. In other words, in the conventional morphological analysis method and apparatus, the morphological analysis in the case where the quality of the input text is inferior to the predetermined value due to the performance limit of the input device or the deficiency of the input means, that is, the natural morphological analysis should originally exist Since morphological analysis of text that matches words without words is not assumed, if a character string that was originally not an unknown word in the input text was treated as an unknown word due to garbled characters or typos, There is a problem that a word part cannot be restored to a correct word, and a desired morphological analysis result cannot be obtained.

Therefore, it is possible to detect an unknown word portion due to garbled characters, typos, etc., and restore the portion to a correct character string. In other words, a natural language processing device capable of estimating a correct character string, There is a need for a natural language processing device that can execute predetermined natural language processing even if a correct character string cannot be estimated.

[0024]

According to a first aspect of the present invention, there is provided a natural language processing apparatus comprising: (1) adding extended information including at least word delimiter information to each character of a character string of a read input text to generate an extended character; An extended character string generating unit that forms an extended character string of all combinations of the character string of the input text using the extended character, and (2) a partial extended character string having a fixed number of characters and the partial extended character string An extended character storage unit that stores partial chain probability information for
(3) Based on the paths of all partial extended character strings from the beginning to the end of the input text and the partial chain probabilities stored in the extended character storage unit, all of the extended character string generation units For each of the extended character strings, a chain probability calculating unit for obtaining chain probability information, (4) a score storage unit for storing the obtained chain probability information, and (5) an optimal chain from the obtained chain probability information. An optimal path search unit that selects an extended character string that gives a probability, and outputs an analysis result including a sequence of word strings corresponding to the extended character string as a morphological analysis result; and (6) an optimal character string generated by the extended character string generation unit. When the partial extended character string of the extended character string does not exist in the extended character storage unit, another extended character string having a common extended character with some extended characters of the partial extended character string stored in the extended character storage unit Partial expansion From the partial chain probabilities information string, characterized by comprising a extended character estimation unit for estimating the partial chain probability information of the partial extension string.

According to a second aspect of the present invention, there is provided a natural language processing apparatus comprising:
(1) a partial character string storage unit that stores a partial character string consisting of a fixed number of characters; (2) a non-target character pattern storage unit that stores a pattern of a non-target character that is registered in advance as constituting an unknown word; (3) An unknown word portion in the read input text is detected based on the storage content of the non-target character pattern storage unit, and the correctness of the unknown word portion is determined based on the storage content of the partial character string storage unit. And an unknown word detection unit for estimating a character string considered to be a character string.

[0026]

DESCRIPTION OF THE PREFERRED EMBODIMENTS (A) First Embodiment Hereinafter, a first embodiment in which a natural language processing device according to the present invention is applied to a morphological analyzer will be described in detail with reference to the drawings.

(A-1) Configuration of the First Embodiment FIG. 1 is a functional block diagram showing the configuration of the morphological analyzer of the first embodiment. That is, the morphological analysis device of the first embodiment is actually realized on an information processing device such as a workstation or a personal computer having an input / output device, a processing device, a storage device (or a communication device), and the like. But,
Functionally, it has the configuration shown in FIG.

In FIG. 1, the morphological analyzer of the first embodiment includes an input device 1, an extended character table 2, a score table 3, an extended character string generator 4, an extended character estimator 5, and a chain probability calculator 6. , An optimum route search unit 7 and an output device 8.

The input device 1 includes a keyboard, a mouse, and the like for inputting a natural language text to the morphological analyzer.
Any means such as an OCR (optical character recognition device) and a voice recognition device may be used, or a means for receiving a communication signal from the outside via a communication medium such as a network may be used.

The extended character table 2 stores extended character strings and their chain probabilities (partial chain probabilities), and is a storage device that has been created by learning in advance with a training text (corpus).

The score table 3 includes an extended character string obtained based on the path of all extended character strings (N-gram) from the beginning to the end of the input text and the partial chain probability stored in the extended character table 2. Is a storage device for storing a chain probability p (W, T) of an extended character string corresponding to the path of (i).

The extended character string generator 4 refers to the extended character table 2 created in advance by training, generates an extended character string of the input text, and stores the path of the extended character string. .

The extended character estimating unit 5 includes an extended character string generating unit 4
Creates an extended string path from the input text,
When the input text includes a character that is not stored in the extended character table 2, the partial chain probability of the partially extended character string including the unknown character is estimated.

The chain probability calculating section 6 calculates the extended character table 2
Is calculated based on the partial chain probability stored in the score table 3 with respect to the path of the extended character string stored in the score table 3.

The optimum route search unit 7 extracts an extended character string that satisfies an optimal condition (for example, gives the maximum value of the chain probability) from the chain probabilities calculated by the chain probability calculating unit 6. Is to be selected as

The output device 8 outputs the result of the morphological analysis obtained by the morphological analyzer to various external display means and communication means as in the case of a normal information processing apparatus.

FIG. 2 is an explanatory diagram showing a configuration example of the extended character table 2. In FIG. 2, a character ci corresponding to the extended character ei defined by the above-described equation (1) and extended information (here, delimiter information) di are sequentially described as a pair in the extended character table 2. . Specifically, each extended character ei-N + forming one record of the extended character table 2
1, ei-N + 2, ..., ei are the corresponding characters ci-N + 1, ci-N +
, Ci, and extended information di-N + 1, di-N + 2,..., Di are described in pairs. In the right column of the record, the partial chain probability p (ei
| Ei-1, ei-2,..., Ei-N + 1). Hereinafter, the notation of the extended character ei is <ci, di>. The delimiter information, which is the extended information, takes two values depending on whether the morpheme is divided immediately after the character position i (di = 1) or not (di = 0).

FIG. 3 is a diagram showing a specific example of the extended character table 2. In this example, when the number N of character sets is 3, ie, when N = 3, the partial chain probability p (ei is corresponding to the partial extended character string ei-2, ei-1, ei which is N-gram.
| Ei-1, ei-2) is stored in each record.

For example, record L309 is <East, 0
>, <Kyo, 1>, <Tokyo, 0>, and the partial chain probability corresponding to the partial extended character string is 0.12.
Is shown. Further, for example, the record L3
The notation <#, 1> that appears in 01 or the like is used when calculating the partial chain probability at the beginning or end of the input text.
This is a special (dummy) extended character inserted for convenience so as to have the same effect as other partial extended character strings.

FIG. 4 is an explanatory diagram showing a configuration example of the score table 3. In FIG. 4, each extended character e- (N-1) +1,
e- (N-1) +2, ..., e1, e2 ..., en, ..., en + (N-1) are the corresponding characters c- (N-1) +1, c- (N-1), respectively. +2, ..., c
, Cn, ..., cn + (N-1) and extended information (here, delimiter information) d- (N-1) +1, d- (N-1) +2, ..., d1, d2
, Dn,..., Dn + (N-1) are sequentially recorded in pairs for all combinations of the extended character strings (each combination is also called a path). The column on the right side of the record of the score table 3 stores the chain probability p (W, T) for the path of the extended character string stored in the record.

FIG. 5 is a diagram showing a specific example of the score table 3 after the storing process is completed. In this example, all of the corresponding extended character strings when the text “Nanjing City” is input (because “Nanjing City” has three characters, delimiter and non-delimiter are used as the cube of binary extended information of 8). Type) and the chain probability are stored.

(A-2) Operation of First Embodiment The operation (morphological analysis method) of the morphological analyzer of the first embodiment will be described below with reference to the drawings.

First, the overall operation of the morphological analyzer according to the first embodiment will be described with reference to the flowchart shown in FIG.

In the morphological analyzer according to the first embodiment, the N-gram corresponding to the partial character string of the input text
When the record of the partial extended character string does not exist in the extended character table 2, that is, when the partial character string is an unknown word, the partial chain probability of the partial character string cannot be obtained. The phenomenon of a remarkable decrease is solved by the following operation shown in FIG.

(Step S601) The input text is read into the morphological analyzer via the input device 1.

(Step S602) An extended character is generated from each character of the read text, and the path of the extended character string from the beginning to the end of the input text is determined and stored in the score table 3. In step S602, when a character string having a fixed number of characters (N-gram character string) that does not exist in the extended character table 2 exists in the input text, the corresponding partially expanded character string, that is, the part corresponding to the unknown word Estimate extended string records.

(Step S603) The chain probabilities for the paths of all the generated extended character strings are obtained. The chain probability of the extended character string is obtained by referring to the extended character table 2 for the partial chain probability corresponding to each of the partial extended character strings constituting the extended character string, and is obtained as the product of the partial chain probabilities ( (Refer to the equation (2) described above.) The calculated chain probability is stored in the record of the path of the corresponding extended character string in the score table 3.

(Step S604) Score Table 3
And selects an extended character string that satisfies an optimum condition (for example, gives the maximum value of the chain probability) from the obtained chain probabilities as the optimum expanded character string.

(Step S605) The output unit 8 outputs a morphological analysis result including a word sequence determined by the optimal expanded character string.

FIG. 7 is a flowchart for explaining in detail the extended character string generation operation in step S602 described above.

A text is input to the extended character string generation unit 4 via the input device 1, and an extended character is formed by adding extended information (for example, delimiter information) to each character of the input text (step S701). ). For example, for a three character input text "Nanjing City", <South, 0
>, <South, 1>, <Kyo, 0>, <Kyo, 1>, <City, 0
>, <City, 1> are generated.

Next, one path of an extended character string corresponding to the input text is created from the generated extended characters and stored in the score table 3 (step S702). At this time, the storage of the chain probability for the entire route (see FIG. 8) is not executed. Since step S702 is repeated as described later, when the series of processes illustrated in FIG. 7 is completed, all the paths determined by the combination of the extended character strings are stored in the score table 3. In the first embodiment, binary delimiter information is added as extended information.
From the input text of M characters, a path of 2 M kinds is created. Considering the special extended characters at the beginning and end of the text, one extended character string is M + 2 * (N-1) characters.

For example, if N-gram is 3-gram, as shown in FIG. 5, <#, 1>-<#, 1> for the input text of three characters "Nanjing City". − <
South, 0>-<Kyo, 0>-<City, 0>-<#, 1>-<
#, 1>, <#, 1>-<#, 1>-<South, 0>-<
Kyo, 0>-<city, 1>-<#, 1>-<#, 1>, ...,
<#, 1>-<#, 1>-<South, 1>-<Kyoto, 1>-<
Eight types of extended character strings consisting of seven extended characters of city, 1>-<#, 1>-<#, 1> are generated.

When the storage of a certain path in the score table 3 is completed, the extended character estimating unit 5 initializes a counter representing the i-th character position (i = 1) (step S703).

Subsequently, the extended character estimating unit 5 selects N stored character string portions at positions determined by the counter i from the extended character strings (paths) created by the extended character string generating unit 4 in the immediately preceding step S702. (N-gram extended character string) e-
(N-1) + i, ..., ei are extracted (step S704). In this embodiment, since N-gram in the case of N = 3 is assumed, the extracted N-gram extended character string is e.
i-2, ei-1, and ei. For example, <#, 1>-<#,
1>-<South, 0>-<Kyo, 0>-<City, 0>-<#, 1
>-<#, 1>, i =
In the case of 1, e-1, e0, e1, that is, <#, 1
>, <#, 1>, <south, 0> are extracted as N-gram extended character strings.

Next, it is checked whether or not the extracted pattern of the N-gram extended character string exists in the extended character table 2 (step S705).

If it does not exist, the chain probability of the N-gram extended character string is estimated (step S706). To estimate the chain probability, for example, an extended character (excluding special extended characters) that does not exist in the extended character table 2 of the N-gram extended character string is regarded as a general character to be matched with all characters, and the extended character table 2 The average value of the chain probabilities of collatable records is used as the chain probability of the N-gram extended character string.

Next, the extended character estimation unit 5 additionally stores the N-gram extended character string having the estimated partial chain probability in the extended character table 2 (step S707).

If the pattern of the extracted N-gram extended character string exists in the extended character table 2 (step S7).
05, or an N-gram extended character string is additionally stored in the extended character table 2 (step S70).
7) Then, after incrementing the counter i representing the character position by one (step S708), the value of the counter i is increased to a value M + N-1 (M is the number of characters of the input text, N is N-g).
ending the confirmation of the presence or absence of the extended character table 2 for all N-gram extended character string portions of the extended character string (path) currently being processed by comparing the number of characters in the extended character table 2 It is determined whether or not the process has been performed (step S709).

If the confirmation of the existence of the extended character table 2 has not been completed for all N-gram extended character string portions of the currently processed extended character string (path), the above-described step S704 is executed. Return.

On the other hand, when all the partial extended character strings forming the path of one extended character string have been processed (step S
(Yes at 709), the extended character estimating unit 5 completes the operation, and then the extended character string generating unit 4 checks whether an unprocessed extended character string path remains (step S71).
0) If there is still an unprocessed route, the above steps S702 to S710 are repeated. When all the paths have been processed, the extended character string generation unit 4 ends the operation.

The operations of steps S704 to S707 will be specifically described with reference to actual examples. Here, N-gr
The initial state of the extended character table 2 storing the am extended character string and its chain probability is the record L301 to L301 in FIG.
It is assumed that L334 is stored. It is also assumed that the input text is “Nanjing City”.

Here, one of the extended character strings (paths) for the input text "Nanjing City" is <#, 1>-<
#, 1>-<South, 0>-<Kyo, 0>-<City, 0>-<
For an extended character string of #, 1>-<#, 1>, i = 1
In the case of <#, 1>, <#, 1>, <South, 0> are extracted as the N-gram extended character strings e-1, e0, e1 (step S704). Since this pattern <#, 1>, <#, 1>, <South, 0> does not exist in the records L301 to L334 of the extended character table 2 shown in FIG. 3 (step S705), the N-gram concerned Extended character not present in extended character table 2 of extended character string <south, 0
> Is regarded as a general character (the N-gram extended character string having the first two extended character portions of this pattern exists in the extended character table 2), and when a record to be collated with the extended character table 2 is searched, the <#,1>,<
#, 1>, <East, 0> and <#, 1 in record L321
>, <#, 1>, <north, 0>. As a result, the average value (0.06 + 0.06) /2=0.06 of the chain probabilities of the records L301 and L321 is equal to the N-gr
It is estimated as the chain probability of the am extended character string <#, 1>, <#, 1>, <South, 0> (step S706). Thereafter, the N-gram extended character string <#, 1>, <
#, 1>, <South, 0> and their chain probability 0.06 are additionally stored in the extended character table 2 (step S70).
7). By this operation, the extended character table 2 is displayed in FIG.
Record L351 is added.

Thereafter, the values of the counter i are changed, and steps S704 to S708 are repeatedly executed. In the same manner as described above, records L352 to L364 are newly added to the extended character table 2.

FIG. 8 is a flowchart for explaining in detail the operation of calculating the chain probability of the extended character string (path) in step S603 described above.

The chain probability calculating section 6 first takes out one extended character string record stored in the score table 3 (step S801). Next, a counter i representing a character position is initialized (i = 1) (step S802).

Then, a partial extended character string of N characters from e- (N-1) + i to ei determined by the value of the counter i, that is, an N-gram extended character string is extracted from the record,
The chain probability p (ei) of the record to be collated with the N-gram extended character string in the extended character table 2 is extracted (step S803).

If the character position counter i is 1 (it is an N-gram extended character string at the beginning of the text) (Yes in step S804), the chain probability p (W, T) indicates the partial chain probability p
(Ei) is stored (step S805). Also, N-
If the gram extended character string is not the head of the input text (a negative result in step S804), the partial chain probability p (W, T) is added to the chain probability p (W, T) of the extended character string record.
(Ei) to obtain a new chain probability p (W, T) (step S806).

Next, after incrementing the character position counter i by 1 (step S807), the value of the counter i is increased to a value M + N-1 (M is the number of characters of the input text, N is N-gr).
am, the number of partial chain probabilities p (ei) for all N-gram extended character string portions of the currently processed extended character string (path), and Then, it is determined whether or not the update processing of the chain probability p (W, T) reflecting the reflection has been completed (step S808).

If not completed, step 8 described above
Return to 08. On the other hand, if the processing is completed, that is, if the partial chain probability of the partial extended character string that constitutes the entire path of one extended character string is processed, the chain probability p (W, T) of the extended character string is calculated. It is stored in the corresponding position of the score table 3 (step S809).

Calculation of chain probability p (W, T) (step S
Steps 801 to S809) are performed for all the records stored in the score table 3, and if all the records have been processed (step S810), the linkage probability calculating unit 6
Ends the operation.

Hereinafter, the operation of calculating the chain probability will be described with reference to a specific example. It is assumed that the input text is “Nanjing City”. Also,
It is assumed that the extended character table 2 stores the partially extended characters and the partial chain probabilities in the state of FIG. It is assumed that the score table 3 stores the path of the extended character string corresponding to "Nanjing City" as shown in FIG.
However, the column of the chain probability of each record of the extended character string shown in FIG. 5 is blank in the initial state.

First, one record is extracted from the score table 3. For example, the record L501 in FIG. 5 is extracted. Record L501 includes <#, 1>, <#, 1>, <
South, 0>, <Kyo, 0>, <City, 0>, <#, 1>, <
#, 1>, first, N-gr when i = 1
The extended character table 2 is searched for the chain probability of the am extended character string <#, 1>, <#, 1>, <South, 0>. The record L351 in FIG. 3 corresponds to this, and 0.06 is obtained as the partial chain probability p (e1) (step S803). Now i = 1
Therefore, p (ei) is stored in the chain probability p (W, T) of the extended character string, and p (W, T) = 0.06.

Next, i is increased by 1 to make i = 2 (step S807). i (= 2) <M + N-1 (=
5) (Step S808), the Step S80
3, the next N-gram extended character string <#, 1>, <
Extended character table 2 for linkage probabilities of south, 0>, <Kyo, 0>
Search from This corresponds to the record L353 in FIG. 3, and a partial chain probability p (e2) of 0.01 is obtained (step S).
803). Now, since i = 2, the chain probability p (W, T) of the extended character string is the original p (W, T) (= 0.0
6) multiplied by p (e2) (= 0.01). That is, the new chain probability is p (W, T) = p (e1)
× p (e2). Similar processing is performed when i> M + N-1.
This is repeated until (= 5).

As a result, finally, <#, 1>, <
#, 1>, <South, 0> and <#, 1>, <South, 0>, <
K, 0>, <South, 0>, <Kyo, 0>, <City, 0>,
<Kyo, 0>, <City, 0>, <#, 1>, and <City, 0>,
The value obtained by multiplying the five partial chain probabilities <#, 1> and <#, 1> is the chain probability p (W, T) of the extended character string (record L501). This chain probability is calculated using score table 3
Is stored in the column of the linkage probability of the record L501 (step S809).

The above operation is performed for all records in the score table (step S810).

FIG. 9 is a flowchart for explaining in detail the operation of selecting the optimum extended character string in step S604 described above.

First, the optimum route search unit 7 determines a condition for selecting an extended character string (step S901). As the selection condition, it is assumed that an arbitrary selection condition can be set, such as, for example, selecting a record of an extended character string having the largest linkage probability in the score table 3. Thereafter, as a condition for selecting an extended character, a record having the maximum chain probability is selected for convenience.

Next, the record having the maximum linkage probability is extracted with reference to the score table 3 (step S90).
2). For example, in the example of the score table shown in FIG. 5, the maximum chain probability (= 0.459 × 10 EXP-3 (EXP
-3 means −3 power)) record L504 <
#, 1>, <#, 1>, <South, 0>, <Kyoto, 1>, <
City, 1>, <#, 1>, <#, 1> are retrieved.

Next, the counter i indicating the character position is initialized (step S903), and the extended character ei = <ci, di
The character ci of> is output (step S904). Here, if the extended information (separation information) di of the extended character is 1 (step S905), a word delimiter (for example, “/”) is output (step S906), and the extended information (separation information) di If is 0, immediately step S90
Go to 7.

Next, after incrementing the counter i by 1 (step S907), the value of the counter i is changed to the value M + N-
In step S9, it is confirmed that the output process for all extended characters has not been completed by comparing with step S1.
Returning to step S908, the process proceeds to the output process for the next extended character.

When output processing for all extended characters of the extended character string is completed by such repetitive processing,
A text delimiter (for example, a line feed code) is output (step S909). As a result, for example, in the record L504 of the extended character string, “# / # / Nanjing / city / # /
# / ”Is output, and“ Nanjing ”and“ city ”are extracted as morphemes.

Finally, it is checked whether or not an extended character string (path; record) meeting the selection condition in step S901 remains (step S910), and if all records meeting the selection condition have been processed. Then, a series of operations for selecting the optimum extended character string ends.

(A-3) Effects of the First Embodiment According to the morphological analyzer of the first embodiment described above, the following effects can be obtained.

Even if an unknown N-gram character string that does not exist in the extended character table exists in the input text, an extended character estimating unit is provided to estimate the unknown partial extended character string and its chain probability from the extended character table. Therefore, a character string conventionally treated as an unknown word can be estimated without deteriorating the accuracy of morphological analysis.

Even if an unknown N-gram character string that does not exist in the extended character table exists in the input text, an extended character estimating unit is provided, and the partial extended character string and the chain probability of the estimated extended character string are calculated. Since it is stored in the extended character table, it is not necessary to estimate extended characters from the next morphological analysis, so that efficient morphological analysis can be performed.

(A-4) Modified Embodiment of First Embodiment In the first embodiment, the condition for selecting an extended character string to be selected from the score table 3 is an extended character string having the maximum linkage probability. If this selection condition is an extended character string having a chain probability equal to or greater than an arbitrary threshold value, morphological analysis results of a plurality of candidates can be output.

In the first embodiment, when the pattern of the N-gram extended character string extracted by the extended character estimating unit 5 does not exist in the extended character table 2, one of the extended character (Excluding special extended characters)
Is regarded as a general-purpose character to be collated with all characters, a record to be collated is extracted from the extended character table 2, and the average (arithmetic average) of the chain probabilities is used as the chain probability of the N-gram extended character string. However, a geometric mean may be used.

Further, when N of the N-gram extended character string is large, one extended character (excluding special extended characters)
A weighted average of the average value obtained assuming that is a general character to be compared with all characters, and the average value obtained by considering two extended characters (excluding special extended characters) as general characters to be compared with all characters Processing and the like may be further performed.

Furthermore, when the extracted pattern of the N-gram extended character string does not exist in the extended character table 2, another chain probability is corrected according to the estimated chain probability for the N-gram extended character string. You may do it. For example, when an N-gram extended character string is stored in the extended character table 2, the N-gram extended character string is generally
The sum of the chain probabilities of all N-gram extended character strings for which one -1 extended character is the same is set to 1. According to the chain probability estimated for the N-gram extended character string, this condition is The chain probability of another N-gram extended character string may be modified so as to be satisfied.

By the way, in the extended character table 2, Ng
Not only the information of the ram extension character string but also (NX) -gr
The information of the am extended character string and the X-gram extended character string is also stored, and N extracted from the extended character string (a certain path) is stored.
-Gram extended character string pattern is extended character table 2
Does not exist, the chain probability of the N-gram extended character string is determined by dividing the N-gram extended character string (N-gram extended character string).
A method has also been proposed in which an X-gram extended character string and an X-gram extended character string are obtained from the chain probability.

In the above embodiment, for example, when the number of records to be subjected to the averaging process is small, the N-
The chain probability of the gram extended character string is calculated by dividing the N-gram extended character string by (NX) -gram extended character string and X-
The method may be switched to a method obtained from the chain probability of the gram extended character string.

Note that the chain probability of the N-gram extended character string is determined by dividing the N-gram extended character string by (NX) -g
The method of obtaining from the chain probability of the gram extended character string and the X-gram extended character string actually requires a huge memory capacity and cannot be processed only by the framework of the N-gram extended character string. It is more difficult to apply to an actual product than the chain probability estimation method of the embodiment.

In the first embodiment, the extended character is composed of the character (character type) and the delimiter information. However, the extended character including the part of speech information (which may include the inflected form) is further described. It may be. In this case, the chain probability estimating process (averaging process) performed by the extended character estimating unit 5 is performed for each record group having the same part of speech information, and addition to the extended character table 2 is performed only when the part of speech information is different. Will be added. Note that addition to the extended character table 2 may not be performed on an N-gram extended character string of input text that does not exist in the extended character table 2 and is related to part of speech information with a small number of records to be compared. .

(B) Second Embodiment Hereinafter, a second embodiment in which the natural language processing apparatus according to the present invention is applied to a morphological analysis apparatus will be described in detail with reference to the drawings.

(B-1) Configuration of the Second Embodiment FIG. 10 is a functional block diagram showing the configuration of the morphological analyzer of the second embodiment. The same and corresponding parts are denoted by the same reference numerals.

In FIG. 10, a morphological analyzer according to the second embodiment has the same input device 1 as that of the first embodiment.
Extended character table 2 (see FIG. 2), score table 3
(See FIG. 4), in addition to the extended character string generation unit 4, the extended character estimation unit 5, the chain probability calculation unit 6, the optimal route search unit 7, and the output device 8, an unknown word detection unit 9, a non-target character pattern It has a memory 10, an input controller 11, an input buffer memory 12, an unknown word buffer memory 13, and an output synthesizer 14.

The input device 1, the extended character table 2, the score table 3, the extended character string generator 4, the extended character estimator 5, the chain probability calculator 6, the optimum route search unit 7, and the output similar to those in the first embodiment. The function of the device 8 is the same as in the first embodiment, and a description thereof will be omitted.

In the second embodiment, the newly provided unknown word detecting unit 9, non-target character pattern memory 10, input control unit 11, input buffer memory 12, unknown word buffer memory 13, and output synthesizing unit 14 It is provided for detecting an unknown word character string existing in a text, estimating the unknown word character string, and reflecting it in a morphological analysis result.

The unknown word detecting section 9 stores the unknown word portion in the input text from the input device 1 into the non-target character pattern memory 1.
The detection is performed based on the information stored in 0, and a character string that is considered to be correct for the detected unknown word character string is estimated by referring to the stored contents of the extended character table 2.

The non-target character pattern memory 10 stores a pattern of characters (non-target characters) that the unknown word detection unit 9 uses when detecting unknown words and cannot be subjected to morphological analysis by the morphological analyzer. Is what you are doing.

The input control section 11 controls an input text that is estimated to be correct when an unknown word is detected in the input text by the unknown word detection section 9.

The input buffer memory 12 temporarily stores one or more input texts newly created (estimated) by the unknown word detection unit 9 and the input control unit 11.

Under the control of the input control unit 11, the unknown word buffer memory 13 temporarily saves the character string of the unknown word portion detected by the unknown word detection unit 9.

The output synthesizing unit 14 calculates the morphological analysis results of the plurality of input texts controlled by the input control unit 11 from the optimum route searching unit 7 and the character string of the unknown word part saved in the unknown word buffer memory 13. And performs processing for obtaining a desired morphological analysis result.

FIG. 11 is an explanatory diagram showing a configuration example of the non-target character pattern table 10 in the second embodiment.

[0107] The non-target character is a character that is considered not to be included in the input text to the morphological analysis device. It is expensive. A set of non-target characters is set and stored in the non-target character pattern memory 10 in advance.

In FIG. 11, for example, record L11
In 01, a set of non-target characters that are unlikely to occur in ordinary sentences such as “∬ ‰ ♪ ¶ ‡” is registered.
Each character of “∬ ‰ ♪ ¶ ‡” appearing in the input text is
It is indicated that it is a non-target character. Also, as shown in record L1102, as a non-target character pattern,
[Starting character code-
Character code (text) in the expression [End character code]
Can be specified in the range. That is, if there is a character having a code in this code range in the input text, this indicates that the character is a non-target character. Most of the current sentences rarely contain second-level kanji, and even if they are included, the second-level kanji is limited to some extent, and many of the second-level kanji are not targeted. It is practical to register as characters.

FIG. 12 is an explanatory diagram showing a storage example of the input buffer memory 12 in the second embodiment.

FIG. 12A shows an unknown word in the input text detected by the unknown word detection unit 9 and the input text (here, two types) after the unknown word candidate is estimated is stored. It shows the state where it is. Record L1
The “answer” and “analysis” parts, which are the differences between “the advantage in this morphological analysis” 201 and “the advantage in this morphological analysis” of the record L1202, are the unknown word detection unit 9
Unknown word part estimated by

FIG. 12B shows a case where there is an unknown word part which could not be estimated even by the unknown word detecting unit 9.
In this state, the unknown word portion is marked and stored by the unknown word detection unit 9. When the actual input text is “this form ♪ ‰ ¶∬ is advantageous”, the character length (= 4) of the unknown word “♪ ‰ ¶∬” detected by the unknown word detection unit 9 is
Three characters (N
= 3 (see FIG. 15 to be described later), the unknown word portion cannot be estimated, so the parentheses defining the region of the unknown word portion.
Marked and stored.

The input buffer memory 12 stores, for example, a FIFO (First In First Ou).
t) format buffer, that is, a first-in first-out format buffer. For example, when the record L1201 is taken out and processed, the record L1202 is shifted to the position of the record L1201 and the previous position of the record L1202 is empty. So that the record is shifted one after another to the area of the upper record.

FIG. 13 is an explanatory diagram showing a storage example of the unknown word buffer memory 13 in the second embodiment.

In the unknown word buffer memory 13, FIG.
When there is an unknown word part that cannot be estimated even by the unknown word detection unit 9 as described in (B), the unknown word part is extracted and stored by the input control unit 11. . If the actual input text is “this form ♪ ‰ ¶∬ is advantageous”, the unknown word part “♪ ‰ ¶∬” is stored in the unknown word buffer memory 13.
Is stored in

(B-2) Operation of the Second Embodiment The operation (morphological analysis method) of the morphological analyzer of the second embodiment will be described below with reference to the drawings.

First, the overall operation of the morphological analyzer according to the second embodiment will be described with reference to the flowchart shown in FIG. In FIG. 14, the same as FIG.
Corresponding steps are denoted by the same reference numerals.

In the second embodiment, when an unknown word as a character string that cannot exist in a normal sentence of a natural language exists in a partial character string of the input text, the unknown word part is detected. The restoration is performed as much as possible by the following operation. In addition, each step of (Step S601) to (Step S605) is the same operation as in the first embodiment.

(Step S601) The input text is read into the morphological analyzer via the input device 1.

(Step S1401) An unknown word portion of the input text is detected, a text in which the unknown word portion is restored as much as possible (hereinafter referred to as an estimated text) is generated and stored in the input buffer memory 12. When the unknown word portion cannot be estimated, the text in which the unknown word portion is marked (hereinafter, referred to as a mark text) is stored in the input buffer memory 12. If the unknown word portion of the input text cannot be detected, the restoration and the marking process are not executed. Also, the next step S
The processing of 1402 is also omitted.

(Step S1402) Step S14
01 controls the input buffer memory 12 that stores the estimated text or the mark text, and passes the text to the following steps. The unknown word portion that cannot be estimated is stored in the unknown word buffer memory 13.

(Step S602) An extended character is generated from each character of the text read from the input buffer memory 12, and the path of the extended character string from the beginning to the end of the input text is obtained and stored in the score table 3. In this step, when a character string having a fixed number of characters (N-gram character string) that does not exist in the extended character table 2 exists in the input text, the corresponding partial expansion character string, that is, the partial expansion corresponding to the unknown word Guess the string record.

(Step S603) The chain probabilities of all the generated extended character strings with respect to the path are obtained. The chain probability of the extended character string is obtained by referring to the extended character table 2 for the partial chain probability corresponding to each of the partial extended character strings constituting the extended character string, and is obtained as the product of the partial chain probabilities. The calculated chain probability is stored in the record of the path of the corresponding extended character string in the score table 3.

(Step S604) Score table 3
And selects an extended character string that satisfies an optimum condition (for example, gives the maximum value of the chain probability) from the obtained chain probabilities as the optimum expanded character string.

(Step S1403) Input control unit 11
The morphological analysis results of a plurality of texts controlled by the above are combined as an output. If there is no unknown word (non-target character) portion in the input text, this step is a process of transferring the morphological analysis result from the optimal route search unit 7 to the output device 8 as it is.

(Step S605) The morphological analysis result including the word sequence is output via the output device 8.

FIG. 15 is a flowchart illustrating in detail the operation of detecting an unknown word (non-target character) in step S1401.

The unknown word detecting section 9 refers to the non-target character pattern memory 10 to detect all non-target character continuous portions in the input text and the number L of non-target characters in each non-target character continuous portion ( Step S1501). In addition,
If no non-target character can be detected by this process, a series of processes is terminated although the illustration of the branch line is omitted.

Next, the degree (number of characters) N of the N-gram extended character string stored in the extended character table 2 is compared with the number L of non-target characters in one non-target character continuous portion (step S1502). ). This comparison means determination as to whether or not a character string considered to be correct can be estimated using the storage contents of the extended character table 2 for the non-target character continuous portion to be processed at present.

If L <N (Yes in step S1502), the unknown word detection unit 9 includes a character other than the non-target character before or after the non-target character continuation part of the current processing target, and The target character portion may be any character, and it is searched whether a record matching the N-gram character string (extended information can be any) exists in the extended character table 2 (step S1503). It is determined whether or not there is a character string pattern that can be replaced with the non-target character string portion determined by (step S150)
4).

If there is a character string pattern that can be replaced with the non-target character string part (step S150)
(Yes at 4), instead of the non-target character string portion, generate an estimated text to which the character string pattern is applied and store it in the input buffer memory 12 (step S150).
5). Here, in the search in step S1503, a case where a plurality of character string patterns that can be replaced with a non-target character string portion may be obtained, in which case,
A plurality of estimated texts having different character string patterns are generated and stored in the input buffer memory 12.

Note that whether or not the estimated text can be stored in the input buffer memory 12 may be determined by using the chain probability. This will be described in a process using a specific example described later.

On the other hand, when the number L of non-target characters is equal to or greater than the order N of the N-gram character string (negative result in step S1502), or when there is no character string pattern that can be replaced with the non-target character string portion (step S1504) In the negative result, it is impossible to estimate the unknown word (non-target character string) to the original character string. Therefore, an unknown word marker is added to the non-target character string of the input text to generate a mark text. Are stored in the input buffer memory 12 (step S150).
6).

Thereafter, it is determined whether or not all the continuous portions of the non-target characters in the input text have been processed (step S1507). S1502-S150
7 is repeated for a continuous portion of another non-target character, and when all the continuous portions of the non-target character in the input text have been processed, the unknown word detection unit 9 ends a series of operations.

For example, in response to an input text “What is the advantage of this morpheme ‰ ¶”, the unknown word detection unit 9
Referring to the non-target character pattern memory 10 shown in FIG. 11, it is found from the record L1101 that “‰” and “¶” are non-target characters, and “‰ ¶” of the input text is an unknown word (non-target character Part), and the length L is detected to be 2 (step S1501).

Here, the degree N of the N-gram character string is set to 3
Then, L <N (step S1502), and the extended character table 2 is searched (step S1503).

Now, this search is performed with the non-target character continuous portion “{
N-g of a character "prime" other than the non-target character preceding "?" And two general-purpose characters for the non-target character continuous part "‰ ¶"
It is assumed that the processing is performed with a ram character string (3-gram character string). The extended character table 2 includes, for example, “<prime, 1
><,0><answer,1>"and"<prime,1><solution,0>
It is assumed that an N-gram character string record “<analysis, 1>” exists.

In this case, by searching the extended character table 2, it is confirmed that “answer” and “analysis” exist as character string patterns to be replaced with the non-target character continuation part “‰ ¶” (step S1504). , The estimated texts "What is the advantage of this morphological answer" and "What is the advantage of this morphological analysis" are generated and these are input buffer memory 1
2 (step S1505). The input buffer memory 12 in this storage state is the same as that shown in FIG.
It becomes as shown in.

It should be noted that “<prime, 1><
, 0><answer,1> ”and“ <prime, 1><solution,0><
It is also possible to take out the chain probability of the N-gram character string of “analysis, 1>”, compare the chain probability with a threshold, and generate an estimated text only when the threshold is exceeded.

An N-gram character string consisting of a character "prime" other than the non-target character preceding the non-target character continuous portion "$" and two general-purpose characters for the non-target character continuous portion "$"(<Prime,1><
, 0><answer,1> ”, a search character string“ <, 0><answer,1> ”that can be replaced with a non-target character continuous portion“ ‰ ¶ ” And an N-gram character string (3-g) of a character other than the non-target character subsequent to the non-target character continuation part “‰ ¶” in the input text.
The extended character table 2 is compared again with the (gram character string), and the N-gram character string (3-gram character string) can be searched. Therefore, the non-target character continuous portion “‰ ¶” is estimated as “the answer”. You may do it. Also in this case, for example, N-gr including the character “prime” other than the non-target character on the front side
am and the chain probability of the N-gram character string found in the search for the character string, and N
After multiplying the chain probability of an N-gram character string found by searching for a gram character string, the multiplied value is compared with a threshold value, and an estimated text is generated only when the threshold value is exceeded. May be.

On the other hand, if the input text is, for example, “this form ♪ {¶}, the advantage is”, the unknown word detection unit 9 operates as follows. Unknown word detection unit 9
Referring to the non-target character pattern memory 10 shown in FIG. 11, it can be seen from the record L1101 that “♪”, “‰”, “¶”, and “∬” are non-target characters, and “♪ ‰ ¶ ∬ ”is detected as an unknown word (non-target character continuous portion), and its length L is detected as 4 (step S1501). If the text received by the communication means is an input text, a burst error is likely to occur, and a large number of non-target characters may continue.

In this case, since L> N is satisfied (step S1502), a marker text “the advantage in this form {♪} ¶¶} is generated by adding a marker to the unknown word part, and is stored in the input buffer memory 12. Is stored (step S
1506). FIG. 12B shows the storage state of the input buffer memory 12 in this case.

FIG. 16 is a flowchart for explaining in detail the operation of the input control in step S1402 described above.

The input control unit 11 is provided with the input buffer memory 1
One record is taken out from Step 2 (Step S160)
2). Then, it is determined whether or not an unknown word marker is added to the retrieved record.

If an unknown word marker is added (that is, the retrieved record is a marker text) (Yes in step S1602), an unknown word marker portion is cut out from the record, the marker is removed, and the unknown word marker is removed. The data is stored in the buffer memory 13 (step S160)
3: see FIG. 13). In this case, the marker text is divided into two with the unknown word part removed.

On the other hand, if no unknown word marker is detected in step S1602 (that is, if the retrieved record is an estimated text), an unknown word-free symbol (for example, “＠”) is stored in the unknown word buffer. It is stored in the memory 13 (step S1604).

Thereafter, the first half of the unknown word portion of the marker text or the estimated text is transferred to the extended character string generator 4 (step S1605). Here, in order to synchronize with the operation of the output synthesizing unit 14 described later, the input control unit 11 monitors that the unknown word buffer memory 13 becomes empty (step S1606), and the unknown word buffer memory 1
When 3 becomes empty, it is checked whether all the records have been processed (step S1607).

If all the records have not been processed, that is, if the latter half of the unknown portion of the marker text is left (negative result in step S1607), steps S1602 to S1607 are repeated.

If all the records have been processed, it is checked whether any text is left in the input buffer memory 12 (step S1608), and an unprocessed record is left in the input buffer memory 12. Then, steps S1601 to S1608 are repeated, and if an unprocessed record is not left in the input buffer memory 12, finally, an input end symbol (for example, “＄”)
) Is stored in the unknown word buffer memory 13 (step S1609), and the input control unit 11 ends a series of operations.

For example, if the input buffer memory 12
If the state is (A), first, “the advantage of this morpheme answer” (record L1201) is extracted (step S1601), and there is no unknown word marker in the record (step S1602). The wordless symbol "$" is stored in the unknown word buffer memory 13 (step S1604). Then, the record "the advantage of this morpheme answer" is passed to the extended character string generation unit 4 (step S1605), and thereafter, morphological analysis is performed in the same manner as in the first embodiment. On the other hand, if the input buffer memory 12 is in the state of FIG. 12 (B), “the advantage in this form {♪} ¶” (record L1203) is extracted (step S1601), and the unknown word marker is detected. (Step S1602), the unknown word part “♪ ‰
¶∬ ”is stored in the unknown word buffer memory 13 (step S1603). Then, the first half “this form” of the unknown part of the marker text is passed to the extended character string generation unit 4 (step S1605), and thereafter, morphological analysis is performed as in the first embodiment.

Thereafter, the unknown word buffer memory 13 is monitored to synchronize with the output combining operation described later (step S1606). When the output combining operation is completed, the next estimated text (L1202 "this morpheme" The advantage in the analysis ") or the latter half of the unknown word part of the marker text (" the advantage in the analysis ") is similarly processed. Finally, the input end symbol (for example, "@") is stored in the unknown word buffer memory 13 (step S16).
09).

FIG. 17 is a flowchart for explaining the output synthesizing operation in step S1403 described above in detail.

The output synthesizing unit 14 synchronizes with the input control unit 11 depending on whether or not the unknown word buffer memory 13 is empty (step S1701). That is, the operation is started when the unknown word or the symbol "@" without unknown word is stored in the unknown word buffer memory 13, and the unknown word text is extracted from the unknown word buffer memory 13 (step S1702). Then, it is determined whether or not the extracted unknown word text is the input end symbol “＄” (step S1703). If the input end symbol is "＄", a series of output control operations are terminated,
If the input end symbol is not "$" (a negative result in step S1703), an output text is received from the optimum route search unit 7 (step S1704).

Thereafter, it is determined whether or not the unknown word text extracted in step S1702 is an unknown word absence symbol “記号” (step S1705). If the symbol without unknown word is not “$”, the unknown word text is added to the output text from the optimum route search unit 7 (step S1706). If the symbol has no unknown word “な し”, the output text from the optimum route search unit 7 is passed to the output device 8 as it is (step S17).
07).

Finally, the unknown word buffer memory 13 is cleared (step S1708), and the above-described step S17 is performed.
Return to 01. When the unknown word buffer memory 13 is cleared, the input control unit 11 starts the next operation.

For example, if the input buffer memory 12
If the state is (B), the unknown word portion “♪ ‰ ¶∬” is stored in the unknown word buffer memory 13 (step S1).
603). Then, the first half “this form” of the unknown part of the marker text is passed to the extended character string generation unit 4 (step S1605), and thereafter, morphological analysis is performed as in the first embodiment. The output synthesizing unit 14 starts operating when the unknown word portion “♪ ‰ ¶∬” is stored in the unknown word buffer memory 13 (step S1701), and the output text “/ this / form / (Step S1704), an unknown word is added to the text (step S1706), and "/ this / form /
♪ ‰ ¶∬ / ”is output (step S1707).

Next, since the unknown word buffer memory 13 is cleared (step S1708), the input controller 11 is cleared.
Restarts the operation, and the second half of the marker text, the “advantage in the word”, is processed in the same way, and receives the output text “in / advantage / ha /” from the optimum route search unit 7 (step S1704). In this case, since the unknown word text is the unknown word-free symbol “＠”,
“OK / Advantage / Wa /” is output (Step S17)
07).

(B-3) Effects of the Second Embodiment In the second embodiment, the same components as those in the first embodiment are provided, so that the same effects as in the first embodiment can be obtained. Can be.

In addition to this, according to the second embodiment,
Since the apparatus includes the unknown word detection unit 9, the input control unit 11, the output synthesis unit 14, and the like, the following effects can be obtained.

That is, according to the second embodiment, an unknown word portion (non-target character string) is detected, and if the unknown word portion is short, a character string considered to be the original character string of the unknown word portion Can be estimated.

For example, with respect to the input text “Advantage of this morpheme ‰ ¶”, “/ this / morpheme /
The morphological analysis results of “/ this / morphology / advantage / ha /” and “/ this / morpheme / analysis / ni / in / advantage / ha /” can be obtained. That is, an unknown word “な る ¶” can be detected, and a character string conventionally treated as an unknown word can be estimated as a character string considered to be correct, such as “answer” or “analysis”. .

According to the morphological analyzer of the second embodiment, an unknown word portion (non-target character string) is detected, and even when the number of characters is large, the accuracy of morphological analysis of a portion other than the unknown word is improved. A desired morphological analysis result can be obtained without any loss.

For example, with respect to the input text “This form ♪ ‰ ¶∬ 利 点 け る 利 点”, the morphological analysis of “This form” and “Advantage in the form” are performed independently, and “/ this / form / ♪” is synthesized. A morphological analysis result of {‰} / can / advantage / ha / ”can be obtained. That is, in the past, the configuration of performing the morphological analysis of the whole "the advantage in this form ♪ ‰ ∬ 利 点」 "was morphologically analyzed. According to the second embodiment, accurate morphological analysis can be performed without being affected by unknown words.

(B-4) Modified Embodiment of Second Embodiment In the second embodiment, when an unknown word cannot be estimated, the input text is divided, morphological analysis is sequentially performed for each of them, and finally, synthesis is performed. However, morphological analysis on a plurality of divided texts may be performed in parallel.

In the second embodiment, the non-target characters for detecting an unknown word are in units of one character. In addition, a combination of two or more characters (in a idiom) However, it may be registered in the non-target character pattern memory 10. If you do this,
For example, idioms that are not used in common sentences
It can be used for morphological analysis by replacing it with idioms often used in general sentences equivalent to it.

Further, in the second embodiment, the non-target character pattern memory 10 is of a fixed type, but a non-target character pattern editing unit and an input device for the same are provided so that the user can register, delete, etc. May be executed.

Further, in the second embodiment,
Although the unknown word detection unit 9 uses the extended character table 3 used by the morphological analysis elements such as the extended character string generation unit 4, the extended character estimation unit 5, and the chain probability calculation unit 6, it is different from this. May be used. For example, a table storing an N-gram character string (preferably having a chain probability) of characters that do not include extended information may be used.

In the second embodiment, the number L of non-target characters is the degree (number of characters) N of the N-gram extended character string.
In the above case, the estimation operation for the character string considered to be correct is not performed. However, if L is 2N−2 or less, the estimation accuracy is reduced, but the estimation operation may be performed. For example, a non-target character string is roughly evenly divided into two,
Scanning the extended character table with the first non-target character string and the target character before it, and scanning the extended character table with the second non-target character string and the target character on the rear side, and integrating both scan results Thus, a character string considered to be correct may be estimated.

In the second embodiment, the morphological analysis is performed by the method according to the first embodiment after replacing the unknown word portion of the input text with an estimated character string considered to be correct. It may be one that performs morphological analysis by another method (for example, one that uses a word dictionary), and that performs natural language processing other than morphological analysis on the input text after the estimated replacement (estimated text) The present invention can also be applied to a case where only the estimated replacement is performed and the natural language processing is not performed thereafter. For example, the features of the second embodiment can be applied only as a means for returning garbled characters (unknown words) in a transmitted text to original characters.

(C) Other Embodiments In the above description of the first and second embodiments, various modified embodiments have been described, but the following modified embodiments can be further mentioned.

In each of the above embodiments, the frequency information is stored with the chain probability for the N-gram extended character string in the extended character table 2, but the frequency itself may be stored. In this case, for example, the probability is 1
The total frequency may also be stored for each group of the N-gram extended character string, and replaced with a probability when calculating the score (evaluation value) of the route. Alternatively, the route score may be calculated based on the sum of the frequencies of the N-gram extended character strings.

The extended character table 2 and score table 3 in each of the above embodiments may be realized by a configuration other than the table configuration.

Furthermore, in each of the above embodiments, the target natural language is Japanese, but the present invention can be applied to input texts in other languages. Here, as the other languages, not only languages in which word breaks are not clearly defined by spaces or the like, but also languages in which word breaks are clear by spaces or the like may be used.
For example, the feature of the second embodiment that considers garbled characters and the like is very effective even in a language in which word breaks are clearly defined by spaces or the like.

[0173]

As described above, according to the first aspect of the present invention,
It is possible to realize a natural language processing device capable of obtaining a good morphological analysis result even for an input text including an unknown word portion due to garbled characters, typos, and the like.

Further, according to the second aspect of the present invention, an unknown word portion can be detected from an input text including an unknown word portion due to garbled characters, typographical errors, etc., and the portion can be restored to a correct character string. A natural language processing device can be realized.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a first embodiment.

FIG. 2 is an explanatory diagram illustrating a configuration of an extended character table according to the first embodiment.

FIG. 3 is an explanatory diagram illustrating a specific example of an extended character table according to the first embodiment;

FIG. 4 is an explanatory diagram illustrating a configuration of a score table according to the first embodiment.

FIG. 5 is an explanatory diagram showing a specific example of a score table according to the first embodiment.

FIG. 6 is a flowchart illustrating an overall operation of the first embodiment.

FIG. 7 is a flowchart illustrating an operation of generating an extended character string according to the first embodiment.

FIG. 8 is a flowchart illustrating an operation of calculating a chain probability according to the first embodiment.

FIG. 9 is a flowchart illustrating an operation of selecting an optimum extended character string according to the first embodiment.

FIG. 10 is a block diagram illustrating a configuration of a second embodiment.

FIG. 11 is an explanatory diagram illustrating a configuration of a non-target character pattern memory according to the second embodiment.

FIG. 12 is an explanatory diagram illustrating a configuration of an input buffer memory according to a second embodiment.

FIG. 13 is an explanatory diagram showing a configuration of an unknown word buffer memory according to the second embodiment.

FIG. 14 is a flowchart showing an overall operation of the second embodiment.

FIG. 15 is a flowchart illustrating an operation of detecting an unknown word according to the second embodiment.

FIG. 16 is a flowchart illustrating an input control operation according to the second embodiment.

FIG. 17 is a flowchart illustrating an output combining operation according to the second embodiment.

[Explanation of symbols]

2 ... extended character table, 3 ... score table, 4 ... extended character string generation unit, 5 ... extended character estimation unit, 6 ... chain probability calculation unit, 7 ... optimal route search unit, 9 ... unknown word detection unit, 10 ... non- Target character pattern memory, 11: input control unit, 12: input buffer memory, 13: unknown word buffer memory, 14
... Output synthesis unit.

Claims (15)

[Claims] An extended character is formed by adding at least extended information including word delimiter information for each character of a character string of a read input text, and all combinations of the character string of the input text are formed using the extended character. An extended character string generation unit that generates an extended character string of: an extended character storage unit that stores a partial extended character string having a fixed number of characters and partial chain probability information for the partial extended character string; Based on the paths of all the partial extended character strings up to the end and the partial chain probabilities stored in the extended character storage unit, for each of the extended character strings generated by the extended character string generation unit, the chain probability A chain probability calculation unit for obtaining information; a score storage unit for storing the obtained chain probability information; and an extension for providing an optimal chain probability from the obtained chain probability information. An optimal path search unit that selects a character string and outputs an analysis result including a sequence of word strings corresponding to the extended character string as a morphological analysis result; and a partial extended character of the extended character string generated by the extended character string generation unit. When a string does not exist in the extended character storage unit, a partial chain of another partially extended character string having a common extended character with some extended characters of the partial extended character string stored in the extended character storage unit A natural language processing apparatus comprising: an extended character estimating unit that estimates partial chain probability information of the partial extended character string from the probability information. 2. The natural language processing device according to claim 1, wherein the input text is a Japanese text. 3. The extended character estimating unit additionally stores the estimated chain probability information together with a partial extended character string related to the information in the extended character storage unit.
Or the natural language processing device according to 2. 4. The extended character estimating unit regards one or more extended characters of the partial extended character string that does not exist in the extended character storage unit as general-purpose characters that can be matched with all characters, 2. A partial chain probability information of a partial extended character string is estimated from partial chain probability information of one or a plurality of partial extended character strings that match with the contents stored in the extended character storage unit. 4. The natural language processing device according to any one of claims 1 to 3. 5. The extended character estimating unit calculates an arithmetic average of partial chain probability information of one or a plurality of partial extended character strings that match with the contents stored in the extended character storage unit. The natural language processing device according to claim 1, wherein the estimated value is partial chain probability information. 6. The optimal route search unit is capable of externally setting an optimal selection condition for selecting an optimal extended character string related to an input text from contents stored in the score storage unit. The natural language processing device according to claim 1. 7. The optimum route search unit prepares, as a settable optimum selection condition, a condition that an extended character string having chain probability information equal to or greater than an arbitrary threshold value is selected. 7. The natural language processing device according to any one of 1 to 6. 8. A partial character string storage unit that stores a partial character string having a fixed number of characters, a non-target character pattern storage unit that stores a pattern of a non-target character registered in advance when forming an unknown word, Based on the content stored in the non-target character pattern storage unit,
And detecting an unknown word portion in the read input text, and, based on the storage content of the partial character string storage unit, including an unknown word detection unit that estimates a character string considered to be correct for the unknown word portion. Characteristic natural language processing device. 9. The natural language processing apparatus according to claim 8, wherein the non-target character pattern storage unit sets in advance a non-target pattern and a non-target character range listing non-target characters. . 10. The natural language according to claim 8, further comprising morphological analysis means for performing morphological analysis on the estimated text, which is the input text after the unknown word detection unit has performed the estimation operation. Language processor. 11. The morphological analysis unit adds extended information including at least word delimiter information to each character of a character string of an input text to form an extended character, and uses the extended character to form the extended text. An extended character string generation unit that generates extended character strings of all combinations of character strings; an extended character storage unit that stores a partial extended character string having a fixed number of characters and partial chain probability information for the partial extended character string; Based on the paths of all the partial expansion character strings from the beginning to the end of the input text and the partial chain probabilities stored in the expansion character storage unit, all of the expansions generated by the expansion character string generation unit For each of the character strings, a chain probability calculating unit that obtains the chain probability information, a score storage unit that stores the obtained chain probability information, and whether the obtained chain probability information is An optimal path search unit that selects an extended character string that provides an optimal chain probability and outputs an analysis result including a sequence of word strings corresponding to the extended character string as a morphological analysis result. The natural language processing device according to claim 10. 12. The extended morphological analysis means stores the extended character string of the extended character string generated by the extended character string generation unit in the extended character storage unit when the extended character string does not exist in the extended character storage unit. An extended character estimating unit for estimating partial chain probability information of the partial extended character string from partial chain probability information of another partial extended character string having a common extended character with a part of the extended character string; The natural language processing device according to claim 11, further comprising: 13. The natural language processing device according to claim 11, wherein the unknown word detection unit uses the extended character storage unit as the partial character string storage unit. 14. The unknown word detecting section, when the number of characters of the detected unknown word portion is equal to or more than a predetermined number of characters, generates a mark text in which the unknown word portion is marked without performing an operation of estimating a correct character string. For the mark text, a character string portion other than the unknown word portion is separated and provided to the morphological analysis means, and the analysis result for each character string portion from the morphological analysis means is combined with the unknown word portion. 14. The natural language processing device according to claim 10, further comprising a morphological analysis input / output control unit that performs the morphological analysis. 15. The natural language processing device according to claim 8, wherein the input text is a Japanese text.