JP2000040085A - Method and device for post-processing for japanese morpheme analytic processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for acquiring rewritting rules capable of automatically correcting an analytic error in the case of morpheme analysis in natural language processing. SOLUTION: By providing a method for generating the rewritting rules from the comparison of the analyzed results including a corrected answer and an error and for further generalizing these rules and a method for measuring the reliability of generated rules, a rewritting rule set capable of automatically correcting the error is provided. further, in order to cancel vagueness, which can not be solved only by local collation, or to utilize rules showing a non- registered word to repeatedly appear in a sentence set within fixed range in spite of low reliability, morpheme analysis and post-processing therefor are applied for each unit called 'window'.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a post-processing method of Japanese morphological analysis for dividing a Japanese sentence into constituent words, and is used for improving the accuracy of machine translation and indexing for information retrieval. .

[0002]

2. Description of the Related Art An overview of conventional techniques will be described with reference to References 1 and 2. Reference 1: Toru Hisamitsu Yoshihiko Nitta "Comparison of general-purpose algorithm for morphological analysis with likelihood and likelihood criterion using it"
IEICE Transactions D-II Vol. J77-D-II No. 5, p
p. 959-969 (1994) Reference 2: Brill, E .; “Transformation-Based Error D
riven Learning and Natural Language Processing: AC
ase Study in Part-of-Speech Tagging ”, Computation
al Linguistics, Vol. 21, No. 4, pp. 543-565 (19
95) The Japanese morphological analysis processing refers to a procedure for recognizing an input Japanese character string as a sequence of a set of a character string of a constituent word and a part of speech while referring to a dictionary and grammar. In the following, since the method of the morphological analysis processing itself is not specified, the details of the method of the morphological analysis processing are transferred to Document 1 and will not be described here.

[0003] From the viewpoint of the likelihood criterion, the morphological analysis processing is performed as follows.
They are broadly classified into those based on rules given in advance and those based on probabilities (such as an adjacent probability model between words).

[0004] Among them, it is possible to construct a relatively high-accuracy system simply by referring to small-scale experimental data in order to utilize human intuition. Many morphological analysis systems based on rules are used because they are readable.

In recent years, with the speeding up of morphological analysis processing, morphological analysis processing is often used for indexing information retrieval of newspapers, patents, etc., and the analysis accuracy of various documents is improved together with the speed. There is a need.

The causes of errors in the morphological analysis processing are the presence of unregistered words and incomplete grammar and likelihood criteria. As a countermeasure against errors caused by unregistered words, it is important to acquire unregistered words offline and register them in a dictionary.However, this alone is not sufficient for unregistered words appearing in unknown documents and deficiencies in grammar and likelihood criteria. Inability to address resulting errors. For this reason, research on manually generating templates and rewriting rules from analysis of error cases has been conducted. However, as the number of rewrite rules increases to deal with various errors, it becomes extremely difficult to maintain consistency between rewrite rules and to evaluate side effects.
It is necessary to automatically obtain and evaluate rewrite rules.

[0007] In Literature 2, for an English POS-tagger (part-of-speech program) based on a rewriting rule, the analysis result is compared with the correct answer (a set of sentences to which the correct part-of-speech is given manually) to start. A method for automatically discovering a group of rewrite rules to modify the output of POS-tagger given to was proposed. The method is called "error-driven rewrite rule acquisition" and is based on three elements: Any POS-tagger. 2． Allowed rewrite rule set (rewrite rules are
Including part-of-speech rewriting and applicable conditions). 3． An evaluation function that compares the effect before and after applying each rewrite rule with the correct answer and evaluates its effect.

[0008] The set of permitted rewrite rules is generally a huge set and cannot be manually written. Therefore, the set is actually given in the form of a template for generating them. For example, the template: za => zb represents a rewrite rule that if the part of speech of the word before the word a is z, a is rewritten to b. By substituting specific values for a, b, and z, a number of specific rewrite rules are obtained. In Reference 2, 14 steps of templates are prepared manually, and a step of rewriting the output result of the given POS-tagger is automatically found as follows according to a plurality of rewriting rules that embody them. : Step 1) Using the given POS-tagger, a part of speech is assigned to a set of learning sentences prepared in advance, and this is set as T. step 2) Apply all the permissible rewrite rules individually to T, select the rewrite rule whose result is closest to the correct answer by the evaluation function, and record it in the ordered list L. If no rewrite rules improve the result, end and return L. step 3) Rewrite T based on this rewriting rule as T again, and return to step 2.

In this way, when L is obtained, the output result of the given POS-tagger is rewritten sequentially using the rewriting rules in L (the rewriting may be performed several tens of times). POS-tagger can be improved.

[0010] This method is based on the assumption that English is given a word boundary. Therefore, it is assumed that only the part of speech of each word is rewritten. It is complicated. In fact, there are four types of errors in Japanese morphological analysis: (A) Incorrect segmentation. It is divided into the following three types: (A-1) Over-division Positive: Today / no / gold / quote / is ... Incorrect: today / no / gold / quote / quotes / is ... (A-2 Insufficient division Correct: Unix / workstation False: Unix workstation (A-3) Other mistakes (word boundary crossing type) Correct: ill / ga / proliferation /. Wrong: illness / gaman / nobu /.

(B) Error in only part of speech Correct: ... "/ and (quoted particle) / i / u False: ..." / and (parallel particle) / i / u Since it is necessary to take into account the change of the rewrite rule, the allowable rewrite rule set becomes astronomically large, and it is almost impossible to prepare an appropriate template manually in advance.

[0012] Even if the rewriting rule can be manually constructed, there is a problem that the conventional method cannot basically handle it. That is, the following errors due to ambiguity cannot be eliminated. For example, a string like "price difference"
Although there are a plurality of possible divisions such as “price / difference” and “price / difference”, since morphological analysis generally determines based on local information, it may not be possible to determine which is correct. Therefore, one must be arbitrarily output, leading to an error. In general, such ambiguity can be resolved by referring to a rather wide range of contexts, but it is difficult to catch errors caused by such ambiguity only with local rewrite rules. In addition, it is impractical to deal with the probability model regarding the arrangement of words because the data is too large.

[0013]

As described above, the conventional post-processing technique has a problem in a method for automatically acquiring a rewrite rule and in eliminating an error caused by ambiguity. The present invention can be applied to Japanese having a complicated error pattern because there is no delimiter between words, and provides a post-processing technique that eliminates errors caused by ambiguity.

[0014]

According to the present invention, the accuracy of the morphological analysis processing is improved by the following steps.

(1) Automatic generation of rewriting rules for abstraction level: Rewriting rules of various abstraction levels (including changes of word boundaries) are automatically generated by comparing the analysis error part of the morphological analysis result with the correct answer. (2) Automatically judging the effect of the rewriting rule: automatically judging the effect of the generated rewriting rule and recording an effective rule; (3) Maximum likelihood solution of the morphological analysis result Is output: the maximum likelihood solution of the morphological analysis result is output for a sentence in a predetermined range. (4) The modification step applying the rewriting rule: the output result of the morphological analysis of the target sentence is The four steps of applying a rewrite rule and making a correction are fundamental.

In addition, if the concept of "window" is introduced, the analysis result can be further improved. That is, the above steps (3) and (4) are modified as follows, and a second-stage correction step (5) is added.

(3) 'Step of outputting the maximum likelihood solution of the result of morphological analysis: The maximum likelihood solution of the morphological analysis is output for a sentence in a predetermined range called a "window", and at the same time it is possible to output an ambiguous part. (4) 'Correction step applying rewriting rule: applying the modification rule to the output result of morphological analysis of the sentence in the window to perform the first-stage modification, (5) Second-stage correction step: The second-stage correction for determining the unregistered words and resolving ambiguity for the first-stage correction of the analysis result of the sentence in the window.

[0018]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a case where a window is used will be specifically described.

(1) Step of Generating a Rewrite Rule The generation of a rewrite rule assumes the existence of correctly analyzed data. The rewriting rule is automatically generated by comparing the analysis result and the correct data in the following steps:-1st step-K and L are natural numbers, and for all error parts, the error part is rewritten correctly as follows. Generate a rewrite rule consisting of various concrete words: a1. . . aKW1W2. . . Wnb1. . . bL =
> A1. . . aKW1'W2 '. . . Wm'b
1. . . bL where W1W2. . . Wn, W1'W2 '... Wm', a1 ... a
K, b1 ... bL are word strings, W1W2 ... Wn are error parts, W1 '
W2 '... Wm' is the correct answer corresponding to W1W2 ... Wn, a1 ... aK, b1 ...
bL is the environment where rewriting is applied (also called context)
It is.

Hereinafter, W1W2... Wn is a first pattern, W1'W
2 '... Wm' is the second pattern, a1 ... aK is the forward environment, b1 ... b
L is called the rear environment. If K and L are too large, data sparseness will occur and, for simplicity, K
= L = 1.

For example, from the error (A-1), the following rewriting rule R1 is obtained: (R1) Fri: ordinary noun / phase: noun affix / field: noun affix /
Ha: adjunct particle => Fri: common noun / market: common noun / ha: adjunct particle-2nd stage-All rewrite rules obtained in the 1st stage are converted into the following generalized patterns G1, G2, G3 .. Generalize using the combination of.: (G1) Focus only on the parts of speech of the first and second patterns.

(G2) Each character of the first and second patterns is replaced with a character type.

(G3) Attention is focused only on the part of speech of the front environment.

(G4) Characters in the front environment are replaced with character types.

(G5) Ignore the forward environment.

(G6) Attention is focused only on the part of speech in the rear environment.

(G7) The characters in the rear environment are replaced with character types.

(G8) Ignore the rear environment.

For example, when the generalized patterns G2, G3, G6 are applied, the rewriting rule R1 is rewritten as follows: (R1) ordinary noun / "C1": noun affix / "C2": noun affix / particle
=> Normal noun / "C1C2": Normal noun / Particle Here, "Ci" represents a kanji. A modified rewrite rule can be generated by an arbitrary combination and can be selected by a rewrite rule evaluation procedure. However, in practice, it is preferable to select a combination that is considered effective by a preliminary experiment in order to reduce the number of selection steps. For example, we limited to 22 combinations.

(2) Procedure for judging the effect of the modified rewriting rule In the method of Brill, the rewriting rules are stored in an ordered list and applied to sequentially rewrite the output of the initial state annotator. In contrast, we intended to finish the post-processing by scanning the output of the morphological analysis once only, and applied the rewrite rule using the reliability of the rewrite rule and the length of the rewrite pattern. " Greedy
method "(discussed below), without creating an ordered list.

In order to express the effect of the rewriting rule, a numerical value called "reliability" is given as follows. First, for a rewrite rule R, a triple of natural numbers {T, CE, EC}
Is defined as follows: T: The number of words rewritten by the rewriting rule. CE: The number of words that were correctly analyzed before rewriting and were incorrectly rewritten. EC: The number of correctly rewritten words among words that were incorrectly analyzed before rewriting. From the triple {T, CE, EC}, the reliability r of R is defined as follows:

[0032]

(Equation 1)

However, in this case, the reliability of the infrequently rewritten rule tends to be extremely large, so the correction was made by a simple method called "+1 rule". In other words, virtually, it is considered that correct → wrong rewrite, wrong → correct rewrite, wrong → wrong rewrite occurred one by one extra,
Recalculate {T, CE, EC}. At this time, if the number of words rewritten by the rewriting rule is n, the rewritten ternary set {T ′, CE ′, EC ′} is EC ′ = EC + n, CE ′ = CE + n, T ′ = T + 3n, and using these,

[0034]

(Equation 2)

Thus, the reliability is redefined. Rules with negative confidence have been eliminated.

The following is an example of the rewrite rule and the reliability. Here, when the pre-environment or the post-environment is ignored by applying the generalized pattern G5 or G8, it is expressed by "*" indicating that anything matches: Example 1) * / and: Parallel Particles / I: consonant verb stem => * / and: quoted particle / i: consonant verb stem ... reliability: 0.947 Example 2) Particle / "C1": ordinary noun / "C2": not yet Registered word / Particle => Particle / "C1C2": Common noun / Particle ... Reliability: 0.667 Example 3) * / "C1C2C3": Proper noun / "C4": Common noun / * => * / "
C1C2 ": common noun /" C3C4 ": common noun / * .... Reliability: -1.58 Among these, the rule with negative reliability as in Example 3 is discarded.

As described above, according to the present invention, the morphological analysis processing is improved by introducing the reliability of the rewriting rule into the application of the rewriting rule. By introducing the concept of "window", it is devised so that sentences in the "window" can be comprehensively evaluated.
That is, following the above-described steps, the following steps are continuously performed.

(3) 'For the sentence in the window, output the maximum likelihood solution of the morphological analysis and at the same time hold the possible solution for the ambiguous part. As a result of the morphological analysis, the given likelihood criterion indicates the maximum likelihood. In some cases, a high-degree solution cannot be determined uniquely, and one solution must be output arbitrarily, so that an error often occurs. On the other hand, as in the conventional case, one solution is output, and at the same time, an ambiguous point and its plurality of solutions are recorded. We do not specify how to keep multiple solutions,
Generally, a data structure called “lattice” schematically shown in FIG. 1 is used.

FIG. 1 schematically shows retention of ambiguity by lattice. In the figure, the character position at the beginning of the sentence is 0.
By inserting numbers while increasing the character by one between characters, the character position occupied by each word in the sentence is represented.

In the "price difference" part of the "product price difference", {"
There are two types of ambiguity: price, inequality and {"price", difference ". These ambiguities cannot be resolved by existing rules alone.
Therefore, these are held as paths connecting character positions 3 and 4, and 4 and 6, respectively, and paths connecting character positions 3 and 5, and 5 and 6 respectively.

The analysis and the holding of the result are performed not for each sentence but for a sentence set called a “window”. "Window"
It can be defined using logical units (articles or chapters) or simple quantitative units (for example, 10 consecutive sentences), but it is intended to include a set of connected sentences If newspaper articles are targeted, it is natural that one article is a "window".

The reason for using the “window” is that if attention is paid to expressions repeatedly appearing in a certain range of sentences, an effect that cannot be obtained by processing each sentence using a simple rewriting rule can be expected. It is. Here, we are based on the assumption that, in the sentence set in the window, a specific content word is likely to be used repeatedly because of the relevance of the content.

(4) 'First-step correction to output result of morphological analysis of sentence in window: Analysis result is scanned from the top, and applicable rules are applied to each position. The rules applied at this stage must satisfy the following conditions: (i) The reliability is higher than a predetermined threshold. (ii) As a result of the rewriting, a word string including an ungrammatical connection is not generated. Here, (ii) needs to be checked when applying a type rewriting rule ignoring context.

When there are a plurality of applicable rewrite rules, the rewrite rule with the highest reliability is selected. When there are a plurality of such rewrite rules, the rewrite rule with the largest number of words to be rewritten is selected. After applying the rewriting rule, the collation between the word string and the rewriting rule is re-established from the next position of the rewritten word string. If the rewrite rule has not been applied, the rewrite rule application start word position is advanced by one word.

Here, if the rule of the type that changes the word boundary and generates a new word is the most reliable rewriting rule at a certain place and does not satisfy only the above condition (1), Rewrite rules and their locations,
It is recorded as a "potentially applicable rewrite rule" and a "potentially rewriteable location", and is referred to at the time of modification in the second stage.

The rewrite rules can be searched efficiently by recording them in a TRIE structure.

(5) Using the result of the first-stage correction of the analysis result of the sentence in the window to perform the second-stage correction for determining an unregistered word and eliminating ambiguity, It is assumed that the first rewriting of the analysis result in the window has been completed by the one-stage correction (4). Some rewriting changes the word boundaries, such as particle / "C1": common noun / "C2": unregistered word / particle => particle / "C1C2": common noun / particle. In the above case, if "C1C2" is not registered in the dictionary, it means that an unregistered word has been obtained), * / and: conjunctive particle / i: consonant verb stem => * / And: Quoted particles / i: Consonant verb stems, there are things that only rewrite the part of speech ("*" indicates that it matches any word).

As a modification of the second step described above, for a rewrite rule of a type that changes a word boundary and has a low reliability, a re-evaluation of the reliability is performed within the window for a rewrite rule that is not actually applied. Do.

Specifically, a set of rewrite rules for generating a word w as a result of changing a word boundary is defined as {Ri,..., Rk}.
And when the reliability of Ri is ri (i = 1, ..., k),
The reliability of the event "word w appears in the window"

[0050]

(Equation 3)

Defined by Here, in {Ri, ..., Rk}, if there are some rewrite rules that are the same and have the same context before and after the application point, then the multiplication in the definition of r (w) will duplicate them. Is not added. This is based on the idea that the presence of w is strongly suggested only by different kinds of appearances.

When r (w) exceeds a predetermined threshold, a rewriting rule of {Ri,..., Rk} not applied in the first rewriting is applied, and the analysis result is rewritten. If w is an unregistered word, it is considered that an unregistered word has been detected.

As described in (3) ′ above, the ambiguous part is stored in a lattice format, but the ambiguity that has not been resolved by the above-described stage is resolved.

In general, most of the window is analyzed without ambiguity, and for each ambiguity point, the one that contains many independent words appearing in the part of the window that is analyzed without ambiguity is optimal. It is determined as a path, and if it is different from the solution in the step (3) ′, it is replaced with this. If the only optimal path cannot be determined by this method, no rewriting is performed.

In the case of the example of FIG. 1, as shown in FIG. 2, the sum of the number of appearances of "price" and "difference" is "value"
If it appears more than the sum of the number of occurrences of "and" disparity ", {" price "," difference "} is selected, and the provisional analysis result" product / Corrected to "Product / Price / Difference". That is, the ambiguity can be resolved by reflecting the analysis result obtained from another sentence appearing in the window of FIG.

Although this method is simple, it has been found from experiments that it is effective in improving the accuracy of analysis results.

It should be noted here that, even with a fixed rewriting rule, different results are returned depending on the context by combining it with simple surface information.

Embodiment In the following embodiment, acquisition of rewrite rules and post-processing will be described in accordance with the processing flow shown in FIGS. 4, 5 and 6.
This will be described using an example. A user interface related to a list and presentation of rewrite rules, window settings, and the like will be described with reference to FIGS.

Since it is obvious that the present invention can be implemented on a computer in accordance with the hardware configuration of a general computer as shown in FIG. 3, for example, no particular description will be given. Here, 3011 is a document data file with a correct answer,
3012 is a dictionary file, 3013 is a morphological analysis program file, 3014 is an additional dictionary file, 3015
Is a rewrite rule storage file, 3016 is a rewrite rule extraction program file, 3017 is a post-processing program file, and 3018 is a document file. 302 is an input device, 303
Is a main memory, 305 is a computing device, 306 is a display, and 303 is a communication device for cooperating with another computer.

First, a step of automatically generating a rewrite rule by comparing an error part of a morphological analysis with a correct answer will be described with reference to FIG.

Prior to generating the rewrite rule, the user needs to provide a learning text and correct answer data obtained by correctly analyzing the text. For this purpose, for example, as shown in a display area 9012 of FIG. 10, a text file (eg, “text
1 ") and the result of parsing it correctly (eg" text1.so
l)) may be specified as a set.In this example, the text name and its analysis result are associated by the extension ".sol".

After specifying the text to be used for extracting the rewrite rule, the work memory M
Clear 1, M2, M3, M4.

Next, a sentence included in the learning text is morphologically analyzed in accordance with step 402, and the result is stored in a memory.
Record on M1.

Next, according to step 403, the analysis result of the learning text designated by the user is compared with the analysis result recorded in M1, and the pair of locations that are inconsistent as a word string {W1 ... Wn, W '1 ... W'm｝ is extracted in order from the top, and the surrounding memory A1 ... AK, B1 ... BL, and the working memory M2
(Where W1 ... Wn and W'1 ... W'm are W1 ≠ W '
1, ... Wn ≠ W'm, which is the shortest n, m such that the words before and after W1 ... Wn, W'1 ... W'm match together). K,
Regarding L, in the method for generalizing the rewrite rule consisting of a specific word string referred to in step 4042 to a rewrite rule not based on individual words, how many words before and after the discrepancy part are referred to Depends on
Let K and L be the largest among them. By default, K = L = 1, but can be specified by the user as described below.

According to step 404, for all {A1 ... AK, W1 ... Wn, W'1 ... W'm, B1 ... BL} recorded in M2, step 4041, Step 4042 is performed.

In step 4041, it is determined that $ A1 ... AK, W1 ...
From Wn, W'1 ... W'm, B1 ... BL｝, generate the following rewrite rules consisting of specific words,
Record in: A1 ... AKW1W2 ... WnB1 ... BL => A1 ... AK W1'W2 '... Wm'B
1 ... BL Next, according to step 4042, as described in 403, a generalized rewrite rule is generated from the rewrite rule composed of all the specific words recorded in M2, as described in 403. Then, it is added to the working memory M3.

Next, according to step 405, for all rewrite rules R recorded in M3, step 40
Steps 51 to 4054 are repeated.

At step 4051, R is applied to the analysis result in M1, and the result is recorded in the memory M4. Next, in step 4042, the contents of M4 and M1 are compared, and the number of words for improvement, deterioration, and total rewriting is checked.
Calculate the reliability of r.

In accordance with step 4053, only the rewrite rule R having a positive reliability r is stored in the rewrite rule storage memory in combination with the reliability.

If M4 is cleared and there is a rewrite rule whose reliability has not been verified, the flow returns to step 4051.

The step of obtaining the rewrite rule has been described above.

In the morphological analysis, if there is no break in the input sentence set, a morphological analysis process is performed for each sentence of a certain amount or less using a buffer. In the following, for simplicity, it is assumed that the input received by the morphological analysis processing is already an amount that can be handled.

First, with reference to FIG. 5, a post-processing method in which only a rewrite rule is used without using a “window” will be described.

In the post-processing, the working memories M5 and M6 are first cleared according to step 504. Next, in accordance with step 505, the input sentence is stored in the working memory M5.

In accordance with step 506, steps 5061 to 506 are executed for all the sentences stored in M5.
Repeat step 4: According to step 5061, each sentence of the input sentence set is morphologically analyzed and its output is recorded in the working memory M6.

In accordance with step 5062, the morphological analysis result recorded in M6 is rewritten using the rewriting rule stored in the rewriting rule storage memory by the method described in the modification step (4) to which the rewriting rule is applied. The threshold for applying the rewrite rule was 0.65.

Next, after outputting the contents of the working memory M6 according to step 5063, the working memory M6 is cleared according to step 5064.

The description of the ordinary post-processing method has been completed.

Next, a post-processing method using a “window” will be described with reference to FIG. Morphological analysis and subsequent processing
This is performed in units of a sentence set called “window”. The “window” starts with a minimum of one sentence, and can be specified by a structure such as the number of sentences or “one article” independent of the number of sentences. When specifying with a structure such as "one article", it is necessary to define a window according to the specification of the text to be input. Although the manner of definition is not specified, it is important that the definition of "window" can be explicitly defined.

In the post-processing, the work memories M5, M6, M7 and M8 are first cleared according to step 501. Next, according to step 502, the input sentence is divided for each predetermined "window" and stored in the working memory M5.

According to step 503, steps 5031 to 503 are performed for all windows stored in M5.
Repeat 6: For the window fixed in step 503,
According to step 5031, the sentence in the window is morphologically analyzed, and the first output is recorded in the working memory M6. The location where the minimum cost solution is not uniquely determined and data retaining ambiguity are recorded in the working memory M7.

In accordance with step 5032, the morphological analysis result recorded in M6 is rewritten using the rewriting rule stored in the rewriting rule storage memory by the method described in the modifying step (4) 'applying the rewriting rule. The threshold for applying the rewrite rule was 0.65. At this time,
A rewrite rule that changes word boundaries and is not used because of low reliability.
When a set of words W generated by rewriting satisfies a predetermined condition such as part of speech (for example, “W is a noun”
Etc.) in the working memory M8.

According to step 5033, for all words W recorded in M8, according to step 50331,
Rewrite rules that may generate W {R1, ..., Rk}
Is re-evaluated as described in the second modification step (5). If the re-evaluation result exceeds a predetermined threshold (0.65 in the experiment), the rewriting rule is applied. And rewrite the analysis result in M6.

Next, in accordance with step 5034, for all ambiguities recorded in M7, condition determination step 50341: "whether or not ambiguity has been eliminated by rewriting up to step 5033" is performed. If not resolved, the content word that appears in the position where the unambiguous analysis is performed in M6 among the possible solutions of the ambiguous part according to the method described in the second modification step (5) according to step 50342. And rewrite if it differs from the current result.

After outputting the contents of the working memory M6 after all post-processing in accordance with step 5035,
In accordance with 036, the working memories M6, M7 and M8 are cleared.
This is the end of the description of the post-processing method using the “window”.

In the case where an unknown word is detected as a result of post-processing, regardless of whether a window is used or not, all of the unknown words are recorded in a working memory, and the recorded unknown words are subjected to all post-processing. After completion of the processing, the user can select the words to be registered in the dictionary, register the words in the additional dictionary 3014, and use the words in the subsequent morphological analysis by using the post-processing method. In addition, the reliability of unknown words is extremely high, that is, a predetermined 1
If the value is higher than the threshold value close to
A method of recording the unknown word and using the unknown word in the morphological analysis after the detection is also feasible.

It is also easy to enable the user to interactively select any of these unknown word registration methods (including the setting of a threshold in the latter case).

As additional functions, it is preferable to be able to specify a pattern of a rewrite rule for post-processing and to specify use / non-use of an individual rewrite rule. Give an explanation.

First, a rewrite rule list window as shown in FIG. 7 is provided for a list of automatically obtained rewrite rules. The rewrite rules in the rewrite rule storage memory can be sorted by the sort method designated by the user as in step 6011, or can be sorted in step 6014.
By specifying a search condition by, a desired rewrite rule set is presented as in step 6012. Each rewrite rule has an identification number, forward and backward contexts, an error pattern and a correct answer pattern sandwiched between them, and reliability, and these can be listed.

By requesting detailed information on a specific rewrite rule as in step 6013,
2 to open the detailed information display window as shown in FIG. 2 and present the basis for extracting the rewrite rule (step 6021), present the location where the rewrite rule is successfully applied in the learning set (step 6022), and present the failure location (step 6023) is there.

It is also possible to specify not to use an individual rewrite rule. As in step 6024, a button for instructing not to use the rewrite rule is provided. In the case of the example of FIG. 7, since the rewriting rule is related to the distinction of the particle “to”, it is unnecessary when focusing only on the content word in the information search, for example, and is not used for improving the speed.

Also, as described in the description of FIG. 4, a method of generalizing the specific word rewriting rule to a rewriting rule that does not depend on individual words in step 4042 is specified in advance by a file. Such a method requires a list / edit interface, since list / edit may be required to improve post-processing.

In FIG. 8, reference numeral 701 denotes an example of a window for presenting a general method of rewriting rules. For example, looking at the character type and part-of-speech of the K-th word before the different part is described as "-K, character type, part-of-speech". “No omission” indicates that the word string is directly taken into the rewrite rule, and “*” indicates that the word string can be matched with an arbitrary word.

When the system adds a pattern other than the generalized pattern prepared by default, the area 7011
9, a new pattern creation window 801 is called as shown in FIG. 9, and it is also possible to designate to see the character type and the part of speech of the two words before and after the error portion, for example, as in an area 8011.

It is assumed that the text with the correct answer for learning has already been given. However, when a new text / correct answer pair is explicitly given, a “Specify correct answer data” button 8012 is clicked on in FIG. A learning data designation window 901 as shown in FIG. 9 is called, and a new text / correct answer pair is designated in the region 9014 for the learning data designated for rewriting rule extraction shown in the region 9012,
An additional operation is performed with an add button 9013. Conversely,
The delete button 9011 can be used to delete the correct answer data that has been used.

Thus, the user returns to the new pattern creation window 801 shown in FIG. 9 and operates the button 8014 for instructing the extraction and re-evaluation of the rewrite rule to execute the re-evaluation. A new pattern created using the new pattern creation window 801 can be canceled by operating the button 8013 for instructing erasure.

As described above, it is easier for the user to change the definition of the "window". For this purpose, for example, using a window 101 as shown in FIG. 11, in a simple case, it is preferable that the "window" can be designated by the number of sentences or the structure of "articles".

An area 1011 indicates the type of window currently in use, and its definition can be confirmed in an area 1012. It is also possible to give data to the area 1013 and define a new window.

[0099] 1500 sentences actually extracted from newspapers are manually corrected after morphological analysis to create correct answer data, rules are extracted and evaluated using 750 sentences, and post-processing effects are obtained using the remaining 750 sentences. As a result, the accuracy before the post-processing was improved by 5% for the training data and 3% for the test data.

Post-processing corrects 603 of the errors correctly, 246 of which are due to the type rewriting that changes word boundaries. Of all the corrections, about 7% were "window" effects.

[0101]

According to the present invention, a rewriting rule for morphological analysis correction, which is difficult to prepare manually or difficult to adjust for mutual relations, is automatically generated with a certain degree of reliability. Post-processing that could not be performed was enabled.

Further, by performing post-processing for each sentence set called a “window”, it becomes possible to refer to the analysis result of another sentence, and to eliminate ambiguity which has been difficult to solve by the conventional method. In addition, it is possible to repair a morphological analysis error due to an unregistered word that cannot be recovered only by local pattern matching.

[Brief description of the drawings]

FIG. 1 is a diagram schematically illustrating an example of ambiguity in morphological analysis by an example of retention of ambiguity by lattice.

FIG. 2 is a view showing an example of appearance of words in a window indicating that ambiguity can be resolved by reflecting an analysis result obtained from another sentence appearing in a “window”;

FIG. 3 is a diagram showing an example of a hardware configuration of a general computer capable of executing the present invention.

FIG. 4 is a diagram showing an example of a processing flow for acquiring a rewrite rule.

FIG. 5 is a view for explaining an example of a post-processing method when simply using a rewriting rule without using a “window”;

FIG. 6 is a view for explaining an example of a post-processing method using a “window” and using a rewriting rule.

FIG. 7 is an exemplary view for explaining an example of one rewrite rule list window of the user interface.

FIG. 8 is a view for explaining an example of a list window of one generalized pattern of a user interface.

FIG. 9 is a view for explaining an example of one new pattern creation window of the user interface.

FIG. 10 is a view for explaining an example of a window for designating addition or deletion of one piece of correct answer data in a user interface.

FIG. 11 defines one window of the user interface;
FIG. 9 is a view for explaining an example of a window for specifying a change.

[Explanation of symbols]

301: storage device, 302: input device, 303: main memory, 305: arithmetic device, 306: display, 3
011: Document data file with correct answer, 3012: Dictionary file, 3013: Morphological analysis program file,
3014: additional dictionary file, 3015: rule storage file, 3016: rule extraction program file, 3
017: Post-processing program file, 3018: Document file, 9012: Display area, M1, M2, M3, M4: Working memory, 401-405, 4041-4054: Processing step, 504-506, 5061-5064: Processing step , 501-503, 5031-5036, 503
31-50342: processing step, 601, 602, 7
01,801,901,101: window.

Claims (6)

[Claims] 1. A method for converting unseparated Japanese character string data into character string data by using data in a storage device that holds a word dictionary and a storage device that holds a data table that defines connection relationships between words. In a Japanese morphological analysis process of dividing a Japanese word into constituent words, a storage device for holding document data to which a correct Japanese morphological analysis result is provided, and a morphological analysis result of a document to be analyzed and the storage The correct Japanese morphological analysis result held in the device is compared, and the analysis error part, the corresponding correct answer part, and the words having a predetermined number of words before and after the error part are specified. In the morphological analysis result of an arbitrary document, a word string that matches the error part, sandwiched between the word groups that match the preceding word group and the backward word group, Application conditions are relaxed by a predetermined matching condition relaxation method such as replacing a word string, generating a morphological analysis result rewriting rule, and replacing the morphological analysis result rewriting rule with a word of speech. Conversion to the rewriting rule, evaluating the reliability of the rewriting rule of the morphological analysis result, and rewriting the morphological analysis result of an arbitrary document using the morphological result rewriting rule having the reliability not less than a predetermined threshold value. Characteristic Japanese morphological analysis post-processing method. 2. The post-processing method of Japanese morphological analysis according to claim 1, wherein the morphological analysis processing and the subsequent processing are performed for each predetermined sentence set. 3. A rewrite rule group of a type that generates a word different from the analysis result by changing a word boundary of the morphological analysis result, and when the rewrite rule group is applied, a plurality of positions in the morphological analysis result of the document are obtained. In the case where a specific word is generated in a different appearance environment, the reliability of the rewriting rule group is re-evaluated in a manner that increases with an increase in the number of occurrences of the word. The rewriting rule group is applied when the threshold value exceeds a predetermined threshold value.
Japanese morphological analysis post-processing method described. 4. Using a data storage device that holds a word dictionary and a data storage device that holds a data table that defines the connection relationship between words, Japanese character string data that is not separated and written is composed of the character string data. In the Japanese morphological analysis process that divides Japanese into constituent words, the analysis results for each sentence must be retained with ambiguity. A post-processing method for Japanese morphological analysis characterized by preferentially outputting a solution candidate in which a content word in the solution candidate appears more frequently in a part that has been analyzed. 5. A storage device comprising a document data file with a correct answer, a dictionary file, a morphological analysis program file, an additional dictionary file, a rewrite rule storage file, a rewrite rule extraction program file, a post-processing program file and a document file, an input device, and a main unit. A Japanese morphological analyzer for performing a Japanese morphological analysis process on a computer having a memory, an arithmetic device, and a display, wherein the Japanese morphological analyzer has a list of rewriting rules of a morphological analysis result on a display. ,
A post-processing device for Japanese morphological analysis, characterized in that designation of use / non-use and display of a matching condition relaxation method can be interactively designated. 6. The post-processing device for Japanese morphological analysis according to claim 5, wherein the contents that can be interactively performed by the user include confirmation and redefinition of a definition of a sentence set for performing morphological analysis and post-processing. .