JP2000259625A - Document calibration device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to automatically estimate an error part by utilizing a text whose error generation inclination has been determined. SOLUTION: An error candidate detection part 1 includes error candidate word group knowledge 1a and detects error candidate words from an inputted text. A co-occurrence information extraction part 2 extracts various co- occurrence information corresponding to the error candidate words from the inputted text. A control part 3 controls the detection part 1 and the extraction part 2 so as to extract co-occurrence information over the whole text area. The extracted co-occurrence data are integrated by a statistic information totalization part 4 and the appearance patterns of a target word in the whole document are totalized. An error judgment part 5 judges whether error candidates extracted on the basis of the co-occurrence information are correct or not. It is also available to correct the knowledge of an error candidate word group by preparing an error statistic integration part 6 for integrating statistic information concerned with errors or a similarity evaluation part 8 for evaluating the similarity of two words in the error word group.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document processing apparatus, in which a user proofreads a document by automatically pointing out an erroneous portion in document data input by a user or obtained as an electronic medium. The present invention relates to a document proofreading technique that can reduce the number of operations to be performed and greatly improve the efficiency of document proofreading.

[0002]

2. Description of the Related Art In general, in the case of an error (context-dependent error) such as a homonym error in which it is not possible to judge whether the word itself is correct (context-dependent error), it is necessary to judge whether the word exists in the context of the word. Generally, a method of detecting an error using knowledge about co-occurrence between words is used. As the most basic system, a system that detects an error when an error pattern of co-occurrence data registered in advance matches an error pattern is known.

For example, considering the words “guarantee” and “guarantee”, which are homonyms, the term “security” is a correct expression usually used, but “security”
The term is rarely used. Therefore, a method of registering an error pattern of co-occurrence data of “guarantee ← → safe” in advance and detecting an error in the document by detecting this error pattern is used. Similarly, there is also a system in which correct co-occurrence data of homonymous words are stored in advance, and the co-occurrence data extracted from the text is compared with the co-occurrence data in the database to determine whether or not there is an error.

[0004]

In order to check for context-dependent errors as described above, it is necessary to accumulate target data of errors and correct answers or correct data. To cover, a large amount of co-occurrence data is required. It is difficult for humans to maintain these data, and there are attempts to automatically extract them from a corpus (a database that collects the same type of text). However, general corpora contain some errors, and the extracted data is used as it is. I can't. In order to avoid this, laborious manual checks have become indispensable.

The present invention has been made in view of the above circumstances, and an object of the present invention is to utilize a text in which an error tendency is determined without using co-occurrence data beforehand. An object of the present invention is to make it possible to automatically estimate an error portion and to automatically prepare co-occurrence data usable in a conventional error detection system.

[0006]

FIG. 1 is a block diagram showing the principle of the present invention. In the figure, reference numeral 1 denotes an error candidate detecting unit that has knowledge 1a of an error candidate word group, which is a group of words that are easily confused with each other, and detects those words from an input text;
Reference numeral 2 denotes a co-occurrence information extraction unit that extracts various types of co-occurrence information in a text for an error candidate word, and 3 denotes a control unit. The control unit 3 controls the error candidate detection unit 1 and the co-occurrence information extraction unit 2 On the other hand, it controls which part of the text is to be applied, and causes the target error candidate to extract co-occurrence information over the entire text. Reference numeral 4 denotes a statistical information totalizing unit that accumulates and totals the co-occurrence information extracted by the co-occurrence information extracting unit 2,
Statistical data is calculated by accumulating the extracted co-occurrence data,
The appearance patterns of the target word in the entire document are totaled.

Reference numeral 5 denotes an error determination unit which determines the correctness of an error candidate extracted based on the co-occurrence information. The error determination unit 5 has an expected value for statistical information such as a probability of occurrence of a confusion error of a specific word pair in a word group. When the similarity is judged by comparing information with the expected value and comparing the same type of statistical information calculated from the actually observed co-occurrence data collected and tabulated by the statistical information tabulation unit Then, these co-occurrence data are determined as errors. Reference numeral 6 denotes an error statistics accumulating unit that accumulates statistical information such as the frequency of the output error, and 7 denotes an error candidate correcting unit that corrects the knowledge of the error candidate word group by referring to the information. 8 uses the extracted co-occurrence statistical information,
A similarity evaluation unit that evaluates the similarity of two words in the pair for all pairs in the error word group, and 9 evaluates the semantic similarity of all combinations in the error word group using the co-occurrence statistical information The error candidate correcting unit corrects the knowledge of the error candidate word group based on the error candidate word group.

In the present invention, as described above, co-occurrence data is extracted, the extracted co-occurrence data is accumulated, and the appearance pattern of the target word in the entire document is counted. Since the error determination unit performs the error determination, it is possible to automatically estimate the error portion in the text without having to store the co-occurrence data in advance.
In addition, it is possible to automatically prepare co-occurrence data. Further, by providing the error statistics accumulating unit 6 and the error candidate correcting unit 7, it is possible to correct the knowledge 1a of the error candidate word group in a direction to increase the evaluation of the error that actually occurred, and to extract the error candidate. Processing accuracy can be improved. Further, by providing the similarity evaluation section 8 and the error candidate correction section 9, information on the likelihood of error of the knowledge 1a of the error candidate word group can be corrected, and similarly, the accuracy of the error candidate extraction processing can be improved. Can be.

[0009]

FIG. 2 is a diagram showing a configuration example of a system for performing a document proofreading process according to the present invention. In FIG. 1, reference numeral 101 denotes a display device such as a CRT or a liquid crystal display;
An input / output device 102 including an input device for inputting characters, symbols, instructions, etc., such as a keyboard and a mouse;
Is a CPU, 103 is a memory composed of a ROM, a RAM, etc., 104 is an external storage device for storing programs, data, etc., 105 is a portable storage medium, such as a floppy disk or CD-ROM, for reading / writing data. Is a communication interface including a modem for performing data communication using a telephone line, a network card for performing data communication using a network such as a LAN, and the like. The external storage device 104 stores a program for performing the document proofreading process of the present invention, error candidate word group knowledge, and the like, and stores co-occurrence data, statistical information, and the like obtained in the document proofreading process of the present invention. . Further, various texts to which the present invention is applied can be read from a CD-ROM or the like by using
Via the communication interface 1
06 from the network.

Hereinafter, first to third embodiments of the present invention will be described with reference to FIGS.
A third embodiment will be described. FIG. 3 shows the first embodiment of the present invention.
It is a figure showing the functional composition of the system of an example. In the figure, reference numeral 11 denotes an error candidate detection unit, and 12 denotes an error word group knowledge storing error candidate words composed of homophonic words such as “guarantee” and “guarantee”.
Detects an error candidate from the input partial text based on the error candidate words stored in the error candidate word group knowledge 12 in advance. Reference numeral 13 denotes a co-occurrence data extraction unit, which extracts co-occurrence data by detecting words (for example, “safe” for “guarantee” described above) that are co-occurring with the error candidate from the partial text. The co-occurrence data extraction unit 13 is the same as that generally used in the existing framework.

Reference numeral 14 denotes a control unit. The control unit 14 performs control for applying the error candidate detection unit 11 and the co-occurrence data extraction unit 13 to the entire text. That is, partial text (eg, sentences, paragraphs, etc.) from the entire text
Is applied, and the error candidate detection unit 11 and the co-occurrence data extraction unit 13 are applied to detect the error candidate and extract the co-occurrence data. Detection of candidates and extraction of co-occurrence data are performed, and similarly, detection of error candidates and extraction of co-occurrence data are repeated for the entire text. As a result, co-occurrence data for the error candidate is extracted from the entire text.
Reference numeral 15 denotes a statistical information totalizing unit that accumulates the co-occurrence data extracted by the co-occurrence data extraction unit 13 and performs statistical processing for each type of co-occurrence data. 16 is an error determination unit,
Statistical information on the co-occurrence data is received as input, correct / incorrect judgment is made for each co-occurrence data, and the result is finally output as word correct / incorrect.

FIG. 4 is a flowchart showing an example of control processing by the control unit 14. In this example, a process of sequentially cutting out sentences from a target text and extracting co-occurrence data for each sentence is continued until the text runs out. Shows the case. In step S1 of the figure, a sentence is cut out from the text, and in step S2, an error candidate is detected by the error candidate detection unit 11. When an error candidate is detected, in step S3, a sentence is input to the co-occurrence data extraction unit 13, and co-occurrence data for the error candidate is extracted. Then
In step S4, the extracted co-occurrence data is input to the statistical information totaling unit 15, and the number of occurrences of the co-occurrence data is totaled. In step S5, it is determined whether the processing for the entire text has been completed. If the processing for the entire text has been completed, the processing ends. If the processing has not been completed for the entire text, the process returns to step S1, and the above processing is repeated.

FIG. 5 is a diagram showing an example of co-occurrence statistics extracted and tabulated as described above. In this example, two groups of "operation" and "operation" and "instruction" and "support" are assumed as error candidates, and the number of occurrences of co-occurrence data for those word groups is determined for each type of co-occurrence data. Is counted. As shown in the figure, the word "resume" is extracted as a co-occurrence word for the error candidates "operation" and "operation",
Also, the word “express” is extracted as a co-occurrence word for the error candidates “instruction” and “support”. Then, the total result of the frequency of each co-occurrence data is “6
0 "," 17 "," 519 ", and" 1 ".

FIG. 6 is a flowchart showing the processing in the error judgment unit 16 shown in FIG.
In, the error determination is performed as follows. In step S1 of the figure, the tallying result obtained by the tallying unit 15 is read. In step S2, using the error candidate word group knowledge 12, words that are likely to be erroneous are grouped. In the case of the example of FIG. 5, "operation" and "operation", and "instruction" and "support" are respectively grouped. In step S3, one of the groups is selected, and in step S4, one word is selected from the group. For example, a group of "operation" and "operation" is selected, and the word "operation" is selected. In step S5, the occurrence frequency and the ratio (relative ratio) of the selected word to the entire error word group are calculated. In addition, as the statistical processing, it is also conceivable to calculate the value of a t-test within the error word group.

In step S6, the above frequency and relative ratio are compared with predetermined expected values to determine an error.
For example, in the example of FIG. 3, since the frequency of “instruction” ← → “assertion” is low and the relative ratio is low, this is determined as an error. In step S7, it is checked whether all the words in the group have been selected. If not, the process returns to step S4 and repeats the above processing. If all the words in the group have been selected, the process goes to step S8 to check whether all the groups have been selected. If all the words have been selected, the process ends. If not, the process returns to step S3 to select the next group and repeat the above processing. In the first embodiment of the present invention, as described above, co-occurrence data is extracted and error determination is performed.
Even if co-occurrence data is not stored in advance, it is possible to automatically estimate an erroneous portion in a text. In addition, it is possible to automatically prepare co-occurrence data.

FIG. 7 is a diagram showing a functional configuration of a system according to a second embodiment of the present invention. The configuration of 11 to 16 is the same as that shown in FIG. 3, and in this embodiment, an error statistics totalizing unit 21 and an error candidate correcting unit 22 are added. The error statistics totalizing unit 21 collects the errors detected by the error determining unit 16 every time the text proofreading process is completed, and totalizes the error frequency for each target word. These pieces of information are sent to the error candidate correction unit 22, and the error candidate correction unit 22 corrects the error that has actually occurred in a direction that increases the evaluation, thereby improving the accuracy of the error candidate extraction process. For example, by deleting from the error candidate word knowledge 12 words that have not actually been mistaken for other words in the group of erroneous words, the accuracy of the error candidate extraction process is increased.

An example of the processing algorithm in the error candidate correction unit 22 will be described below. Here, words that are not actually mistaken for other words in the error word group in which data is gathered to some extent are deleted. <Example of processing algorithm in error candidate correcting section> A specific error group is selected and the following processing is repeated. 1. 1. If the number of errors in the entire error group does not exceed the threshold, the process ends. A specific word in the error group is selected and the following processing is repeated. i. If there are no confusion errors with other words in the same group, delete the word from the erroneous word group

FIG. 8 is a flowchart showing the above algorithm. In step S1 of the figure, a specific error group is selected from the error statistics data of the error statistics totalizing unit 21. In step S2, it is checked whether the number of co-occurrence data items in the entire error group is equal to or larger than a threshold value. If it is equal to or smaller than the threshold, the process returns to step S1. If it is equal to or greater than the threshold value, the process goes to step S3, where a specific word in the error group is selected, and in step S4, it is checked whether there is any confusion error with another word in the same group. If there is a confusion error, the process returns to step S3 and the above processing is repeated. If there is no confusion error, the selected word is deleted from the error candidates of the error candidate word knowledge 12 in step S5. In step S6, it is checked whether or not all the words in the error group have been selected. If not, the process proceeds to step S3.
And the above processing is repeated. If the selection has been completed, the process goes to step S7 to check whether the error candidate group has been selected. If not, the process proceeds to step S7.
Returning to step 1, the above processing is repeated, and when the selection is completed, the processing is terminated.

FIG. 9A shows an example of error candidate information tabulated by the error statistics tabulation section 21. In this example, "speak", "release", and "release" are assumed as error candidate information. After the error detection in the text is performed on these words, the counting results of the error statistics counting unit are “5”, “0”, and “0” as shown in FIG. Note that the error statistics tallying unit 21 simply counts the number of cases. FIG. 9B shows a processing example in the error candidate correction unit 22 for the error statistical data example shown in FIG. In this example, since there was no confusion error for the error candidate word "speak", "speak" is deleted from the error word group as shown in FIG. 9B. As a result, the error word group information of the error candidate word knowledge 12 is as shown in FIG.
It is modified from (c) as shown in FIG. 9 (d).

In the second embodiment of the present invention, as in the first embodiment, it is possible to estimate an erroneous portion in a text without storing co-occurrence data in advance, Since words having no confusion errors are automatically deleted from the error candidate word knowledge 12 as described above, the accuracy of error candidate extraction processing can be improved. In addition, it becomes possible to prepare co-occurrence data for easily erroneous words.

FIG. 10 is a diagram showing a functional configuration of a system according to a third embodiment of the present invention. The configuration of 11 to 16 is the same as that shown in FIG. 3 above, and in this embodiment,
A similarity evaluation unit 31 and an error candidate correction unit 32 are added. The similarity evaluation unit 31 receives the total number of generations for each word together with the co-occurrence statistical information from the statistical information totaling unit 15. The error candidate correction unit 32 evaluates the degree of overlap of the co-occurrence statistical information for any two words in an arbitrary group of words that are easily confused, and thereby corrects the error probabilities of the error candidate word knowledge 12.

An example of the processing algorithm in the similarity evaluation section 31 and the error candidate correction section 32 will be described below. Here, words that are not actually mistaken for other words in the error word group in which data is gathered to some extent are deleted. <Example of processing algorithm in similarity evaluation section and error candidate correction section> A specific error group is selected and the following processing is repeated. 1. If the number of co-occurrence data items in the entire error group does not exceed the threshold, the process ends. I. A specific word in the error group is selected and the following processing is repeated. i). Evaluate the similarity with all the other words in the same group by the following method.-Compare the co-occurrence data and all co-occurrences of the type of co-occurrence data that exists in common in two words Calculate the percentage of the data type ii) If the similarity to all other words is less than a certain threshold, delete the target word from the error word group

FIG. 11 is a flowchart showing the above algorithm. In step S1, a specific error group is selected from the statistical information totalizing unit 15. Step S
In step 2, it is checked whether the number of co-occurrence data items in the entire error group is equal to or larger than a threshold value.
If the difference is equal to or larger than the threshold value, the process proceeds to step S3, and a specific word in the error group is selected. In step S4, the co-occurrence data is compared with all other words in the same group, and the ratio of the type of co-occurrence data that is commonly present in the two words to the type of all co-occurrence data that the user has is determined. Calculate and calculate similarity.

In step S5, it is checked whether or not the similarity is less than the threshold value with respect to all other words. If the similarity is less than the threshold value, the process proceeds to step S6, and the target word is determined from the error candidates in the error candidate word knowledge 12. delete. If the value is equal to or larger than the threshold, the process returns to step S3, and the above processing is repeated. Then, in step S7, it is checked whether or not all the words in the error group have been selected. If not, the process returns to step S3 and repeats the above processing. If all the words in the error group have been selected, go to step S8,
It is checked whether the error candidate group has been completely selected. If not, the process returns to step S1 and repeats the above processing. If the selection has been completed, the process is terminated.

FIG. 12 is a diagram showing an example of correction of error candidate information in this embodiment. In this example, it is assumed that “project”, “transfer”, and “transfer” are defined as the error candidate information, as shown in FIG. Here, it is assumed that the calibration process is performed, and the co-occurrence statistical information as shown in FIG. Based on the algorithm described above, the similarity evaluation unit 31 first compares the co-occurrence data to determine the number of types of co-occurrence data that are commonly present in two words. In this example, words co-occurring with “sharing” are “light”, “shape” and “photograph”, and words co-occurring with “mirror” are “light” , “Appearance”, and “movie”. Accordingly, since “light” and “shape” are common to “project” and “project”, the number of common co-occurrence data is “2” as shown in FIG. On the other hand, “copies” and “move” and “mirrors” and “move” have a common co-occurrence data number of “0”.

Next, as described above, the ratio of the type of co-occurrence data that exists in common to the type of all the co-occurrence data that the user owns is calculated, and the similarity is obtained. In this case,
As described above, there are three types of co-occurrence data of “copy”, “light”, “shape”, and “photograph”.
As shown in (d), the similarity of “mirror” to “mirror” is “2/3”. Similarly, the similarity of “mirror” to “mirror” is “2/3”. Also, the similarity of “transfer” to “transfer” and the similarity of “mirror” to “transfer” are calculated as “0/3”, and “transfer” of “transfer”
The similarity of “project” is calculated as “0/2”.

As apparent from FIG. 12D, the similarity of "transfer" to other "shooting" and "shooting" is 0, and there is no actual mistake in other words. Therefore, it is subject to deletion. Therefore, in the case of this example, the error candidate correction unit 32 deletes “move” from the error candidate word knowledge 12. In this embodiment, as in the second embodiment, it is possible to estimate an erroneous portion in a text without storing co-occurrence data in advance, and to automatically obtain error candidate word knowledge 12. This can be corrected, and the accuracy of the error candidate extraction process can be increased. In addition, it becomes possible to prepare co-occurrence data for easily erroneous words.

[0028]

As described above, according to the present invention, it is possible to automatically estimate an erroneous portion by preparing a large number of texts having a tendency to generate errors without having to previously store co-occurrence data. It is possible to do. In addition, since co-occurrence data can be extracted from text and error-prone co-occurrence data can be estimated and accumulated, this system can also be used to prepare co-occurrence data used in a conventional error detection system.

[Brief description of the drawings]

FIG. 1 is a principle configuration diagram of the present invention.

FIG. 2 is a diagram illustrating a configuration example of a system for performing a document configuration process according to the present invention.

FIG. 3 is a block diagram illustrating a configuration of a system according to a first exemplary embodiment of the present invention.

FIG. 4 is a flowchart illustrating an example of a control process performed by a control unit.

FIG. 5 is a diagram illustrating an example of co-occurrence statistics extracted and tabulated in the first embodiment.

FIG. 6 is a flowchart illustrating a process in an error determination unit.

FIG. 7 is a block diagram illustrating a configuration of a system according to a second exemplary embodiment of the present invention.

FIG. 8 is a flowchart showing processing in an error candidate correction unit.

FIG. 9 is a diagram illustrating an example of a processing result according to the second embodiment of the present invention.

FIG. 10 is a block diagram showing a configuration of a system according to a third example of the present invention.

FIG. 11 is a flowchart showing processing in a similarity evaluation unit and an error candidate correction unit.

FIG. 12 is a diagram illustrating an example of correcting error candidate information according to the third embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Error candidate detection part 1a Knowledge of an error candidate word group 2 Co-occurrence information extraction part 3 Control part 4 Statistical information totalization part 5 Error judgment part 6 Error statistics accumulation part 7 Error candidate correction part 8 Similarity evaluation part 9 Error candidate correction part Reference Signs List 11 error candidate detection unit 12 error word group knowledge 13 co-occurrence data extraction unit 14 control unit 15 statistical information totaling unit 16 error determining unit 21 error statistical totaling unit 22 error candidate correcting unit 31 similarity evaluation unit 32 error candidate correcting unit

Claims (6)

[Claims] 1. An error candidate detecting unit that extracts an error word from a text based on knowledge of an error candidate word group that is a group of words that are easily confused with each other, and extracts co-occurrence information in the text for the error candidate word. A co-occurrence information extraction unit, a statistical information aggregation unit that accumulates and aggregates co-occurrence information, and an error determination unit that determines whether the co-occurrence information is correct based on the statistical information of the co-occurrence information. Characteristic proofreading device. 2. An error candidate detecting section which has knowledge of an error candidate word group, which is a group of words that are easily confused with each other, and detects those words from an input text. A co-occurrence information extraction unit that extracts the co-occurrence information, a control unit that controls which part of the text is applied to the error candidate detection unit, and the co-occurrence information extraction unit. A statistical information summarizing unit for summarizing, and an error determining unit for judging whether the error candidate extracted based on the co-occurrence information is correct, and the control unit controls the error candidate detecting unit and the co-occurrence information extracting unit. The co-occurrence information over the entire text is extracted for the target error candidate, and the statistical information aggregation unit accumulates the extracted co-occurrence information and calculates statistical data. Aggregation of patterns A document proofreading apparatus, wherein the error determination unit determines correctness or incorrectness in units of co-occurrence information based on the statistical information. 3. An error judging unit has information of expected values for statistical information such as a probability of occurrence of a confusion error of a specific word pair in a word group, and is actually observed and collected by a statistical information collecting unit. Comparing the statistical information of the same type calculated from the co-occurrence data with the expected value and determining the similarity, and when the similarity is high, determining the co-occurrence data as an error. Item 2. The document proofreading device according to item 2. 4. An error statistics accumulating unit for accumulating statistical information such as the frequency of an output error, and after performing a proofreading process on a text, correcting the knowledge of an error candidate word group by referring to the information. 4. The document proofreading apparatus according to claim 2, further comprising an error candidate correcting unit. 5. A similarity evaluation unit that evaluates the similarity of two words in the error word group for all pairs in the error word group using the extracted co-occurrence statistical information, and performs a proofreading process on the text. Later, an error candidate correction unit that evaluates the semantic similarity of all combinations in the error word group using the co-occurrence statistical information and corrects the knowledge of the error candidate word group based on the evaluation is provided. The document proofreading device according to claim 2 or 3, wherein: 6. A recording medium storing a document proofreading program for executing a document proofreading process by a computer, wherein the document proofreading process program stores knowledge of an error candidate word group which is a group of words that are easily confused with each other. Based on the co-occurrence information, based on the statistical information of the co-occurrence information, the co-occurrence information is extracted based on the co-occurrence information. A recording medium storing a document proofreading processing program characterized in that correctness is determined in units of co-occurrence information.