JPH08221434A - Corpus preparing method - Google Patents

Abstract

PURPOSE: To attain the reconstruction of a corpus by providing processing time corresponding to the usage purpose of the corpus by imparting a keyword to each sentence in the corpus and extracting representative example sentence while using the including relation of the keyword. CONSTITUTION: A keyword number table is searched to calculate a sentence number IM of a sentence for which a representative sentence flag is '0' and the number of keywords is maximum (504). Then, a sentence S (IM) of an IM-th record in a large scale corpus is read (506) and written in a small scale corpus (507). Further, a representative sentence flag F (IM) of the keyword number table is changed into 1 (508). Besides, the keyword table is searched to successively read keywords K (IM, k) [0<<=m(IM)] of the snetence having the sentence number IM, and existing flag G(j) of the same keyword K(j) in the keyword table is turned to 1 (512). This processing is performed until all the existing flags of the keyword table are turned to 1 and all the keywords are covered.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for creating document data, and more particularly to a method for creating a small scale corpus from a large scale corpus.

[0002]

2. Description of the Related Art A corpus refers to accumulated text data for use in natural language processing and document processing. As a specific usage method, for example, a translation support system may be used to search and refer to an example sentence similar to the sentence to be translated from the corpus.

Conventionally, it is general to accumulate and store unsent sentences in the corpus, but the capacity may increase as the sentences are simply accumulated. In this case, a data compression technique is used to reduce the capacity.

[0004]

However, when the number of sentences reaches millions, data compression alone may not be able to obtain a practical processing time when accessing the corpus.

An object of the present invention is to reconstruct a corpus so that a practical processing time can be obtained according to the purpose of use of the corpus.

[0006]

[Means for Solving the Problems] During a large corpus,
Since it may contain several similar sentences,
The number of sentences can be reduced by deleting similar sentences. However, when calculating the degree of similarity by simple string matching, important words may be included in different parts of similar sentences, and valuable information may be reduced when the number of sentences is reduced. .

In order to achieve the above object, the present invention assigns a keyword to each sentence in a corpus and extracts a representative example sentence by using the inclusion relation of the keywords.

[0008]

[Function] Gives a keyword to each sentence in the corpus, preferentially selects a sentence with a large number of keywords, and selects a sentence so as to cover all or as many keywords as possible with the smallest number of sentences. To do.

[0009]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT FIG. 1 is a block diagram of the entire corpus creation process of the present invention. Keywords are added to the large-scale corpus 11 by the keyword addition processing unit 12, and the large-scale corpus 13 in which the keywords are entered is created. Furthermore, the representative sentence selection processing unit 14 selects a representative sentence, and the small-scale corpus 15 is selected.
Create The keyword assignment processing unit 12 is composed of a morphological analysis processing unit, a translated word association processing unit, a machine translation processing unit, and the like.

First, the keyword assignment processing section will be described.
There are several methods of assigning keywords depending on the purpose.

FIG. 6 shows the structure of a large-scale corpus. Each record in the large corpus has a sentence number 61
And a sentence 62.

FIG. 7 shows the structure of the morphological analysis result storage table. The morphological analysis result storage table includes word numbers 71, words 72, and parts of speech 73.

FIG. 8 shows the structure of a keyword table showing the correspondence between sentences and keywords. The keyword table includes sentence numbers 81 and keywords 82.

FIG. 9 shows the structure of a keyword number table showing the correspondence between sentences and keyword numbers.
The keyword number table has a sentence number 91 and a keyword number 9
2. It consists of a representative sentence flag 93. Representative sentence flag 9
In 3, the initial value 0 is described.

FIG. 10 shows the structure of the keyword table 2 displaying the sentence numbers in which the keywords appear. The keyword table 2 includes a keyword 101, a sentence number 102, and a flag 103 that has already appeared. Sentence number 102
In, the numbers of the sentences in which the keywords appear are described separated by arbitrary commas. An initial value 0 is described in the already-existing flag 103.

The first method of keyword extraction is to use a predetermined part of speech H (k) (0≤k≤n3) as a keyword. This method will be described with reference to the flowchart shown in FIG.

First, the sentence S (i) (0≤i≤n1) of each record in the large-scale corpus is read in order (202),
Morphological analysis is performed (203). Since a known technique may be used for the method of morphological analysis, detailed description thereof will be omitted. The morphological analysis result is stored in the morphological analysis result table as
Described as a word number 71, a word 72, and a part of speech 73. It should be noted that the word 72 describes an end form in the case of an inflection word. The morphological analysis result table is searched in the order of word numbers j (0 ≦ i ≦ n2), and it is checked whether the part of speech I (i, j) is equal to the part of speech H (k) (206). The word W (i, j) of j is registered in the keyword table and the keyword table 2 (207). After processing all words, the number of keywords in each sentence is calculated (210) and registered in the keyword number column 92 of the keyword number table (212).

The second method of assigning a keyword is to use a predetermined word X (k) (0 <i≤n3) as a keyword. In this case, the part-of-speech matching process of FIG. 2 is replaced with a word matching process. That is, step 206 may be replaced with W (i, j) = X (k).

When the large-scale corpus is a bilingual corpus, a third method of assigning a keyword is to use a portion where the bilingual relationship cannot be correctly identified as a keyword.
The fact that the bilingual relationship cannot be specified can be regarded as a difficult part of translation that cannot be simply translated, and is effective as a search key in the translation support system. This processing method will be described with reference to the flowchart shown in FIG.

FIG. 11 shows the structure of the bilingual corpus. Each record in the bilingual corpus has a sentence number 110.
1 and an original sentence 1102 and a translated sentence 1103.

First, the original sentence S (i) and the translated sentence T (i) of each sentence of the bilingual corpus are read in order (302), and the words having a bilingual relationship are associated with each other while referring to the translation dictionary (303).
For this method, for example, the technique described in Japanese Patent Application No. 3-315981 can be used. This technique, when the same phrase appears twice in a sentence, when it is not possible to make a correspondence only by the parallel translation of a dictionary,
Correct correspondence can be made using syntactic information. As a result, the word associated with the parallel translation word is described as the parallel translation word I (i, j) in the column 73 of the morphological analysis result table. Next, the morphological analysis result table is searched in the order of the word number j to check whether there is a bilingual word (30
5), if no parallel word is described, word number j
Is registered in the keyword table and the keyword table 2 with the word W (i, j) of (30
6). After processing all words, the number of keywords in each sentence is calculated (309) and registered in the keyword number column 92 of the keyword number table (310).

The large-scale corpus is a bilingual corpus,
In addition, when the machine translation system can be used, as a fourth method of assigning a keyword, a portion having an incorrect translation result can be used as a keyword. An incorrect translation result can be regarded as a difficult part of translation and is effective as a search key in the translation support system. It is also highly useful as an example sentence for evaluation of machine translation systems. This processing method will be described with reference to the flowchart shown in FIG.

FIG. 12 shows the structure of the translation result table. The translation result table is the original word number 12
01, an original sentence word 1202, and a translated word 1203. First, the original sentence S (i) in the large-scale corpus is machine translated (402), and the result is written in the translation result table (403). This machine translation result is compared with the translated sentence T (i) in the bilingual corpus, and each translated word E (i,
It is checked whether (j) appears in the translated text T (i) (405), and if it does not appear, the word W (i, j) in the original text to which the word corresponds is extracted from the translation result table, and the keyword table And it is registered in the keyword table 2 (406). After processing all words, the number of keywords in each sentence is calculated (409) and registered in the keyword number column 92 of the keyword number table (4).
10).

Next, a method of selecting a representative sentence will be described. Here, the representative sentence is selected so as to cover as many keywords as possible with the smallest number of sentences. A specific processing method will be described with reference to the flowchart shown in FIG.

FIG. 13 shows the structure of a small corpus that stores only representative sentences. This small-scale corpus is composed of a representative sentence number 1301, a sentence 1302, and a large-scale corpus sentence number 1303.

First, the keyword number table is searched to obtain the sentence number IM of the sentence in which the representative sentence flag is 0 and the number of keywords is maximum (504). The sentence S (IM) of the IMth record of the large-scale corpus is read (506) and written in the small-scale corpus (507). Further, the representative sentence flag F (IM) of the keyword number table
Is changed to 1 (508). Also, by searching the keyword table, the keyword K (IM,
k) (0 <k ≦ m (IM)) is sequentially read, and the already-existing flag G (j) of the same keyword K (j) in the keyword table 2 is set to 1 (512). The above processing is performed until all the existing flags in the keyword table 2 become 1 and all the keywords are covered.

If the number of registered sentences in the small corpus has an upper limit, a method of terminating the process when the number of registered sentences reaches the upper limit can be considered. In this case, step 514 of the representative sentence selection process may be changed to "whether the number of representative sentence registration sentences has reached the upper limit value".

The processing example of the present invention will be described below by using an actual example.

FIG. 14 shows an example of a large-scale corpus. The large corpus has sentence numbers 1401 and 14
It consists of 02.

FIG. 15 shows an example of a morphological analysis result storage table. This table uses word number 15
01, a word 1502, and a part of speech 1503.

FIG. 16 shows an example of the keyword table. This table includes sentence numbers 1601 and keywords 1602.

FIG. 17 shows an example of the keyword number table. This table includes a sentence number 1701, a number of keywords 1702, and a representative sentence flag 1703.

FIG. 18 shows an example of the keyword table 2. This table contains keywords 1801
And a sentence number 1802 and an already-existing flag 1803.

FIG. 19 shows an example of a bilingual corpus. The bilingual corpus consists of sentence number 1901 and bilingual sentence 19.
It consists of 02.

FIG. 20 shows an example of the translation result table. This table is composed of original text word numbers 2001, original text words 2002, and translated words 2003.

FIG. 21 shows an example of a small corpus. For small corpus, sentence number 2101 and sentence 21
02, the sentence number 2103 of the large-scale corpus.

When processing is performed on a large-scale corpus as shown in FIG. 14, the morphological analysis result of the fourth sentence is as shown in FIG. If the keyword is a verb, the keyword table showing the keyword for each sentence is as shown in FIG. 16, and the keyword table 2 showing the sentence number that appears for each keyword is as shown in FIG. The keyword number table is as shown in FIG. Here, the first sentence S (1) and the second sentence S
Since the keyword of (2) matches and the keyword of the fourth sentence S (4) is included in the keyword of the third sentence S (3), three sentences 1, 3, 5 are selected as the representative sentence. Will be done. As a result, a small-scale corpus as shown in FIG. 21 can be created.

As another example, consider the case where the Japanese-English parallel corpus shown in FIG. 19 is used. For example, 2
The Japanese-English translation result table for the second sentence is as shown in FIG. Here, since the eighth word'wait 'does not appear in the translated sentence of the bilingual corpus, the Japanese word "wait" corresponding to'wait' is a keyword.

[0039]

According to the present invention, from a large corpus,
A small corpus can be created by extracting only sentences that match the purpose of use, and, for example, it is possible to improve search efficiency. Further, when a keyword is extracted using a parallel translation corpus, a word that is difficult to machine translate becomes a keyword, and therefore it can be used as a small-scale corpus for machine translation evaluation.

[Brief description of drawings]

FIG. 1 is a block diagram of corpus creation processing according to the present invention.

FIG. 2 is a flowchart of a first method of processing for extracting a keyword.

FIG. 3 is a flowchart of a third method of processing for extracting a keyword.

FIG. 4 is a flowchart of a fourth method of processing for extracting a keyword.

FIG. 5 is a flowchart of processing for selecting a representative sentence.

FIG. 6 is an explanatory diagram of a large-scale corpus.

FIG. 7 is an explanatory diagram of a morphological analysis result storage table.

FIG. 8 is an explanatory diagram of a keyword table.

FIG. 9 is an explanatory diagram of a keyword count table.

FIG. 10 is an explanatory diagram of a keyword table 2.

FIG. 11 is an explanatory diagram of a bilingual corpus.

FIG. 12 is an explanatory diagram of a translation result table.

FIG. 13 is an explanatory diagram of a small corpus.

FIG. 14 is an explanatory diagram of a large scale corpus.

FIG. 15 is an explanatory diagram of a morphological analysis result storage table.

FIG. 16 is an explanatory diagram of a keyword table.

FIG. 17 is an explanatory diagram of a keyword count table.

FIG. 18 is an explanatory diagram of a keyword table 2.

FIG. 19 is an explanatory diagram of a bilingual corpus.

FIG. 20 is an explanatory diagram of a translation result table.

FIG. 21 is an explanatory diagram of a small corpus.

[Explanation of symbols]

11, 13 ... Large-scale corpus, 12 ... Keyword assignment processing, 14 ... Representative sentence selection processing, 15 ... Small-scale corpus.

Claims (6)

[Claims] 1. A step of assigning a keyword to each sentence in a corpus and a step of calculating a keyword set of each sentence are provided, a sentence in the corpus is selected, and a small-scale corpus is created. Characteristic corpus creation method. 2. The corpus creating method according to claim 1, wherein the sentences are selected so that the union of the keyword sets of each sentence covers all the keywords. 3. The corpus creating method according to claim 1, wherein sentences are selected in order from a sentence having a large number of keywords. 4. The corpus creating method according to claim 1, wherein when a keyword is added to the corpus, a predetermined word or part of speech is used as a keyword candidate. 5. The corpus creation method according to claim 1, wherein, when the corpus is a parallel translation corpus, when a keyword is added, a word having no correspondence between an original sentence and a translated sentence is used as the keyword. 6. The corpus creation method according to claim 1, wherein, when the corpus is a parallel translation corpus, when a keyword is added, a word at a different portion between a machine translation result of an original sentence and a translated sentence is used as a keyword.