JP2520195B2 - Japanese sentence proper term extraction device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION [Industrial Application Field] The present invention automatically extracts proper nouns such as product names, company names, and personal names used in Japanese documents from the documents, and further The present invention relates to a Japanese sentence proper term extraction device that automatically extracts a new word or a word that is considered to be used only in the document (including proper nouns are called proper terms) even if it is a combination of common words. is there.

[Conventional technology]

Techniques for extracting unique terms include Japanese Patent Application Nos. 62-238385 and 62-197843 by the same applicant. In this method, the characters used in Japanese are categorized into about 10 types of characters, such as "Kanji numbers", "Kanji", "Hiragana", "Katakana", and they are called "delimiter" and "delimiter". Whether the words sandwiched by or the words sandwiched between "Hiragana" and "Hiragana" are extracted as unique term candidates, narrowed down by applying the candidate narrowing rule regarding the combination of character types, and whether they are registered in the general word dictionary. The remaining words were extracted as proper terms by narrowing down by.

[Problems to be Solved by the Invention]

In the conventional technology, it was possible to extract proper terms consisting of Kanji, Katakana, English letters, etc. However, regarding proper terms when Hiragana is mixed such as "handwritten characters" and "built-in functions", There was a drawback that it could not be extracted.

An object of the present invention is to overcome the above-mentioned problems and to automatically extract proper terms mixed with Hiragana by utilizing the structural features of proper terms mixed with Hiragana.

[Means for solving the problem]

The present invention includes kanji, katakana, and English characters extracted from the target Japanese documents by the methods of Japanese Patent Application Nos. 62-197843 and 62-238385 among the proper terms included in Japanese documents. Or, it consists of a combination of them, and targets a character string that is obtained by dividing the target document before and after a word recognized as a proper term, and many of the proper terms mixed with hiragana have other nouns added to the morphological endings of verbs. By utilizing the fact that it is configured as
A hiragana that can be a verb conjunctive ending is differentiated from other hiragana characters to a different character type, the character string of the target document is expanded into a character type code string, and a "verb conjunctive form + inflection ending" is constructed from it. The unique terms that are elements are extracted as unique term candidates, and those candidates are narrowed down in the first stage by narrowing down rules regarding the length of character strings and combinations of character types, and the unique term candidates narrowed down in the first stage. On the other hand, by referring to the common word dictionary, the second stage of narrowing down the words already registered in the common word dictionary from the candidates is performed.

[Work]

Unique terms that are composed of "verb verbs + inflectional endings" are extracted as proper term candidates, and the candidates are narrowed down in the above two stages to extract proper terms mixed with hiragana.

〔Example〕

FIG. 1 is a basic configuration diagram showing one embodiment of the present invention.
Reference numeral 1 is a Japanese document, 2 is a proper term list “1”, and 11 is a proper term list “2”. Reference numeral 12 is a Japanese sentence proper term extraction device main body, which includes a character string separating section 3, a Hiragana-blended proper term candidate extraction section 4, a character type code table 5, a Hiragana-blended proper term candidate narrowing section A6, a narrowing rule table 7,
Hiragana mixed proper term candidate narrowing down section B8, common language dictionary 9,
It is composed of a proper term output unit 10.

The Japanese document 1 is an input document to the Japanese sentence proper term extraction device main body 12, and the proper term list "1" 2 is a hiragana name previously extracted from the Japanese document 1 by the conventional Japanese document proper term extraction device. This is a list of unique terms that are not included.

The character string separating unit 3 searches for a proper term “1” 2 with the Japanese document 1 as an input, deletes the proper term in the input sentence, and separates the character strings before and after the proper term. Create this list as a separate string.

The Hiragana-mixed proper term candidate extraction unit 4 inputs the separated character string list created by the character string separation unit 3 and converts it into a code string using the character type code table 5,
Using this code string as an input, proper term candidates with Hiragana mixed are extracted to create a proper term candidate list with Hiragana mixed.

The character type code table 5 is a table that separates characters used in a Japanese document into a plurality of character types and shows the character type codes corresponding to each.

The Hiragana-mixed proper term candidate narrowing down unit A6 applies the rules of the narrowing rule table 7 with the Hiragana-blended proper term candidate extraction unit 4 created by the Hiragana-mixed proper term candidate list as input and applies the rule that meets the selection condition to the original one. Convert to a string.

The narrowing-down rule table 7 is a table describing rules for narrowing down candidates created based on information about a character string length, arrangement of character type codes, and the like.

The Hiragana-mixed proper term candidate narrowing down unit B8 narrows down the Hiragana-mixed proper term candidate narrowing down unit A6 and narrows down the general character dictionary 9 for the Hiragana-mixed proper term candidates converted into the original character strings.

The general-word dictionary 9 is a general-word dictionary that describes the characters, readings, parts of speech, etc. of general Japanese words.

The proper term output unit 10 accumulates the results narrowed down by the proper term candidate narrowing unit B8 mixed with Hiragana and outputs the proper term list “2” to the output device.

The specific term list “2” 11 is a list of the output specific terms.

FIG. 2 shows a schematic flow chart of an operation example of the Japanese sentence proper term extraction device. The operation of FIG. 1 will be described below with reference to these figures.

Step S1-1: Input an arbitrary Japanese document.

Step S1-2: Set n = 1 to process the first sentence included in the input Japanese document (hereinafter referred to as input document).

Step S1-3: The processes up to step S1-7 are repeated until all the input documents are processed.

Step S1-4: The character string separating section 3 sequentially searches the proper term list “1” for the purpose of retrieving the proper term contained in the nth sentence.

Step S1-5: Further, in the character string separating unit 3, if a character string matching the unique term retrieved in step S1-4 exists in the nth sentence, it is deleted from the sentence. Furthermore, the character strings before and after that are separated,
Let each be a separate string.

Step S1-6: Set n = n + 1 to process the next sentence.

Step S1-7: The processing is divided depending on whether the processing for all the sentences in the input document has been completed. If all the processes have not been completed, the process returns to step S1-3, and if they have been completed, the process proceeds to step S1-8.

Step S1-8: A list of separated character strings separated and accumulated by the following processing is created.

Step S2-1: Input the list of separated character strings created in step S1-8.

Step S2-2: Set i = 1 to process the first character string included in the input separated character string list (hereinafter, input separated character string).

Step S2-3: Step S2-7 until all input separated character strings are processed
Repeat the process up to.

Step S2-4: In the Hiragana-mixed proper term candidate extraction unit 4, one character of the i-th character string is converted into a plurality of types of codes based on the character type according to the description of the character type code table 5 (for example, 7 shown in FIG. 7). Type code) to generate a code string for the i-th character string.

Step S2-5: Further, in the Hiragana-mixed proper term candidate extracting section 4, the i-th
Regarding the character strings, all of the combinations of the character strings of A, B, C, D, and E that are separated or enclosed by the character type J or the character type F, including the character code B, are all unique terms mixed with hiragana. Extract as a candidate.

Step S2-6: Set i = i + 1 to process the next character string.

Step S2-7: The processing is divided depending on whether the processing for all the character strings in the input character string is completed.

If all processing has not been completed, step S2-
Returning to step 3, if completed, proceed to step S2-8.

Step S2-8: A list of proper term candidates mixed with Hiragana extracted in the above process is created.

Step S3-1: Input the list of proper term candidates mixed with hiragana created in step S2-8.

Step S3-2: In order to process the first proper term candidate character string included in the input proper term candidate list with mixed hiragana, set j = 1.

Step S3-3: The processes up to step S3-7 are repeated until all the input unique term candidate character strings are processed.

Step S3-4: In the proper term candidate narrowing down section A mixed with hiragana, the rule of the narrowing rule table 7 is applied to the first proper term candidate character string to perform a selection process, and it is determined whether or not it is a proper term candidate.

Step S3-5: Further, in the proper term candidate narrowing section A mixed with hiragana,
The processing is divided depending on whether or not it is selected as a proper term candidate by applying the narrowing rule.

If it is selected, the process proceeds to step S3-6, and if it is not selected, the process proceeds to step S3-8.

Step S3-6: In the Hiragana-mixed proper term candidate narrowing down section B, the character type code of the proper term candidate character string selected in the Hiragana mixed proper term candidate narrowing down section A is replaced with the original character.

Step S3-7: Further, in the proper term candidate narrowing section B mixed with hiragana,
Whether or not the proper word candidates replaced with the original characters in step S3-6 are common words is narrowed down by referring to the common word dictionary 9.

Step S3-8: Set j = i + 1 to process the next proper term candidate.

Step S3-9: The processing is divided depending on whether the processing has been completed for all the unique term candidates in the input unique term candidate list.

If all the processing has not been completed, step S3−
Return to No. 3, and if completed, proceed to step S3-10.

Step S3-10: The proper term candidate selected in step S3-8 is output to the proper term output unit 10 as a final result.

Step S3-11: A unique term list “2” is created from the unique terms of the final result.

Next, the outline of the operation will be described using a specific example of the input document. The input sentence example shown in FIG. 3 will be described as a Japanese document 1.

First, in the character string separating unit 3, as shown in FIG. 4, the unique term list “1” 2 which is a list of already-extracted unique terms that does not include hiragana is examined, and the unique terms in the list are included in the input sentence example. If so, delete it as shown in FIG. Furthermore, as shown in FIG. 6, the sequence of characters before and after the deleted proper term is made into an independent character string,
Create a separated string list.

Next, the Hiragana-mixed proper term candidate extraction unit 4 sequentially processes each character string in the separated character string list created by the character string separation unit 3. First, for the first separated character string “one side, each”, the character separation code table 5 in which the Japanese characters shown in FIG. 7 are classified into seven types is used, and the encoded separated character string shown in FIG. 8 is used. The first coded character string "AAF
Convert to "A". The second separated character string "the movement of the appropriate one" is converted into "JABJFAAJAJJ" by the same operation. Similarly, the coded separated character string list shown in FIG. 8 is obtained.

Next, for this coded character string, all of the combinations of the character type codes A, B, C, D, E extracted or enclosed by the character type code F and the character type code J and including B Candidates for unique terms mixed with hiragana.

First, regarding the first coded character string "AAFA", the coded character strings "AA" and "A" are separated with the character type code F as a boundary, but these are not candidates because they do not include the character type code B. .

Next, for the second coded character string "JABJFAAJAJJ", "AB", "AA", and "A" are extracted in the same manner. Of these, "AB" including the character type code B is used as a proper term candidate mixed with hiragana. To do.

By repeating the process in the same manner, a unique term candidate coded character string list including hiragana as shown in FIG. 9 is obtained.

Next, in the proper term candidate narrowing section A6 mixed with Hiragana,
The narrowing-down rule is applied to each element of the Hiragana-mixed proper term candidate coded character string list created by the Hiragana-blended proper term candidate extraction unit 4 to perform the narrowing process.

First, the first rule in the narrowing rule table shown in FIG. 10 is applied to the first ruled character string “AB”, and it is dropped from the candidates. Next, the second coded character string "AA
The fourth rule is applied to "B" and it is dropped from the candidates.

In the same way, the rule application result as shown in Fig. 11 is obtained. As a result, "motion frame" and "motion amount"
And are extracted as proper term candidates mixed with hiragana.

Next, in the Hiragana-mixed proper term candidate narrowing section B8,
The general word dictionary 9 is searched for the proper term candidates created by the Hiragana-mixed proper term candidate narrowing unit A, and if there is a dictionary, the proper term candidates are dropped, and the remaining proper term candidates are extracted as proper terms mixed with the Hiragana. In this example, both the unique term candidates “motion frame” and “motion amount” are not in the general word dictionary and are extracted as unique terms mixed with hiragana.

As described above, the two-step extraction method using the unique term list “1”, that is, the expansion method to the unique character type code string and the narrowing rule of the code string arrangement, which pays attention to the continuous usage inflected hiragana By narrowing down the candidates by applying them, and narrowing down the candidates by referring to the general language dictionary, "motion frame", which is a proper term mixed with hiragana,
The “movement amount” can be extracted.

〔The invention's effect〕

According to the present invention, it is possible to extract unique terms mixed with hiragana, for example, a process of extracting a term unique to a document from a document to be translated to create a bilingual document or a Japanese document such as a paper or a literary work. It can be expected that the efficiency of creation processing such as glossary and index creation will be greatly improved.

For example, regarding the creation of a Japanese-English bilingual collection for Japanese-English translation,
Conventionally, before translating a large amount of documents to be translated, technical terms and special terms used in the documents are manually extracted from the documents and the translated words are given to them to create a bilingual collection. However, by using the present invention, the term specific to the document is extracted at high speed from the document to be translated in the preprocessing of translation without performing analysis processing such as morphological analysis and dependency analysis. It becomes possible to do. In addition, by performing morphological analysis on unknown words, it is possible to extract proper terms that cause analysis failure or fail to be extracted. In the present invention, it is expected that the quality of translation will be remarkably improved by registering and using a glossary in which a translation is given to the extracted unique term in the user dictionary.

With regard to automatic index creation, conventionally, when creating an index, the author manually picks out terms that seem to be important from the manuscript he wrote and sorts the extracted terms to create an index manually. I was going through the procedure. By using the present invention, the writer can automatically extract index candidate terms by applying an electronic Japanese document to this device. The author can extract the terms used for the index by deleting some unnecessary candidates from the candidates presented by this device. The computerized candidate terms can be easily sorted in order of AIUEO, and a high-speed index can be created.

[Brief description of drawings]

FIG. 1 is a basic configuration diagram showing an embodiment of the present invention, FIG. 2 is a schematic flow chart of the operation of the present invention, FIG. 3 is an example of a Japanese document which is an input of the present invention, and FIG. List of extracted unique terms that do not include hiragana, Fig. 5 is an example of deleting previously extracted unique terms from the input document, 6th
The figure shows a list of separated strings created by separating the sequence of characters before and after the deleted proper term.
Example of contents of character type code table divided into types, FIG. 8 is a list of separated character strings converted into codes using the character type code table, and FIG. 9 is separated by character type codes F and J,
Or, a list of only enclosed A, B, C, D, E character strings that include B. Figure 10 is a narrowing that is applied to the coded character strings that remain as candidates. An example of the contents of the rule table, Fig. 11 shows the final candidate list and the result of applying the narrowing rules. 1 …… Japanese document, 2 …… Unique term list “1”, 3 …… Character string separation section, 4 …… Unique term candidate extraction section with Hiragana mixed, 5…
… Character type code table, 6 …… Hiragana mixed unique term candidate narrowing down section A, 7 …… Narrowing rule table, 8 …… Hiragana mixed unique term candidate narrowing down section B, 9 …… General language dictionary, 10
…… Unique term output part, 11 …… Unique term list “2”, 12…
… Japanese sentence proper term extraction device.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (56) References Japanese Patent Laid-Open No. 1-266670 (JP, A) Oku, Higashida “Automatic Extraction Method of Proper Terms in Japanese Documents” IPSJ 41st Annual Conference Shu Vol. 3, No. 3J-5 (1990-09-04 to 06) PP. 3-227 ~ 228

Claims (1)

(57) [Claims] 1. A Japanese document written in Japanese and a unique term list "1" that does not include hiragana contained in the document are input, and the input Japanese document and unique term list "1" are input. Is used to separate the above-mentioned proper term and the character string sandwiched between proper terms.
A character string separation unit to be extracted, a character type code table in which characters used in Japanese documents are classified into a plurality of specific character types, and each of the character string separation units separated using the character type code table. Using the Hiragana-blended proper term candidate extraction unit that extracts Hiragana-blended proper term candidates for a character string and the Hiragana-blended proper term candidates, only those that satisfy the selection conditions for narrowing are extracted from the Hiragana-blended proper term candidates. Select Hiragana mixed specific term candidates A
And a candidate term narrowed down by the Hiragana mixed proper term candidate narrowing down section A is compared with a common word dictionary to delete words registered in the common word dictionary and further narrow down the candidates. B, and a Hiragana-mixed proper term output section that accumulates the narrowed results and outputs a proper term list “2” to the output device, and extracts the Hiragana-blended proper term. Extractor.