JPH0528871B2 - - Google Patents

Description

[Detailed Description of the Invention] [Field of Industrial Application] This invention is applicable to Japanese documents, even if they are proper nouns or combinations of words such as product names, company names, or people's names that are only used in the document. This invention relates to a Japanese target sentence-specific term extraction device that automatically extracts new words and words that are considered to be used only in the document (target sentence-specific terms).

[Conventional technology]

Even if a Japanese document is a combination of proper nouns or words such as product names, company names, or people's names that are used only in that document, new words or words that are considered to be used only in that document (target sentence) In the conventional method for extracting unique terms), the input Japanese sentence is separated using a pre-prepared analysis dictionary, and as a result, the portions that cannot be properly separated are extracted as terms specific to the target sentence. In addition, in the method that focuses on the points of change in character type, the points at which one character type changes to another are extracted as candidates for target sentence-specific terms, and from among these candidates, Japanese words prepared in advance for analysis are selected. Candidates not registered in the dictionary are extracted as target sentence-specific terms.

[Problem that the invention seeks to solve]

However, the former method had the following problem.

Extraction takes time because input Japanese sentences are separated using an analysis dictionary.

Japanese separation processing assumes that the words are registered in the analysis dictionary, so it also attempts to analyze the target sentence-specific terms, and if it is partially successful in parsing, that part is used as the target sentence-specific term. (Example: Although "○× company" is a term specific to the target sentence that represents the company name as a whole, "company" is a general word and the analysis is successful, so only "○×" is used. is recognized as a target sentence-specific term), making it impossible to accurately extract target sentence-specific terms.

If a target sentence-specific term is composed of general words, the term is not recognized as a target sentence-specific term. An example is shown below. "Nippon Electric Power and Telephone Corporation" is a proper noun and a term specific to the target sentence, but since it is composed of only general words, it is separated into "Japan/Electricity/Telephone/Stocks/Company". succeed in Therefore, "Nippon Electric Power Telephone Co., Ltd.", which should be extracted as a target sentence-specific term, is not extracted.

In addition, although the latter method is faster because it requires fewer dictionary lookups, it is difficult to extract target sentence-specific terms that span multiple character types, and it is difficult to extract target sentence-specific terms that include affixes with okurikana such as ``for ~''. There were problems such as the inability to extract.

An object of the present invention is to provide a Japanese target sentence-specific term extraction method for automatically extracting target sentence-specific terms from a Japanese document at high speed and accurately, which solves the above-mentioned problems.

[Means for solving problems]

This invention provides a means for developing a character string in a Japanese document into a code string classified into nine types of characters, and a means for developing a character string in a Japanese document into a code string classified into nine types of characters.
An extraction means for extracting candidates for target sentence-specific terms, and from among the candidates extracted by the extraction means, conditions for character strings that do not become Japanese target sentence-specific terms due to the nature of Japanese, and conditions for character strings that become Japanese target sentence-specific terms. By using a language information table that stores conditions for character strings that are easy to find, and retaining only candidates that meet the conditions for character strings that are likely to become Japanese target sentence-specific terms, more accurate candidates for Japanese target sentence-specific terms can be created. The most important feature is to have a processing means for extracting words, and a selection means for outputting only words that are not included in the Japanese analysis dictionary from among the candidates extracted by the processing means as terms specific to the Japanese target sentence. Features.

What is the conventional technology? When extracting candidates for terms specific to the Japanese target sentence, the extraction means uses a high-speed method to extract candidates by focusing on changing points of character types, without performing analysis using an analytical dictionary such as separation. The extraction method extracts candidates by focusing on the change points of character types, so words that consist only of common words or words that partially contain common words can be used as candidates for terms specific to the Japanese target sentence. The processing means that can extract hyuristic rules such as ``A string of five or more kanji characters is likely to be a term specific to a Japanese target sentence'' or ``If an affix is included in the candidates, perform processing on the affix.'' Linguistic information consisting of processing information such as "Shikosu" is used to narrow down candidates for terms specific to the Japanese target sentence, so accurate candidate extraction is possible. All Japanese target sentence-specific terms are output as Japanese target sentence-specific terms, so Japanese target sentence-specific terms that consist only of common words and Japanese target sentence-specific terms that include some common words are also output as Japanese target sentence-specific terms. Each point of , which can be extracted as , is different.

〔Example〕

FIG. 1 is a basic configuration diagram when this invention system is configured by hardware, and the code string expansion unit 1 converts the input Japanese document into nine types of characters (Kanji code, Kanji code, Expand into code strings classified into Hiragana code, Katakana code, Arabic numeral code, Alphabet code, punctuation mark code, delimiter code, and other codes). The Japanese target sentence-specific term candidate extraction unit 2 extracts candidates for Japanese target sentence-specific terms from the code string obtained by the code string expansion unit 1, focusing on points of change in character types. The Japanese target sentence specific term language processing section 3 generates a linguistic information table 8 for each candidate among the candidates obtained by the Japanese target sentence specific term candidate extraction section 2.
After searching for and processing the candidates according to that information, a candidate group consisting only of more accurate candidates is extracted. The Japanese target sentence specific term selection unit 4 searches the Japanese language analysis dictionary 9 using the character face of each candidate in the candidate group obtained in the Japanese language processing unit 3 as a key, and searches the Japanese language analysis dictionary 9. Only words that are not registered in are selected as terms unique to the Japanese target sentence. The Japanese target sentence specific term registration unit 5 registers the Japanese target sentence specific term selected by the Japanese target sentence specific term selection unit 4 into the Japanese target sentence specific term file 6. The Japanese target sentence specific term file 6 is a file in which the finally extracted Japanese target sentence specific terms are registered. The classification table 7 is a table that specifies how to classify the character type string extracted by the Japanese target sentence specific term candidate extraction unit 2, and the linguistic information table 8 is a table that defines how to classify character type strings extracted by the Japanese target sentence specific term candidate extraction unit 2. , is a table that describes linguistic information and processing methods used when extracting more accurate candidates, and the Japanese language analysis dictionary 9 registers the fonts, parts of speech, etc. of common Japanese words.
The code string expansion section 1, the Japanese target sentence-specific term candidate extraction section 2, the Japanese target sentence-specific term language processing section 3, the Japanese target sentence-specific term selection section 4, and the Japanese target sentence-specific term registration section 5 are computing devices. A Japanese target sentence specific term extraction device 10 is comprised of a memory and a memory.

FIG. 2 is a schematic flowchart of the operation of the Japanese target sentence specific term extraction device 10. Next, the operation will be explained according to the schematic flowchart shown in FIG. First, set n to 1 (step S ₁ ) Japanese target sentence specific term extraction device 1
Processes the nth sentence of the Japanese document, which is the input of 0 (step S ₂ ), and the code string expansion unit 1 processes the nth sentence
Each character in a Japanese document is divided into 9 types of characters (Kanji code, Kanji code, Hiragana code,
katakana code, Arabic numeral code, alphanumeric code, punctuation mark code, delimiter code, and other codes), and generates a code string for the Japanese document (step _S3 ).
At this time, each code is given information indicating which character of the Japanese document the code was generated from. The code string generated by the code string expansion section 1 is the Japanese target sentence-specific term candidate extraction section 2.
sent to.

The Japanese target sentence-specific term candidate extraction unit 2 starts at a change point where a delimiter code, hiragana code, or punctuation mark code changes to one of the other six types of codes, and then extracts a delimiter code, hiragana code, or punctuation mark code. Extract all original character strings corresponding to code strings that end at a change point that changes to one of the following as candidates for Japanese target sentence-specific terms, and classify the candidates according to the conditions described in classification table 7. (Step S ₄ ). FIG. 3 shows an example of the contents of the classification table 7. Here, "kanji" also includes "kanji numerals".

The classified Japanese target sentence specific term candidates are sent to the Japanese target sentence specific term language processing unit 3. The Japanese target sentence specific term language processing unit 3 sets i=1 (step S ₅ ) and processes the i-th candidate (step S ₆ ), that is, first searches the linguistic information table 8 to obtain information (step S 5 ). _S7 ). Next, each candidate for Japanese target sentence-specific terms is processed according to the obtained information (Step S ₈ ), thereby extracting more accurate Japanese target sentence-specific term candidates (Step S ₉ ). , sends the candidates to the Japanese target sentence specific term selection unit 4.

The Japanese target sentence specific term selection section 4 searches the Japanese language analysis dictionary 9 using the font of the candidate Japanese target sentence specific term sent from the Japanese target sentence specific term language processing section 3 as a key. As a result of the search, if the candidate is registered in the Japanese analysis dictionary 9,
The candidate is rejected from the list of candidates as it is not a term unique to the Japanese target sentence (Step S ₁₀ ). On the other hand, if the candidate is not registered in the Japanese language analysis dictionary 9, the candidate is determined to be a term unique to the Japanese target sentence and is sent to the Japanese target sentence unique term registration section 5 (step _S10 ).

The Japanese target sentence-specific term registration unit 5 stores the Japanese target sentence-specific terms sent from the Japanese target sentence-specific term selection unit 4 into the Japanese target sentence-specific term file 6.
Register by writing (Step _S11 ). Step S ₉
If you do not leave that candidate as a candidate for the Japanese target sentence-specific term, or after step S ₁₁ , select i+1.
is set to i (step S ₁₂ ), and then it is checked whether there are any more candidates. If there are still candidates, the process returns to step S _6. If there are no candidates, n is +1 (step S 12).
_S14 ) and checks whether there are any more sentences (step _S15 ). If there are still sentences, the process returns to step _S2 ; if there are no sentences, the process ends.

Next, an outline of the operation will be explained using an example. The example sentence shown in FIG. 4 will be explained as a Japanese document that is input to the target sentence specific term extraction device 10. First, the code string expansion unit 1 converts each character of the example sentence shown in FIG. 4 into a corresponding code (any of nine types of codes) to generate a code string (FIG. 5). The code string in Figure 5 shows the kanji codes (all kanji except kanji numerals), the kanji numeric codes (〇, 1, 2, 3, 4,
5, 6, 7, 8, 9), hiragana code (all hiragana), katakana code (all katakana), Arabic numeral code (0, 1,
2, 3, 4, 5, 6, 7, 8, 9), alpha alphabet codes (uppercase and lowercase letters of A to Z, a to z), punctuation mark codes (comma marks ",", period marks ".",
comma “,”, period “.”, question mark “?”, and exclamation mark “!”).
Separator codes (key cutlets such as ``'', round cutlets (), large cutlets [], medium cutlets {}, quotation mark ``''', double quotation mark ``''), and other codes (as mentioned above) Characters and symbols that do not fit into any code are abbreviated as .

At this time, each code has information indicating which character it was generated from (for example, the first one in Figure 5 is generated from the character ``base'' in Figure 4, the next one is generated from the character ``board'') (information indicating that). There are several ways to assign this, such as adding a number to the original character string and assigning that number to the code string, or having the original character and code as a pair.
This method is not discussed here. Code string generated by code string expansion unit 1 (Figure 5)
is sent to the Japanese target sentence-specific usage error candidate extraction unit 2.

Next, the Japanese target sentence-specific term candidate extraction unit 2
The code sequence shown in Figure 5 starts with a change point that changes from one of the delimiter code, hiragana code, or punctuation mark code to any of the other six types of codes, and then changes to either the delimiter code, hiragana code, or punctuation mark code. A code string that ends at a point where the code changes to `` is extracted'' (Fig. 6), and all original character strings corresponding to that code string are extracted as candidates for terms specific to the Japanese target sentence. From the example sentences shown in Figure 4,
As shown in FIG. 7, 21 candidates are extracted. Assuming that there is a change point at the beginning of the sentence, the change points up to the next change to either a delimiter code, a hiragana code, or a punctuation mark code are extracted as candidates. Furthermore, the Japanese target sentence-specific term candidate extracting unit 2 classifies the extracted candidates according to the information on the change points of character types according to the classification table 7. The results are shown in FIG. The classification results are sent to the Japanese target sentence specific term language processing section 3.

The Japanese target sentence specific term language processing section 3 searches the language information table 8 and narrows down the candidates according to the obtained information. An example of the contents of the language information table 8 is shown in FIG. The Japanese target sentence specific term language processing unit 3 searches the language information table 8 using the candidate classification shown in FIG. 8 as a key. Among the words belonging to Category 2 in Figure 8, ``last year'' matches the term ``Nouns used as adverbs are excluded from the candidates'' in the section of Category 2 in Figure 9, so it is dropped from the candidates. ”
is dropped from the list because it matches ``those with a Chinese numeral and a particle following it are dropped from the list of candidates'', and ``sale'' matches ``those with a kanji numeral followed by a particle are dropped from the list of candidates'', and ``sample nouns are dropped from the list of candidates''. Therefore, among the words belonging to category 2, three words remain as candidates: "Nishiden,""Nanshiba," and "Company." All words belonging to categories 3, 4, 6, 7, and 19 remain as candidates. The four words "70%,""3.99%,""3%," and "1.69% invested each" that belong to classification 21 are all "those with a Chinese numeral followed by a fractional word are not candidates. "Drop" (Figure 9 Classification
21), it is dropped from the list of candidates. As a result of the above, "Nishiden", "Nanshiba", "company", "new model",
``Nichikai Denki'', ``Toppan Denwa'', ``Nichikai Iwai'', ``Sumisai Bank'', ``Kokusai Electric Telephone'', ``Matsugiwa Tsushin Kogyo'', ``Nippon Asahi Shimbun'', ``Fundamental Technology Development Promotion Center'' , "Mazukiru", and "Kiyadan", a total of 14 character strings, are sent to the Japanese target sentence-specific term selection unit 4 as candidates for Japanese target sentence-specific terms.

The Japanese target sentence-specific term selection unit 4 searches the Japanese language analysis dictionary 9 for candidates for Japanese target sentence-specific terms, and selects only words that are not registered in the Japanese language analysis dictionary 9 as Japanese language target sentence-specific terms. Select as. For example sentences, the Japanese analysis dictionary 9 is searched for each of the above 14 candidates using the character face as a key. As a result of the search, "company" is listed in Japanese parsing dictionary 9.
Since only ``company'' is registered as a general word, ``company'' is dropped from the list of candidates. Therefore, a total of 13 character strings excluding "company" from the 14 character strings mentioned above are sent to the Japanese target sentence-specific term registration unit 5 as Japanese target sentence-specific terms.

The Japanese target sentence specific term registration unit 5 writes and registers the sent 13 Japanese target sentence specific terms in the Japanese target sentence specific term file 6. The Japanese target sentence specific terms written in the Japanese target sentence specific term file 6 are shown in FIG.

Because of this structure and action, compared to conventional methods, when extracting candidates for terms specific to the Japanese target sentence, we do not perform analysis using an analytical dictionary such as separation, and instead use the method that analyzes changes in character types. It is fast because it extracts candidates by focusing on For candidates extracted by focusing on the change points of character types that can be extracted as candidates for target sentence-specific terms, information in the linguistic information table (``Sequences of 5 or more kanji characters are likely to become Japanese target sentence-specific terms'') is used. ” and Hyurystateskrule,
Processing information such as ``If it contains an affix, process the affix'') is used to narrow down the candidates, allowing for accurate candidate extraction of Japanese target sentence-specific terms. All words that do not match are extracted as Japanese target sentence-specific terms, so Japanese target sentence-specific terms that consist only of common words and Japanese target sentence-specific terms that include some common words are also extracted as Japanese target sentence-specific terms. Improvements were made in each of the points that can be extracted as target sentence-specific terms.

〔Effect of the invention〕

As explained above, according to the device of the present invention, even if a product name, company name, person's name, or other proper noun or word combination is used only in the document, it can be recognized as a new word or a combination of words that is used only in the document. Possible word candidates (the above words are collectively referred to as Japanese target sentence-specific terms) are extracted from the document using the information on the change points of character types, and the linguistic information of the candidates is used to create candidates. After narrowing down the words, we search the analysis dictionary to remove common words and output them as Japanese target sentence-specific terms. It has the advantage of being able to extract quickly and accurately.

[Brief explanation of drawings]

Fig. 1 is a basic configuration diagram of the hardware configuration of this inventive device, Fig. 2 is a schematic flowchart of the operation of this inventive device, Fig. 3 is a diagram showing an example of the contents of a classification table, and Fig. 4 is a diagram showing an example of the contents of a classification table. Figure 5 shows the example sentence used to explain the operation, Figure 5 shows the code string for the example sentence (Figure 4), Figure 6 shows candidates for Japanese target sentence-specific terms extracted from the code string in Figure 5. FIG. 7 is a diagram showing the original character string for the code string in FIG.
The figure shows an example of the contents of the language information table 8, and FIG. 10 shows the extracted terms specific to the Japanese target sentence.

Claims (1)

[Scope of Claims] 1. A linguistic information table that stores conditions for character strings that do not become terms specific to Japanese target sentences due to the nature of Japanese language and conditions for character strings that tend to become terms unique to Japanese target sentences, and the font of Japanese words. A Japanese analysis dictionary that stores characters such as characters and parts of speech, and converts character strings in input Japanese documents into kanji codes, kanji numeric codes, hiragana codes, katakana codes, Arabic numerals codes, alphanumeric codes, and punctuation marks codes. , delimiter code, and other codes; a code string expansion means that expands into code strings classified into nine types of characters; Even if they are proper nouns such as product names, company names, or people's names, or combinations of words, words that are thought to be used only in new words or documents (these words are collectively referred to as Japanese target sentence-specific terms) are A Japanese target sentence-specific word candidate extraction means to extract candidates, and from among the candidates extracted by the extraction means, only candidates that meet the conditions of character strings that are likely to be Japanese target sentence-specific words using the linguistic information table. A Japanese target sentence-specific term language processing means extracts candidates for Japanese target sentence-specific terms with higher accuracy by leaving Japanese target sentence specific term selection means for outputting only words not included in Japanese target sentence specific terms as Japanese target sentence specific terms.