JP3389285B2 - Proper noun identification method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a processing method for specifying a noun in a database or the like, and more particularly to a proper noun specifying method for specifying a proper noun such as a company name, a person's name or a place name appearing in Japanese sentences.

[0002]

2. Description of the Related Art A database or the like requires a search key for obtaining desired data. This search key is often a proper noun, and if a proper noun existing in the database can be specified without any operation, the number of operations by an operator or the like will be small and the effect on the database will be great.

Conventionally, the identification of the proper noun has generally been performed by collating a dictionary in which the proper noun is registered with a sentence. Therefore, it was not possible to identify proper nouns that were not registered in the dictionary. However, in addition to this method, in addition to collating a dictionary with a sentence, a method of identifying a proper noun using a prefix and suffix frequently appearing before and after the proper noun as a key for pattern matching has been put into practical use.

FIG. 7 is a flowchart of a conventional process for estimating proper nouns. When a Japanese sentence is fetched from the input device or the like by the input process of step S1, step S2
By the proper noun pattern matching process of the input character string, proper noun modifier dictionary D1 and proper noun pattern dictionary D
Pattern matching with No. 2 is performed to find proper nouns such as company names, personal names, and place names. The proper noun modifier dictionary D
1 is a prefix, suffix that often appears before and after proper nouns,
Stores information such as synonyms, and proper noun application pattern dictionary D
Reference numeral 2 stores information that defines the appearance forms of proper nouns and their predecessors such as prefixes, suffixes, and synonyms.

After the step S2, the overlapping pattern selection processing of the step S3 indicates that the pattern matching result between the information and the input character string indicates the degree of coincidence of the proper nouns and the length of the matched pattern when the proper noun patterns overlap. And select a probable pattern based on the character position. Then, by the abbreviated proper noun search processing S4 of step S4, even if the prefix and suffix are omitted, it is obtained. Then, the result obtained by the above-described processing is output to the output device by the output processing of step S5.

Although proper nouns have been specified by the above processing, some proper nouns cannot be specified by this method.

[0007]

As described above, in the conventional proper noun specifying technique, a method using a dictionary search or pattern matching using a prefix and a suffix frequently appearing before and after the proper noun is used. There was also the problem that proper nouns could not be identified depending on the method.

According to the present invention, even a proper noun that is not registered in a dictionary and cannot be identified by a prefix and a suffix frequently appearing before and after the proper noun can be identified with high accuracy. The purpose is to provide a noun identification processing method.

[0009]

Means for Solving the Problems and its effect of the present invention, input
Forced to extract proper nouns from Japanese sentences added from force device
Sentences in the Japanese sentence in the noun identification processing system
In a method of estimating whether a character string is a proper noun
Is. The proper noun specifying method according to the present invention is based on the character string
Corresponding to a sentence pattern in which proper nouns frequently appear
The probability that a proper noun exists in the pattern
The proper noun pattern data is stored in the storage means.
Read from the sentence pattern dictionary of proper nouns
At least one of the number of characters and the type of characters
A rule that defines certainty as a proper noun that is a parameter
Probability table with total data stored in storage means
Read out the proper noun pattern data and the
By referring to statistical data,
It is estimated whether or not the result is output from the output device.

In the present invention, proper nouns such as a company name, a person's name, and a place name, which could not be identified by the prior art, can be identified by probability inference from the certainty obtained by pattern matching and the certainty obtained from the features of the proper noun. it can.

The proper noun specified in this way can be used in various application programs such as database registration information and a search key for database search.

[0012]

The present invention will be described in detail below with reference to the drawings. FIG. 1 is a flowchart of proper noun specifying processing for Japanese sentences according to the embodiment of the present invention. The proper noun specifying process for Japanese sentences of the embodiment of the present invention starts when a Japanese sentence is added from an input device (not shown). First, in step S11, the morphological analysis process divides the input character string input in sentence units into words and adds a part of speech. Then, in step S12, a character string including a proper noun such as a company name, a person's name, or a place name is searched for by pattern matching between the input character string divided into words by the proper noun pattern matching process and the proper noun sentence pattern dictionary D11.

The sentence pattern dictionary D11 of proper nouns stores the pattern of sentences in which proper nouns frequently appear and the likelihood that proper nouns exist in the patterns. That is, the pattern defined in the sentence pattern dictionary D11 of proper noun is taken out, and if the character string field in the pattern exists in the input sentence in the defined order, the character string corresponding to the arbitrary field is substituted.

FIG. 2 is a view showing a part of the contents of the proper noun sentence pattern dictionary D11. The content of the pattern dictionary D11 shows an example of a sentence pattern in which proper nouns representing a part of the stored information frequently appear and the certainty of the proper nouns in the patterns. A field starting with the symbol "@", that is, an arbitrary field indicates matching with an arbitrary character string, and a field other than that, that is, a character string field, has the defined character string itself as a target of pattern matching. The optional field "@CNAM"
"E-SUBJ" includes a company name that is a proper noun as an attribute of a pattern, and means that it is a syntactic subject.
The statistical certainty that the proper noun exists in the matched character string is defined immediately after the symbol ":".
This "certainty" is "negative certainty" and represents the degree of uncertainty. For example, a negative likelihood of 1.00 means nothing at all.

After the proper noun pattern matching process in step S12, in step S13, a sequence of proper noun candidates is searched for a continuous word that can be a proper noun from the character string including the found proper name, and a proper noun range is determined. Perform decision processing. That is, in this step S13, the part-of-speech of the word is examined from the tail end of the character string substituted in the arbitrary field having the attribute of proper noun, and the part-of-speech is not an adjunct word Adverbs, symbols,
Except for conjunctions, proper nouns are expanded forward. When a word that does not satisfy the above conditions appears, the next word to the last word is set as a proper noun candidate.

Then, in step S14, the probability that the proper noun estimation process includes the proper noun in the collation character string defined in the proper noun sentence pattern dictionary D11,
The expert system MYCIN (Shortliff) is referred to by referring to the certainty as the proper noun of the proper noun candidate obtained by using the number of the proper noun candidates and the character type defined in the certainty noun table D12 as parameters.
e, EH, Computer-Based Medical Consultations: M
YCIN, New York, NY: El-sevier, 1976). That is, in step S14, the following processing is performed.

The probability that the proper noun is included in the collated character string is acquired from the proper noun sentence pattern dictionary D11 (this value is set to MB). Also, the negative likelihood as a proper noun of the proper noun candidate obtained by using the number of characters of the proper noun candidate as a parameter is acquired from the likelihood table D12 of the proper noun (this value is referred to as MD1). Similarly, the negative certainty as a proper noun of the proper noun candidate obtained by using the type of the first character of the proper noun candidate as a parameter is acquired from the proper noun certainty table D12 (this value is referred to as MD2). Using the probabilistic inference method in MYCIN of the expert system, MD1 and M
The negative certainty MD that integrates D2 can be calculated as MD = MD1 + MD2-MD1 * MD2 (1). Finally, the total likelihood CF as a proper noun can be obtained as CF = MB-MD (2). When the total likelihood (CF) is larger than a predetermined value, the proper noun candidate is estimated as a proper noun, and the proper noun is output from an output device (not shown).

One sentence is processed by the above-described steps S11 to S14, and it is determined whether all the sentences have been processed in step S15 after step S14. When all the sentences have not been processed (NO), the process is repeated from step S11 to process the next sentence to be input. When all the sentences have been processed, the output processing of step S16 outputs the determined processing result to an output device (not shown).

Further, the steps S12 and S1 described above are performed.
The process of S3 and S14 will be described in detail. FIG. 3 is a detailed flowchart of sentence pattern matching processing for proper nouns. In this sentence pattern matching process for proper nouns, it is checked whether or not all the patterns defined in the sentence pattern dictionary D11 for proper nouns are matched.

First, when this process is started, step S2
In step 1, it is determined whether all the patterns defined in the sentence pattern dictionary D11 of proper nouns have been examined. When all the patterns have been checked (YES), the process ends. If all the patterns have not been checked yet (NO), step S22.
One pattern is extracted in step S23, and it is determined in step S23 whether the character strings in the pattern are present in the input sentence in the defined order. That is, it is determined whether the pattern matches the input character. When there is no matching pattern, the process is repeated from step S21. When it exists, the character string corresponding to the arbitrary field is substituted in step S24. By this substitution, one character string in the sentence to be discriminated is determined as a proper noun.

With the above processing, the sentence pattern matching process for proper nouns in step S12 is completed. Then, in step S13, proper noun range determination processing is started.

FIG. 4 is an operation flowchart of proper noun range determination processing. When the proper noun range determination process starts, first, in step S31, the head position and end position of the proper noun candidate are set to the last word of the pattern that may include the proper noun.

In general, when the arbitrary field ends immediately before the postpositional particle, for example, the company name is likely to come at the end of the arbitrary field. Therefore, in the proper noun range determination processing, the initial value of the head position of the proper noun is set to the last word position of the arbitrary field, and it is expanded as far as possible forward. In order to determine such a range, the process of step S31 is performed, and the range for discriminating proper nouns is limited.

Subsequently, in step S32, it is determined whether the word at the head position is not an adjunct word and the part of speech is other than a verb, an adjective, an adjective verb, an adverb, an adverb, a symbol, or a conjunction. If the word is other than the above words (YES), it is determined in step S33 whether or not the leading word of the pattern including the proper noun has been examined. If the leading word is not checked, the leading position of the proper noun candidate is set to 1 in step S34.
One word ahead.

Subsequently, the process is repeated from step S32. When the first word of the pattern including the proper noun is checked in step S33, the portion between them is output as a proper noun candidate in step S35, and the whole process is terminated.

On the other hand, when the word at the head position is an adjunct word or the part of speech is one of a verb, an adjective, an adjective, an adverb, an adverb, a symbol, and a conjunction (NO) in the determination of step S32, In step S36, it is determined whether or not the head position of the proper noun candidate is ahead of the end position. If it is not before the head position of the proper noun candidate, the entire process is ended. When it is in front (YES), the head position of the proper noun candidate is moved back by one in step S37, and the part between them is output as a proper noun candidate in step S35, and the whole process is ended. The range of proper nouns is determined by the above operation.

The range of proper nouns is determined by the processing in step S13 described above. FIG. 5 is a detailed flowchart of the proper noun estimation process in step S14. After the range of proper nouns is determined by the process of step S13, this process is executed. In this processing, the sentence pattern dictionary of proper nouns and the probability table D12 of proper nouns are used.
Estimate proper nouns using.

First, when this processing is started, it is determined in step S41 whether or not the arbitrary fields having the attributes of all proper nouns have been processed. In an embodiment of the invention,
Since the whole sentence to be input is estimated, when the arbitrary fields having the attributes of all proper nouns are processed (YES), the proper noun estimation process is terminated. On the other hand, if not completed (NO), the certainty (BB) that the proper noun is included in the collation character string is obtained from the proper noun sentence pattern dictionary D11 in step S42.

FIG. 6 (a) is a table chart showing an example of statistical information indicating the probability of proper noun candidates as proper nouns obtained by using the number of characters of proper noun candidates as a parameter, and FIG. 6 (b) is proper noun. 7 is a table chart showing an example of statistical information indicating the certainty as a proper noun of a proper noun candidate obtained by using the character type at the beginning of the candidate as a parameter. In the embodiment of the present invention, the probability of not becoming a proper noun is defined as negative certainty.

Further, returning to FIG. 5, the description will be continued. After step S42, in step S43, the negative likelihood (MD1) of the proper noun candidate is obtained from the likelihood table D12 of the proper noun using the number of characters of the proper noun candidate as a parameter. Further, in step S44, the negative likelihood (MD2) of the proper noun candidate is set to the proper noun probability table D12 using the leading character type of the proper noun candidate as a parameter.
Ask from.

In steps S43 and 44 described above, the negative certainty (MD1, MD2) of the proper noun is obtained. Using this result, in step S45, the negative likelihood (MD) as a proper noun is calculated using the equation (1). Then, the negative likelihood (MD) integrated with the certainty (MB) including the proper noun obtained in step S42 described above.
From the above, the total likelihood (CF) is obtained in step S46 using the equation (2).

In the embodiment of the present invention, when CF is larger than 0.10.
Then, it is determined whether or not it is larger than the specified value 0.10.
Execute from 41. Also, when it is greater than 0.10 (YES)
Is determined as a proper noun and is output as a proper noun in step S48. Then, the process is executed again from step S41. By repeating the above, the proper noun of the input sentence is obtained.

In the following, the input character string is "2
On the 2nd, it was announced that the XYZ trust has formed a tie-up with regard to sales in Japan. "

In the sentence pattern dictionary of proper nouns, the character string fields "ha", "to", and "affiliation" in the pattern of FIG. 2 exist, and they exist in the above order in the input character string. Therefore, pattern matching is performed. In the optional fields, @ COMPANY-SUBJ = “Company” @ COMPANY-PARTNER = “22 days, XY
“Z Trust”, @ SKIP = “Regarding sales in Japan”. This is performed by the proper noun sentence pattern matching process in step S12.

Arbitrary fields with proper noun attributes are @
COMPANY-SUBJ and @ COMPANY-PAR
Since it is TNER, the range of proper noun candidates is individually determined. The former is a noun consisting of one word "company", so it is a proper noun candidate as it is. The latter morphological analysis result of the character string is the word division position by "/", "[]"
The part of speech is shown as "22 [number] / day [suffix] /,
[Symbol] / XYZ [noun or unregistered word] / trust [noun]. Then, the word is traced back from the rear and the part of speech is examined. Since "," is a symbol and does not become part of a proper noun, the word "XYZ trust" after that becomes a candidate for a company name. This processing is performed in the proper noun range determination processing in step S13.

Next, proper nouns in the sentence are estimated by proper noun estimation processing in step S14. First, @COMPANY
-The company name "Company" estimated from SUBJ and @COMP
It is individually judged whether or not the “XYZ trust” estimated from ANY-PARTNER is a proper noun. For the former, the probability MB that the proper noun is included in the matching character string is 0.97 from the proper noun sentence pattern dictionary D11. Also, "company" is a two-letter word,
From the certainty probability table D12 of the proper noun (FIG. 6A), the negative certainty MD1 as a proper noun is 0.85. Similarly, "company" is a word consisting of kanji,
The negative likelihood MD2 as a proper name is 0.77 according to the proper noun certainty table (FIG. 6 (b)). Therefore, the overall negative likelihood MD is
By (1), it becomes 0.97. Therefore, the probability CF as a comprehensive proper noun is 0.00 by the above formula (2).
Therefore, if the specified value indicating whether or not it is recognized as a proper noun is 0.10, it is below the specified value, so it is not judged as a proper noun.

On the other hand, for "XYZ trust", MB = 0.94, MD1 = 0.59, MD2 = 0.45 are obtained by the same calculation, and the certainty as a comprehensive proper noun is calculated by the above formula (1). , (2) gives CF = 0.17. Since this is larger than the above-mentioned specified value, it is determined to be a proper noun and is output.

It is needless to say that the above embodiment shows one example of the present invention, and the present invention should not be limited to this.

[0039]

As described above in detail, according to the present invention, it is possible to specify by the method by the dictionary search and the method by the pattern matching using the prefix and suffix frequently appearing before and after the proper noun. A proper noun
A proper noun identification processing system that can identify with high accuracy can be realized.

[Brief description of drawings]

FIG. 1 is an operation flowchart showing an outline of proper noun identification processing for a Japanese sentence according to an embodiment of the present invention.

FIG. 2 is a diagram showing a part of the contents of a proper noun sentence pattern dictionary D11 according to an embodiment.

FIG. 3 is an operation flowchart of proper noun pattern matching processing according to the embodiment.

FIG. 4 is an operation flowchart of proper name range determination processing according to the embodiment.

FIG. 5 is an operation flowchart of proper noun estimation processing according to the embodiment.

FIG. 6 is a diagram showing a part of the contents of a probability table of proper nouns according to the embodiment.

FIG. 7 is a flowchart of a conventional process for estimating proper nouns.

Continuation of the front page (56) References JP-A-2-51765 (JP, A) JP-A-3-278176 (JP, A) JP-A-4-188364 (JP, A) Tsuyoshi Kitani, specific function of proper noun Morphological analysis processing, IPSJ Research Report 92-NL-90, Vol. 92 No. 55, p. 73-80 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06F 17/30

Claims (2)

(57) [Claims] 1. A proper noun identification processing system for extracting proper nouns from a Japanese sentence added from an input device.
Estimating whether a character string in a Japanese sentence is a proper noun
In the method, proper noun pattern data corresponding to the character string is stored in a storage unit, the proper noun pattern data defining the probability that the proper noun is present in the pattern in a pattern unit of a sentence in which the proper noun frequently appears.
It is read from the sentence pattern dictionary of remembered proper nouns, and statistical data that determines the certainty as proper nouns with at least one of the number of characters forming the proper noun or the character type as a parameter is stored in the storage means.
Read from the size table and refer to the proper noun pattern data and the statistical data to estimate whether or not the proper noun by probabilistic inference
Then, the proper noun specifying method is characterized by outputting the result from an output device . 2. The probabilistic inference uses a first value obtained from the statistical data with the number of characters of proper noun candidates as a parameter, and the statistics from the first certainty with respect to proper noun candidates obtained from the proper noun pattern data. The second obtained from the data using the type of the first character of the proper word candidate as a parameter
Is added, and the third certainty obtained by subtracting the second certainty obtained by subtracting the multiplied value of the first value and the second value from the addition result is greater than or equal to a specific value. 2. The proper noun specifying method according to claim 1, further comprising estimating whether or not the proper noun candidate is a proper noun.