JP5585961B2 - Predicate normalization apparatus, method, and program - Google Patents

Description

  The present invention relates to a predicate normalization apparatus, method, and program. In particular, in natural language processing, the functional expression of a predicate is simply paraphrased and the paraphrased description is targeted for a predicate that is the center of sentence information. The present invention relates to a predicate normalization apparatus, method, and program for judging grammatical properties so that parts become natural Japanese.

  In order to obtain useful information from a large amount of documents such as minutes, questionnaires, web texts, contact center conversation logs, etc., extract important information such as “who did what” from the text, and elements with the same meaning It is indispensable to collect and aggregate every time. However, since the same content is written in various expressions on the text, preprocessing is necessary for the machine to collect expressions having the same meaning. In particular, the predicate that expresses “what happened” consists of a combination of “content words” such as verbs, nouns, adjectives, adjectives, adverbs, and “function words” such as particles and auxiliary verbs. The difference is intense. For example, in the following example sentence “I may be out of order”, one content word and five different function words are attached to one predicate. A case where one or more function words constitute a single semantic unit is called “function expression”.

Failed (content word) / is (function word 1) / (function word 2) /
May be (Function word 3) / Wa (Function word 4) / Ne (Function word 5)

  Thus, since the predicate is composed of a combination of various elements, the difference in the surface layer form is significant, and it is difficult to extract synonymous expressions by a machine.

  One method for solving the difficulty of extracting these synonymous expressions is “paraphrase”. For example, a predicate that often appears in colloquial expressions, such as “I think it may be out of order,” is rephrased into a simple form as “It may be out of order,” so that the predicate becomes Without changing the meaning of the expression, it can be judged by the machine that ““ It may have failed ”is synonymous with“ May have failed ”. In other words, predicates that represent the same meaning can be aggregated by a machine by automatically rephrasing the same surface layer form. This process is called predicate normalization. The preprocessing using these paraphrases can be used not only for predicate summarization but also as preprocessing for summarization and machine translation.

  Conventionally, as a method of simply rephrasing predicates, function words or function expressions of predicates are classified using abstract semantic labels based on whether or not they affect the meaning of the event represented by the predicate. However, a method for normalizing predicates by leaving functional expressions belonging to “influence” has been proposed (see, for example, Non-Patent Document 1). In the method of Non-Patent Document 1, three indicators, “difference in tense”, “difference in negative”, and “difference in modality” are set as abstract semantic labels for classifying functional expressions. In normalization, functional expressions that do not belong to these are deleted. Also, the auxiliary verb “da” and the auxiliary particle “no” are classified into the category “Grammar”, and the functional expressions classified as “Grammar” before or after the functional expression belonging to the modality are left without deletion. Yes.

Izumi T., Imamura K., Kikui G. & Sato S., `` Standardizing Complex Functional Expressions in Japanese Predicates: Applying Theoretically-Based Paraphrasing Rules '', Proceedings of the Workshop on Multiword Expressions: From theory to applications (MWE 2010), 63-71.

  However, as in the technique described in Non-Patent Document 1, when only the functional expression that affects the meaning of the event represented by the predicate is left and the rest is simply deleted, a grammatically incorrect paraphrase is generated. There is a problem that there is.

  For example, a predicate “You may be weak” is explained as an example.

Not good (content word) / NA (functional expression / G) / no (functional expression / G) /
It may be (functional expression)

  As described above, the predicate includes three functional expressions. Note that “G” represents that the above-described functional expression is classified into “Grammar” indicating that it is classified into the auxiliary verb “da” and the auxiliary particle “no”. In the method of Non-Patent Document 1, the function expression classified as “Grammar” before or after the function expression belonging to the modality is left, and the others are deleted. Therefore, in the above example, as shown in FIG. 17, “no” positioned before “may be” that is a functional expression belonging to the modality is left, and there is no functional expression belonging to the modality before and after “n”. Is deleted. Then, “may be weak” is generated as a normalized predicate. In this case, in order to connect the noun-adjective verb stem “poor” to the particle “no”, the auxiliary verb “da (the surface form is“ na ”)” has been deleted. The generated paraphrase is not correct in Japanese. This is because there are two points necessary for paraphrasing the predicate that (1) the predicate is simplified without changing the meaning represented by the predicate, and (2) the paraphrased predicate is correct as Japanese. This is because the above (2) is not considered.

  The present invention has been made in view of the above problems, and an object thereof is to provide a predicate normalization apparatus, method, and program capable of simply and grammatically correct predicate paraphrase.

  To achieve the above object, the predicate normalization apparatus of the present invention uses a morpheme analysis unit that performs morpheme analysis on an input document, and a predicate of the document based on a result of morpheme analysis by the morpheme analysis unit. Labeling means for assigning a judgment label indicating that the grammatical necessity is different to each of functional expressions that are included and have different grammatical needs depending on words existing in the vicinity, and the judgment label is given. The morpheme constituting the predescription part when at least one of the function expressions included is included in the predescription part, and the morpheme constituting the predescription part when each of the functional expressions to which the determination label is assigned is excluded For each of the columns, a morpheme N-gram model constructed with pseudo-words that have morpheme surface forms as elements for function words and words other than the function words do not have the surface layer form as elements. Based on the calculation means for calculating a score indicating whether or not the functional expression to which the determination label is given is grammatically required, and normalization from the morpheme sequence selected based on the score calculated by the calculation means Generating means for generating the predicate.

  According to the predicate normalization apparatus of the present invention, the morpheme analysis unit performs morpheme analysis on the input document, and the label addition unit is included in the predicate of the document based on the result of the morpheme analysis by the morpheme analysis unit. At the same time, a judgment label indicating that the grammatical necessity is different is assigned to each functional expression having a different grammatical necessity depending on the words existing in the vicinity. When the calculation means excludes each of the morpheme sequence constituting the predicate when the predicate includes at least one of the functional expressions to which the determination label is given and the functional expression to which the determination label is given. For each of the morpheme strings constituting the predicate, a score indicating whether or not the functional expression to which the determination label is assigned is grammatically necessary is calculated based on the morpheme N-gram model. This morpheme N-gram model is an N-gram model constructed with pseudo-words having the surface form of the morpheme as an element for function words and the surface layer form for elements other than the function words. Then, the generation unit generates a normalized predicate from the morpheme sequence selected based on the score calculated by the calculation unit.

  In this way, since the grammatical nature of functional expressions having different grammatical needs depending on the words existing in the vicinity is judged using the morpheme N-gram model, it is possible to simply rephrase predicates that are grammatically correct. . For function words, the morpheme surface form is used as an element, and for words other than function words, a morpheme N-gram model constructed with pseudo-words that do not use the surface form as an element is used. In other words, the surface form of the functional expression can be used as a grammatical judgment criterion while suppressing the fluctuation of the score due to variations in the surface form of the content word, that is, the grammatical judgment of high accuracy.

  Further, the calculation means can calculate the score based on the morpheme N-gram model and a part-of-speech N-gram model constructed by pseudo words having elements other than the surface form of the morpheme as elements. Thus, by using the part-of-speech N-gram model together with the morpheme N-gram model, the accuracy of the grammatical judgment of the target functional expression is further improved.

  In addition, the label assigning means includes the determination label and an unnecessary label indicating that the functional expression is semantically and grammatically unnecessary, and whether the functional expression affects the meaning of the previous description part. Is added to each of the functional expressions included in the previous description part, and the calculation means includes a functional expression to which an unnecessary label is assigned by the label attaching means, and a functional expression to which the same semantic label is assigned. The score can be calculated using a predicate in which a functional expression other than at least one of each of the above is deleted. As a result, a more simplified predicate can be generated.

Further, the predicate normalization method of the present invention performs a morphological analysis on an input document to a computer, and includes a grammatical expression based on a word existing in the predicate of the document based on the result of the morphological analysis. When a determination label indicating that the grammatical necessity is different is assigned to each functional expression having a different necessity, and at least one of the functional expressions having the determination label is included in the previous description section For each of the morpheme strings constituting the preceding description part when excluding each of the morpheme strings constituting the preceding description part and the function expression to which the determination label is assigned, the function word is composed of the surface form of the morpheme. Whether the functional expression to which the determination label is attached is grammatically required based on a morpheme N-gram model constructed by a pseudo word that does not have the surface layer as an element for words other than the functional word A score indicating the calculated, from the morpheme string selected based on the calculated score is a method for executing the processing comprising generating a predicate normalized.

  The predicate normalization program of the present invention is a program for causing a computer to function as each means constituting the predicate normalization apparatus.

  As described above, according to the predicate normalization apparatus, method, and program of the present invention, a pseudo word that has a morpheme surface form as an element for a function word and a surface form as an element for a word other than a function word. Using the morpheme N-gram model constructed by, we can determine the grammatical nature of functional expressions that have different grammatical needs depending on the words existing in the vicinity. The effect of is obtained.

It is a block diagram which shows the functional structure of the predicate normalization apparatus of this Embodiment. It is a figure which shows an example of the result of a morphological analysis. It is a figure which shows an example of a function word meaning label dictionary. It is a figure which shows an example of the meaning label which belongs to "Grammar". It is a figure which shows an example of provision of a semantic label, and a predicate extraction result. It is a figure which shows an example of the processing result in a NULL deletion part. It is a block diagram which shows the functional structure of a Ngram grammar property judgment part. It is a figure which shows an example of the conventional morpheme Ngram model. It is a figure which shows an example of a part of speech Ngram model. It is a figure which shows the paraphrase precision comparison at the time of changing the mixing rate of the conventional morpheme Ngram and the part of speech Ngram. It is a figure which shows an example of the morpheme Ngram model of this Embodiment. It is a figure which shows the paraphrase precision comparison at the time of changing the mixing rate of the morpheme Ngram of this Embodiment, and the part of speech Ngram. It is a figure which shows an example of the Ngram score calculation result for every candidate. It is a flowchart which shows the content of the predicate normalization processing routine in the predicate normalization apparatus of this Embodiment. It is a figure which shows the other example of the processing result in a NULL deletion part. It is a figure which shows the other example of the Ngram score calculation result for every candidate. It is a figure for demonstrating deletion of the function expression in a prior art.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  A predicate normalization apparatus 10 according to the present embodiment is configured by a computer including a CPU, a RAM, a program for executing a predicate normalization processing routine described later, and a ROM storing various data. Yes. As shown in FIG. 1, this computer functionally uses a morpheme analysis unit 12 that performs morphological analysis on an input document (text data) and a semantic label / predicate model 14 to perform morpheme analysis results. The semantic label assignment / predicate extraction unit 16 that assigns the semantic label of the functional expression and extracts the predicate of the document, and the NULL deletion unit 18 that deletes the functional expression to which the semantic label “NULL” is assigned are the same. Redundant rule application unit 20 that leaves and deletes only one functional expression with a semantic label, Ngram grammar determination unit 22 that determines the grammatical necessity of the functional expression using the Ngram model, and unnecessary functions It can be expressed by a configuration including a utilization generation unit 24 that generates a predicate using the remaining morpheme from which the expression is deleted.

  The morpheme analysis unit 12 performs morpheme analysis for each sentence using a known morpheme analyzer on the input document. In morphological analysis, a sentence is divided into words, and information such as part of speech, utilization type, and utilization form is given to each word. FIG. 2 shows an example of the result of the morphological analysis for one sentence of the input document “He may not be good at singing”.

  The semantic label / predicate model 14 adds a semantic label indicating whether or not the meaning of the event represented by the predicate affects each functional expression and extracts a predicate from the input document. It is a model. The semantic label / predicate model 14 is based on the functional word semantic label dictionary, and the corpus in which the correct semantic label is manually assigned to the functional word is used as learning data, and the likelihood of the predicate range and the semantic label arrangement is estimated. Has been generated by learning. In the function word meaning label dictionary, function expressions that are supposed to affect the meaning of the predicate are defined. For example, as shown in FIG. They are classified and stored in advance based on three indicators of “modality differences”. Furthermore, there is a functional expression classified as “Grammar”. “Grammar” means “elements that are not semantically important but are grammatically necessary depending on the surrounding words”. FIG. 4 shows semantic labels belonging to “Grammar”. Typical function words belonging to “Grammar” are the auxiliary verb “da” and the auxiliary particle “no”. Here, the meaning label of “da” is “judgment”, “no” according to the usage. The semantic label is “nounization”.

  The semantic label assignment / predicate extraction unit 16 automatically assigns a functional expression semantic label to the morphological analysis result by a statistical method using the semantic label / predicate model 14, and further determines the range of the predicate. Extract. FIG. 5 shows an example of the result of assigning the semantic label and extracting the predicate. Since the predicate is composed of a series of one or more content words (C) and zero or more function words (F), “maybe I am not good” is extracted as a predicate. This predicate has four functional expressions of “I am not good” and “Nano / Maybe / Ne”.

  Also, in this example, a BI tag is used, and the content word of the predicate, the first word of the content word (“poor”) is given the label “C, B-PRED”, and A word “C, I-PRED” is assigned to the word of the predicate other than the head of the content word (here, no corresponding word). C is an acronym for content words. Further, a function word (Function words) is given a label of “F”, and in addition, a corresponding semantic label is given in units of function expression. Since the function expression “may be” is composed of the function words “kamo”, “know” and “no”, “F” indicating the function word in each of “kamo”, “know” and “no”. ”And a semantic label“ inference ”corresponding to the function expression“ may be ”. Similarly to the content word, when the corresponding function word is the first word (“Moka”) of a certain function expression (“Maybe”), the function word is “B-estimation”. If the word “B” is added to the front of the semantic label given to the functional expression including “”, and it is a word other than the beginning (“Know”, “None”), the front of the semantic label such as “I-inference” Add "I" to In addition, in the function words belonging to “Grammar” (here, the auxiliary verb “da” and the particle “no”), similarly to the above, the label “F” indicating the function word, the head of the function expression other than the head or the head Along with a label “B” or “I” indicating a word, a meaning label of “judgment” or “nounization” is given according to the use of the word. For words other than the words that are not registered in the function word meaning label dictionary and the words that belong to “Grammar”, similarly to the above, the label “F” indicating the function word, and the word other than the head or the head of the function expression are indicated. The label “NULL” is given together with the label “B” or “I”.

  The NULL deletion unit 18 deletes the functional expression assigned “NULL” as the semantic label. The fact that “NULL” is given is a word that is not defined in the function word meaning label dictionary as a function expression that affects the meaning of the predicate, and is not semantically important. Indicates that it is not classified into the category “Grammar” which is a grammatically necessary element. Such a functional expression is deleted because there is no problem even if it is deleted.

  The redundancy rule application unit 20 deletes the functional expressions to which the same semantic label is assigned, leaving only one. As a result, unnecessary function expressions are deleted, and the redundancy of predicates is eliminated. FIG. 6 shows an example after NULL deletion and redundancy rule application. Redundancy rules are applied in units of function expressions. Therefore, in the example of FIG. 6, “Know” and “None” are assigned the same semantic label, but are not deleted because they are part of the functional expression “may be”.

  The Ngram grammaticality determination unit 22 determines which function expression is necessary and which function expression is unnecessary among the function expressions belonging to “Grammar”. As shown in FIG. 7, the Ngram grammaticality determination unit 22 includes a candidate generation unit 22a that creates all combination candidates when a functional expression to be subjected to grammatical determination is left and when it is excluded, a morpheme Ngram model 22b, and a part of speech. An Ngram calculation unit 22d that calculates an Ngram score for each generated candidate based on the Ngram model 22c and a selection unit 22e that selects one of the candidates based on the Ngram score are used. be able to.

  The candidate generation unit 22a uses the semantic label assignment / predicate extraction unit 16 as a function expression belonging to “Grammar” for the predicate from which the unnecessary function expression is deleted by the NULL deletion unit 18 and the redundancy rule application unit 20. ”And“ Nounization ”create all combination candidates for predicate morpheme sequence candidates that have left function words and predicate morpheme sequence candidates that exclude target function words. . Here, since “na (judgment)” and “no (nounization)” are the target function words, both “weak” / “ka” / “know” / “na” are deleted and “na” is deleted. "I'm not good" / "" / "" / "" / "" / "" "" No / May / Know / No ”is generated as a candidate.

  The morpheme Ngram model 22b and the part-of-speech Ngram model 22c (hereinafter, when these two models are described collectively or without being distinguished from each other, are simply referred to as “Ngram models”) This is a model for giving a score of “likelihood as a word string” based on “For example,“ Ken Kenji, Satoshi Nakamura, Masaaki Nagata “Spoken Language Processing-Corpus-based Approach”, Morikita Publishing, 2.4 See chapter). In order to calculate the score of a sequence having an arbitrary number of words, the Ngram model obtains a word generation probability from a sequence of N words (this is called Ngram, and when N = 3 is called Trigram), It is calculated by calculating this total product for the entire word string. In the present embodiment, the logarithmic value of the probability is called a score, and the score of the entire input word string is called an Ngram score. The generation probability of each Ngram is learned in advance from the corpus.

  The Ngram score calculation unit 22d calculates an Ngram score for each candidate generated by the candidate generation unit 22a using an Ngram model. In the case of calculating one Ngram score using two or more Ngram models as in the present embodiment, for example, the Ngram score can be calculated by the following equation (1).

logP = αlogP _a + (1−α) logP _b (1)

Where logP is the Ngram score representing the final “likelihood as a word string”, logP _a is the Ngram score calculated from the morpheme Ngram model 22b, and logP _b is the Ngram score calculated from the part-of-speech Ngram model 22c. Is the mixing ratio of both. Using this Ngram score, the grammatical judgment of the functional expression belonging to “Grammar” is performed.

  Here, the principle of the grammatical judgment of the function expression belonging to “Grammar” in the present embodiment will be described.

  If the input word string has already been morphologically analyzed, it is more accurate to calculate the Ngram score using information such as the morpheme part of speech and the utilization type rather than simply using the surface form of the morpheme. The “probability as a word string” can be calculated. For example, in the conventional morpheme Ngram model, an Ngram model is constructed using a combination of the morpheme surface form, part of speech, and utilization type as a pseudo word, and in the part of speech Ngram model 22c, a combination of morpheme part of speech and utilization type is simulated. An Ngram model is constructed as a word, both Ngram scores are complemented with an appropriate mixing ratio α, and a final Ngram score can be calculated using the equation (1). FIG. 8 shows an example of a conventional morpheme Ngram model when N = 3, and FIG. 9 shows an example of a part-of-speech Ngram model 22c when N = 3.

  In a language model used in machine translation, the conventional morpheme Ngram model and part-of-speech Ngram model 22c are combined with an appropriate mixing ratio (for example, α = 0.8, that is, model a: model b = 0.8: 0. 2) is used to calculate the Ngram score. However, when this method is applied to the grammatical judgment of the functional expression targeted by the present embodiment, the predicate paraphrase accuracy varies greatly depending on the mixing ratio, as shown in FIG.

  In order to judge grammaticality more correctly, it is appropriate to increase the ratio of the conventional morpheme Ngram model that includes the surface form element, but in the case of grammatical judgment of functional expression, the mixing ratio of the conventional morpheme Ngram model When the value is increased (that is, when α is brought close to 1), the accuracy decreases. This is considered to be because the content word of the predicate has an adverse effect. The predicates targeted by the present invention also include content words, but the content words have more surface layer forms that have the same meaning than the function words, so the scores of the conventional morpheme Ngram model vary. Occurs. This is because the score cannot be calculated correctly because of insufficient data when learning the Ngram probability. However, in order to perform learning correctly without facing these problems, a large amount of learning data is required.

  Therefore, in the present embodiment, in the Ngram score calculation when the grammatical judgment of the functional expression is performed, a pseudo word including a surface form as an element for a functional word constituting the functional expression, and a surface layer for a word other than the functional word An Ngram model (morpheme Ngram model 22b) constructed by pseudo words that do not include a shape as an element is used. As a result, for the functional expression for which grammatical judgment is to be made, the accuracy of the Ngram model can be improved by including the surface layer form in the element, leaving important information in the Ngram model. For content words that do not affect and have a large variation in surface layer shape, the accuracy of the Ngram model can be avoided by removing the surface layer shape from the elements.

  FIG. 11 shows an example of a morpheme Ngram model 22b in the present embodiment. Which type of morpheme surface form to use is determined by the location where grammatical judgment is desired. Here, in the part of speech, if the type of part of speech is included in the part of speech, “participant, auxiliary verb, filler, other, symbol, non-independent, special, suffix, conjunction-like, verb non-independent” Are used as elements of the morpheme Ngram model 22b. As shown in FIG. 11, the surface layer forms of content words such as “verb-independence” and “adjective-independence” are unified to “*”, and the surface form is also an element for morphemes belonging to “auxiliary verbs” and “particles”. Used as Since there are fewer types of function words than content words, learning can be performed with a limited number of data, and the arrangement of the surface layer form of the function words can be added to the morpheme Ngram model 22b. Using these pseudo words, the morpheme Ngram model 22b is learned using an existing learning tool. The part-of-speech Ngram model 22c is also learned using an existing learning tool by using pseudo-words having the part-of-speech and the utilization type as elements.

  As shown in FIG. 12, it can be seen that by using the Ngram model of the present embodiment, variation in accuracy due to the mixing ratio can be suppressed, and paraphrase accuracy can be maintained high.

  Then, when calculating the Ngram score using this Ngram model, in the calculation of the score based on the morpheme Ngram model 22b, the word other than the function word includes the surface layer for the function word as in the learning. For, the score is calculated without including the surface form as an element.

  In addition, the Ngram score calculation unit 22d adds a punctuation mark “.” To the end of each candidate generated by the candidate generation unit 22a, and calculates the Ngram score of each candidate using the Ngram model. The reason for adding a punctuation mark “.” At the end of each candidate is to see “the most likely word sequence among predicates appearing at the end of the sentence (ie, the final form)”. FIG. 13 shows an example of the Ngram score calculated for each candidate.

  The selection unit 22e selects a candidate having the highest Ngram score based on the Ngram score of each candidate calculated by the Ngram score calculation unit 22d, and outputs a morpheme sequence that constitutes the selected candidate. In the example of FIG. 13, “I'm not good / may / know / not” from which all function words belonging to “Grammar” are deleted is the candidate with the highest Ngram score, so this candidate is selected. As a result, for the predicate of the input document, “He may not be good at singing”, a combination of “Maybe weak” which is the most simple and grammatically correct paraphrase expression is output.

  The utilization generation unit 24 correctly utilizes all elements of the morpheme string output from the selection unit 22e to generate a final predicate. In the present embodiment, a utilization generator based on a language model is used. This is a generator based on a model in which “what connection is likely” is learned from the correct data in advance by using the surface type / part of speech / utilization type of the front word and the surface type / part of speech of the rear word as features. Based on this language model, an optimal notation is generated when the surface form / part of speech / utilization type of the previous and next words is input. Here, the morpheme strings “unsatisfactory / noun-adjective verb stem”, “kam / particle-adverb”, “know / verb-independent / one-step”, “no / auxiliary verb / special / nay” ”Is input, and the predicate correctly connected,“ may be weak ”is generated.

  Next, a predicate normalization processing routine executed in the predicate normalization apparatus 10 of the present embodiment will be described with reference to FIG.

  In step 100, the input document is subjected to morphological analysis for each sentence using a known morphological analyzer, the sentence is divided into words, and information such as part of speech, utilization type, and utilization form is provided for each word. Give. Here, as an example of one sentence included in the input document, “the main character is like him” will be described.

  Next, in step 102, a functional expression semantic label is automatically assigned to the morphological analysis result in step 100 by a statistical method using the semantic label / predicate model 14. To extract. Here, “he” is extracted as a predicate, with “he” as a content word and four functional expressions “no / yo / da / ne”. Among the functional expressions, the meaning label “noun” belonging to “Grammar” is assigned to “no”, and the meaning label “judgment” belonging to “Grammar” is assigned to “da”. The meaning label “NULL” is given to “Ne”.

  Next, in step 104, the functional expression assigned “NULL” as the semantic label in step 102 is deleted. Here, “Ne” is deleted.

  Next, in step 106, referring to the meaning label assigned in step 102, the function expression to which the same meaning label is assigned is deleted except for one. Here, since there is no corresponding function expression, the routine proceeds to step 108 as it is. As shown in FIG. 15, the predicate becomes “his / no / yo / da” through the processing of steps 104 and 106 described above.

  Next, in step 108, “determination” and “nounization” are performed as function expressions belonging to “Grammar” in step 102 for predicates from which unnecessary function expressions have been deleted through the processing in steps 104 and 106. All combination candidates are created for the morpheme sequence candidate of the predicate that retains the functional expression to which the semantic label is assigned and the predicate morpheme sequence candidate that excludes the target functional expression. Here, because “no (nounization)” and “da (judgment)” are the functional expressions of the object, both “hi / yo” and “da” are deleted and “no” is left. In addition, “hi / no / yo”, “hi”, “no”, “hi”, “hi”, “no”, and “hi” are created as candidates.

Next, in step 110, after adding a punctuation mark “.” To the end of each candidate generated in step 108 above, for each function word, the morpheme surface form is included as an element for the function word. the term non without including the surface-shaped elements, morphemes Ngram calculates a score logP _a based on the model 22b, also a part of speech and conjugations of each word as an element, the score logP _b based on parts of speech Ngram model 22c And Ngram score is calculated according to the above equation (1). FIG. 16 shows the Ngram score of each candidate.

  Next, in step 112, based on the Ngram score of each candidate calculated in step 110, the candidate having the highest Ngram score is selected, and the morpheme string that constitutes the selected candidate is output. Here, “his / no / yo / da” is selected.

  Next, in step 114, all the elements of the morpheme string selected in step 112 are correctly used to generate a final predicate. As a result, a combination of “Like him” is output, which is the simplest and grammatically correct paraphrase expression for the input “Like him is the main character”. As a result, the predicate is normalized from “Like him” to “Like him”.

  As described above, according to the predicate normalization apparatus of the present embodiment, the meaning of the predicate is not affected, but depending on the surrounding words, “Grammar” can be expressed as a functional expression that may be required as a Japanese grammar. ”, And when the functional expression belonging to“ Grammar ”is included, the Ngram score indicating the likelihood as a word string is calculated using the Ngram model for each of the removed cases so that the Ngram score becomes high Since the function expression belonging to “Grammar” is left or deleted, the predicate paraphrase can be simply and grammatically correct. In addition, when using the Ngram model, the morpheme Ngram model and the part-of-speech Ngram model are used. In the morpheme Ngram model, the surface form of the morpheme is used as an element for a function word, and the appearance form is used as an element for a word other than the function word. Therefore, the surface form of the functional expression can be used as a grammatical judgment criterion while suppressing the fluctuation of the Ngram score due to the variation of the surface form of the content word contained in the predicate, and the grammatical judgment can be made with high accuracy. Can do.

  In addition, the predicates normalized in the present embodiment can be used as preprocessing of natural language processing techniques such as predicate compilation performed in text mining, machine translation, summarization, etc. The accuracy of the process can be improved.

  In the above embodiment, the case where the morpheme Ngram model and the part-of-speech Ngram model are used together to determine the grammatical property of the functional expression belonging to “Grammar” has been described, but the determination may be made using only the morpheme Ngram. .

  In the above embodiment, the morpheme Ngram model has the surface layer form (only function words), the part of speech and the inflection type as elements, and the part of speech Ngram model has been described with the part of speech and the inflection type as elements. Instead of or in addition to the utilization type, other elements of the morpheme, for example, a standard form or a utilization form may be used as an element.

  In the above embodiment, the case where the functional expression belonging to “Grammar” is the particle “no” and the auxiliary verb “da” has been described, but other function words are assumed to be functional expressions belonging to “Grammar”. You may register and give a new semantic label. In addition, as in Non-Patent Document 1, using a rule, a semantic label belonging to “Grammar” is assigned, and only when a functional expression belonging to a modality exists before or after, the target of grammatical judgment of this embodiment You may add a restriction such as

  In the above embodiment, the case where the semantic label assignment / predicate extraction unit performs the assignment of the semantic label to the functional expression and the extraction of the predicate using the learned semantic label / predicate model has been described. However, a predicate may be extracted according to a predetermined predicate extraction rule after giving a semantic label to the result of morphological analysis based on a function word semantic label dictionary.

  The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  Further, in the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10 Predicate normalization apparatus 12 Morphological analysis part 14 Semantic label and predicate model 16 Semantic label provision and predicate extraction part 18 NULL deletion part 20 Redundancy rule application part 22 Ngram grammatical judgment part 22a Candidate generation part 22b Morphological Ngram model 22c Part of speech Ngram model 22d Ngram score calculator 22e selector 24 utilization generator

Claims (5)

Morphological analysis means for morphological analysis of the input document;
Based on the result of the morpheme analysis by the morpheme analysis means, the grammatical necessity is different for each functional expression that is included in the predicate of the document and has different grammatical necessity depending on the words existing in the vicinity. Label giving means for giving a determination label indicating that;
Before excluding each of the morpheme sequence constituting the previous description part when at least one of the functional expressions to which the determination label is assigned is included in the previous description part, and the functional expression to which the determination label is assigned For each of the morpheme strings that make up the description part, based on the morpheme N-gram model constructed with pseudo-words that have the surface form of the morpheme as an element for the function word and the surface layer form for the word other than the function word Calculating means for calculating a score indicating whether the functional expression to which the determination label is given is grammatically necessary;
Generating means for generating a normalized predicate from a morpheme sequence selected based on the score calculated by the calculating means;
Predicate normalization device containing   2. The predicate normalization according to claim 1, wherein the calculation unit calculates the score based on the morpheme N-gram model and a part-of-speech N-gram model constructed by a pseudo word including elements other than the surface form of the morpheme. apparatus. The label assigning means includes the determination label and an unnecessary label indicating that the functional expression is semantically and grammatically unnecessary, and whether or not the functional expression affects the meaning of the previous description part. A semantic label is assigned to each functional expression included in the preceding description part,
The calculation means uses the predicate in which the function expression to which the unnecessary label is assigned by the label assignment means and the function expression other than each of the function expressions to which the same semantic label is assigned is deleted, The predicate normalization apparatus according to claim 1 or 2, wherein a score is calculated. On the computer,
Morphological analysis of the input document,
Based on the result of morphological analysis, a determination label indicating that the grammatical necessity is different for each functional expression included in the predicate of the document and having different grammatical necessity depending on words existing in the vicinity And grant
Before excluding each of the morpheme sequence constituting the previous description part when at least one of the functional expressions to which the determination label is assigned is included in the previous description part, and the functional expression to which the determination label is assigned For each of the morpheme strings that make up the description part, based on the morpheme N-gram model constructed with pseudo-words that have the surface form of the morpheme as an element for the function word and the surface layer form for the word other than the function word Calculating a score indicating whether or not the functional expression to which the determination label is given is grammatically necessary,
Generate normalized predicates from morpheme sequences selected based on the calculated score
Predicate normalization method for executing processing including the above.   The predicate normalization program for functioning a computer as each means which comprises the predicate normalization apparatus of any one of Claims 1-3.

Images