JP5115239B2 - Character processing device - Google Patents

Description

  The present invention relates to a character processing device and a program for generating a conversion rule for converting a result of text division by morphological analysis or the like.

A technique for extracting keywords from text (a technique for dividing text into words) plays an important role in performing language processing tasks such as text search and text classification. For example, if an appropriate keyword can be assigned to text, highly accurate text search and text classification can be performed.
The process of dividing the text into words (including the process of adding parts of speech) is generally called morphological analysis. The analysis accuracy of the morphological analysis has a practically sufficient value for general text such as newspaper text. However, when targeting texts that contain many technical terms such as medical texts, the word dictionary necessary for morphological analysis is not sufficiently prepared, so the analysis accuracy has reached a practically sufficient level. Is hard to say.

Here, a technique of morphological analysis based on an error driving model has been proposed as a technique for obtaining high analysis accuracy even when text in a field where a dictionary is not maintained (for example, patent literature) 1, Non-Patent Document 1, Non-Patent Document 2).
In such an error driving model, a correct corpus with correct word break information added to text is prepared, and the morphological analysis result of the same text is compared with the word break of the correct corpus, thereby performing morphological analysis processing. Generate a rule to convert the word breaks obtained from to the correct word breaks. By applying this rule to the morphological analysis result, it is possible to obtain a word segmentation result with extremely high accuracy even when text in a field where the word dictionary is not sufficiently prepared.

Japanese Patent Laid-Open No. 2000-04-0085 "Post-processing of morphological analysis using rewrite rules and context information", IPSJ Research Report, NL-126, 1998, p.55-62 "Recognition of unregistered Chinese words based on error-driven model", IPSJ Research Report, NL-134, 1999, p.123-129 Okanohara, “Development of Next-Generation Data Compression Method Using Word Extraction”, [online], 2002 Unexplored Software Creation Project Unexplored Youth Research Report, Internet <URL: http://homepage3.nifty.com/DO/okamito04 .pdf>

  In the morphological analysis method based on the error driving model as described above, a correct corpus needs to be prepared. Here, the creation of the correct corpus generally requires a large amount of man-hours, and in particular, the creation of the correct corpus in a highly specialized field such as the medical field must be performed by an expert. It gets bigger. In addition, a method for extracting keywords using only the statistical value of word likeness without using the correct corpus has been proposed (for example, see Non-patent Document 3). Since the dictionary for morphological analysis that has been used cannot be used at all, only an analysis result with low accuracy can be obtained.

  The present invention has been made in view of the above-mentioned conventional circumstances, and is a morpheme that can obtain a high-accuracy word segmentation result even when a text in a field where a word dictionary and a correct corpus are not prepared is targeted. The purpose is to provide the technology to realize the analysis system.

  According to the first aspect of the present invention, a calculation means for calculating word likelihood indicating the word likeness of a character string composed of one or more characters, and a concatenated character string obtained by concatenating the one or more character strings, First determination means for determining whether or not the first average value that is the average value of word likelihoods of the character string is a first concatenated character string that is below a predetermined first threshold; and the character string in the first concatenated character string A condition in which the second average value, which is the average value of the word likelihood of each character string in the concatenated character string, exceeds the predetermined second threshold, or the second average value of the concatenated character strings with different delimiters Second determination means for determining whether or not the second concatenated character string satisfies a condition in which a value obtained by subtracting the first average value from the condition exceeds a predetermined third threshold value, or a condition that satisfies both of them, and , The delimiter position in the first concatenated character string to the delimiter position in the corresponding second concatenated character string Generating means for generating a conversion rule for conversion, a character processing apparatus, comprising the.

  According to a second aspect of the present invention, in the first aspect of the present invention, the conversion rule generated by the generating means is located immediately before or immediately after the first concatenated character string in the sentence from which the first concatenated character string is extracted. The conversion rule includes the above character string.

  According to a third aspect of the present invention, in the first or second aspect of the present invention, the generating means generates a new conversion rule by generalizing the generated conversion rule.

  According to a fourth aspect of the present invention, in the first to third aspects of the present invention, the character processing device includes a delimiter that delimits a sentence into a plurality of character strings, and the first determination unit is delimited by the delimiter It is determined whether or not a concatenated character string obtained by concatenating one or more character strings in the order in the sentence is a first concatenated character string.

  According to a fifth aspect of the present invention, in the first to fourth aspects of the present invention, the first determining means is a concatenated character in which one or more character strings included in a plurality of sentences are connected in the order in the sentence. Whether the column is a first concatenated character string, the character processing device, from the application means for applying a conversion rule to the entire sentence, and a plurality of conversion rules generated by the generation means, A value obtained by subtracting an average value of word likelihoods of each character string in the sentence based on the separation position before application of the conversion rule from an average value of word likelihoods of each character string in the sentence based on the separation position after application of the conversion rule. Selecting means for selecting a conversion rule exceeding the fourth threshold value.

  According to a sixth aspect of the present invention, there is provided a calculation function for calculating word likelihood indicating a word likelihood of a character string composed of one or more characters and a concatenated character string obtained by concatenating the one or more character strings. A first determination function for determining whether or not the first average value, which is an average value of word likelihoods of each character string, is less than a predetermined first threshold, and a first connected character string In the concatenated character string in which the character string delimiter positions are different, the second average value that is the average value of the word likelihood of each character string in the concatenated character string exceeds a predetermined second threshold, or A second that determines whether or not the second concatenated character string satisfies a condition in which a value obtained by subtracting the first average value from the two average values exceeds a predetermined third threshold value, or a condition that satisfies both of them. The determination function and the delimiter position in the first concatenated character string are converted into the corresponding second concatenated character string. Kicking a program for implementing a generating function for generating a conversion rule for converting the separated position, to the computer.

  According to the character processing device of the first aspect of the present invention, since the conversion rule is generated based on the average value of the word likelihood of each character string in the connected character string, there is no need to prepare a word dictionary or a correct corpus. Therefore, a morphological analysis system that can obtain a high-accuracy word segmentation result is realized by using the generated conversion rule even when the text in a field where a word dictionary or correct corpus is not prepared is targeted. can do.

  According to the character processing device of the second aspect of the present invention, since the application scene of the conversion rule is limited by the character string before and after the first concatenated character string, the conversion rule applied in a more appropriate scene can be generated. it can.

  According to the character processing device of the third aspect of the present invention, since the application scene is alleviated by the generalization of the conversion rule, it is possible to generate a conversion rule that is applied more generally.

  According to the character processing device of the fourth aspect of the present invention, the conversion rule can be generated based on the result of the morphological analysis.

  According to the character processing device of the fifth aspect of the present invention, a more suitable conversion rule can be selected from a plurality of generated conversion rule candidates.

  According to the program of the sixth aspect of the present invention, a computer can function as the character processing device.

The present invention will be specifically described based on an embodiment.
The morpheme analysis system exemplified below is based on the premise of using morpheme analysis means capable of obtaining practically sufficient analysis accuracy (about 99%) when texts in a certain field are targeted. When applied to text in a field (for example, medical field), it is assumed that the analysis accuracy is one step before the actual use level (about 95%). It is also assumed that there is no correct corpus in the text of the latter field. When there is a sufficient amount of correct corpus, it is preferable to construct a morpheme analyzing means by learning from the correct corpus without using the method according to the present invention. Moreover, even if it is applied to texts in fields where accuracy much lower than the actual usage level (such as 90% or less) can be obtained, it is difficult to improve to practically sufficient analysis accuracy by an error driving model.

FIG. 1 is a functional block diagram of a morphological analysis system configured by applying the present invention.
The morpheme analysis system of this example includes a text storage unit 1 that stores a plurality of texts in the same field (medical field in this example), and a morpheme analysis unit 2 that performs morpheme analysis on the text stored in the text storage unit 1. Morphological analysis result storage means 3 for storing morphological analysis results, word likelihood calculation means 4 for calculating a word-likeness value (word likelihood) of an arbitrary character string, and conversion rules for correcting the morphological analysis results A conversion rule generation unit 5 that generates candidates, a conversion rule selection unit 6 that selects an effective conversion rule from conversion rule candidates, and a conversion rule application unit 7 that applies a conversion rule to a morpheme analysis result are provided.

In this example, text in the medical field is stored in the text storage means 1. The morpheme analysis unit 2 performs morpheme analysis on all the texts stored in the text storage unit 1 and divides each text into a plurality of character strings (word candidates) according to the morpheme analysis dictionary. , Each part of speech is assigned and stored in the morphological analysis result storage means 3.
For example, when a morphological analysis is performed on a sentence (text) that “accepts mild and chronic”, a result as shown in FIG. 2 is obtained and stored in the morphological analysis result storage means 3. According to the figure, as a result of the morphological analysis, "mild (noun)", "nobi (verb)", "chronic (noun)", "wo (case particle)", "acknowledge (verb)", ". It can be seen that the punctuation)) was obtained.

The word likelihood calculation means 4 refers to the text stored in the text storage means 1 when an arbitrary character string (character string consisting of one or more characters) is given, and thereby the word of the character string A value (word likelihood) indicating the likelihood is calculated.
Although various definitions of the word likelihood can be considered, in this example, the following definition proposed in Non-Patent Document 3 is used as the word likelihood WT of the character string C.
WT = length * ((log (totalCount) + log (count))
Here, length is the number of characters constituting the character string C, totalCount is the total number of words stored in the morpheme analysis result storage means 3, and count is the character string C in all the texts stored in the text storage means 1. The number of occurrences.
Of course, the word likelihood may be defined by another method, for example, “word length × word appearance frequency” may be used as the word likelihood.

Next, conversion rule candidate generation processing by the conversion rule generation means 5 of this example will be described.
In this example, for example, the following conversion rule proposed in Non-Patent Document 1 is used as a template.
a ₁ ... a _K W ₁ ... W _n b ₁ ... b _L ⇒ a ₁ ... a _K W ₁ '... W _m ' b ₁ ... b _L
Here, a _p (p = 1... K), b _q (q = 1... L), W _r (r = 1... N), and W _s ′ (s = 1... M) are words (character strings), respectively. A ₁ ... A _K , b ₁ ... B _L , W ₁ ... W _n , W ₁ ′... W _m ′ are each a word string (concatenation of one or more character strings). That is, the rules, before and after the word string word string _W 1 ... _{W n} are each _a 1 _{... a K,} in the case of _b 1 ... _{b L,} the _W 1 ... _{W n} in _{W 1} '... _{W m'} It is a rule to convert. In this example, L = K = 1. (Normally, L = K = 1 because of the problem of data sparseness. If a value of 2 or more is used, the application situation of the conversion rule is remarkably limited, so the versatility of the conversion rule is reduced and the practicality is reduced. Because it becomes scarce.)

In Non-Patent Document 1, it is assumed that there is a correct corpus, but in this example, conversion rule candidates are generated by the process shown in the flowchart of FIG. 3 without using the correct corpus.
The conversion rule generation unit 5 receives the word likelihood WT of each character string of the morpheme analysis result stored in the morpheme analysis result storage unit 3 from the word likelihood calculation unit 4.
Then, (concatenated string linked in the order of the extraction source of 1372 or more strings morphological analysis result) morphological analysis result word obtained by scanning from the top row for W 1 _... W _n, the configuration It is determined whether or not the word likelihood average value (WTn) of each character string that is an element is a first concatenated character string that is smaller than a threshold value T1 (a preset non-negative real number). The connected character string and one word before and after it are extracted as the left side of the conversion rule (step S11).

For example, when the average value (WTn) of WT of “Nobi” and “Chronic” is smaller than T1, from the above morphological analysis result, as the left side of the conversion rule, “mild (noun) / nobi (verb) / chronic ( Noun) / O (case particle) "is extracted. In this case, a ₁ ... A _K = “mild (noun)”, W ₁ ... W _n = “Nobi (verb) / chronic (noun)”, b ₁ ... B _L = “((case particle)”).

Next, all the delimiter candidates (concatenated character strings with different delimiter positions) are listed for the first concatenated character strings W ₁ ... W _n obtained in step S11 (step S12).
For example, when the first concatenated character string W ₁ ... W _n is “long / chronic”, as shown in FIG. 4, “long / chronic”, “long / chronic”, “long / chronic”, “long / long / Each delimiter candidate of “sex”, “no / bi / chronic”, “nobi / arrogant / sex”, and “no / bi / arrogant / sex” is obtained.

  Thereafter, for each delimiter candidate, the average value (WTm) of word likelihood of each character string that is a constituent element thereof is larger than a threshold value T2 (a preset non-negative real number), and WTm-WTn is a threshold value T3 (preliminary value). It is determined whether or not the second concatenated character string is larger than the set non-negative real number), and a conversion rule having a right side based on the delimiter candidate determined as the second concatenated character string is generated and added to the conversion rule candidate. Note that if the break candidate does not satisfy the above-described determination condition, no conversion rule is generated for the break candidate (step S13).

For example, if the delimiter candidate satisfying the above determination condition is “no / chronic”, the following conversion rule can be obtained.
Mild (noun) / Nobi (verb) / Chronic (noun) / O (case particle)
⇒ Mild (noun) / no (unknown word) / chronic (unknown word) / (case particle)
That is, a conversion rule is generated that converts the break position in the first connected character string “Nobi / Chronic” into the break position in the corresponding second linked character string “No / Chronic”.
In this example, part-of-speech information is assigned to each word in the same manner as in Non-Patent Document 1, but the part-of-speech is unknown because the part-of-speech has not been identified for the rewrite character string on the right side of the conversion rule. "Unknown word" meaning that is given.

  The above processing is performed on all connected character strings obtained by scanning the morphological analysis results from the top to generate conversion rules. Moreover, the conversion rule production | generation means 5 also generalizes the conversion rule proposed by the nonpatent literature 1. That is, the conversion rule obtained by the above process is generalized to generate a new conversion rule and added to the conversion rule candidate (step S14).

Here, as a generalization of the conversion rule, for example, a conversion rule for the part of speech rather than the word of W ₁ ... W _n is used. That is, rather than the conversion rules applied to match the specific character string such W 1 _... W _n the condition, the conversion rule to be applied on the condition matching of each part of speech corresponding to W 1 _... W _n. For example, b ₁ ... B _L is a conversion rule that ignores (deletes from the rule). That is, the conversion rule is applied regardless of the character string following W ₁ ... W _n . Note that these are merely examples, and various generalization techniques that can improve the versatility of the conversion rules can be employed.

As a result of the above processing, a plurality of conversion rule candidates for the same first concatenated character string can be generated. In this example, as shown in FIG. It is a set of conversion rules.
That is, the conversion rule selection means 6 selects one of the conversion rule candidates generated by the conversion rule generation means 5 (step S21), and the morpheme analysis result stored in the morpheme analysis result storage means 3 is selected. The word likelihood of each character string in the morpheme analysis result storage means 3 before application of the conversion rule is calculated from the average value (WTa) of the word likelihood of each character string consisting of the new break position obtained as a result. It is determined whether or not the value (WTc) obtained by subtracting the average value (WTb) is larger than the fourth threshold value (preset non-negative real number), and a conversion rule satisfying the condition is selected and final conversion is performed. The process (step S22) added to the rule set is repeated for all the conversion rule candidates (step S23). In other words, from the candidates for each conversion rule, the one that shows a certain improvement in the average value of word likelihoods is selected.

  The conversion rule applying means 7 applies each conversion rule included in the conversion rule set obtained from the conversion rule selecting means 6 to the result of the morphological analysis of the arbitrary text by the morphological analysis means 2, and performs the final morphological analysis Get results. In this example, the conversion rules are applied in descending order of the WTc value, and the process ends when there are no more conversion rules to apply.

  As described above, in this example, since the conversion rule is generated based on the average value of the word likelihood of each character string in the connected character string, it is not necessary to prepare a word dictionary or a correct corpus, so the medical field Even if the target is text in a field that does not have a word dictionary or correct corpus, it is possible to generate conversion rules to correct the morphological analysis results. By applying to the results, we have realized a morphological analysis system that can obtain highly accurate word break results.

Here, the conversion rule generation means 5 of this example has a WTm (average value of word likelihood associated with the target break candidate) from the threshold value T2 for each break candidate with different break positions in the first connected character string. If the value obtained by subtracting WTm (average value of word likelihoods related to the first concatenated character string) from WTm is larger than the threshold T2, the delimiter candidate is determined as the second concatenated character string. If either one of the conditions is satisfied, the second concatenated character string may be determined.
In short, it is only necessary to determine that the average value of the word likelihood before and after the change of the delimiter position can be determined as the second concatenated character string. For example, when only the first condition is used, the threshold T2 is set. What is necessary is just to set it as the value which added predetermined value (non-negative real number) to threshold value T1.

  In this example, whether the average value of word likelihood is larger (or smaller) than the threshold is compared, but the average value of word likelihood may be equal to or greater than (or less than) the threshold. For this reason, in this application, being larger than a threshold value or more than a threshold value is expressed as “beyond the threshold value”, and being smaller than or less than the threshold value is expressed as “below the threshold value”.

The expansion of the function of the conversion rule selection means 6 of this example will be described.
For example, a user interface capable of manually adding a new conversion rule or selecting a conversion rule for the selection result of the conversion rule selection means 6 is provided. This makes it possible to generate and select conversion rules by making use of expert knowledge, and to construct a more accurate morphological analysis system.

  For example, when the conversion rule selection means 6 selects a conversion rule, the next conversion rule is selected using not the initial morpheme analysis result but the morpheme analysis result to which the already selected conversion rule is applied. That is, the first selected conversion rule (the optimal conversion rule that improves the average value of word likelihood) is applied to the morpheme analysis result by the morpheme analysis means 2, and each other conversion rule is applied to the result. Is applied to select the next conversion rule, and then this is repeated recursively. Thus, by selecting a recursive conversion rule, it becomes possible to select a conversion rule more suitably.

  For example, when the conversion rule selection means 6 selects a conversion rule, it uses a morpheme analysis result to which the already selected conversion rule is applied instead of an initial morpheme analysis result, and further generates a conversion rule based on the result. The next conversion rule is selected from the conversion rule candidates generated by the means 5. In other words, the conversion rule selected first is applied to the morpheme analysis result by the morpheme analysis unit 2, and based on the result, the conversion rule generation unit 5 generates a new conversion rule candidate, and the next one is generated. A conversion rule is selected, and this is repeated recursively. Thus, it becomes possible to select a conversion rule more suitably by generating and selecting a recursive conversion rule.

FIG. 6 shows a main hardware configuration of the character processing device constituting the morphological analysis system according to this example.
That is, the character processing apparatus of this example includes a CPU 11 that performs various arithmetic processes, a RAM 12 that is a work area of the CPU 11, a ROM 13 that stores a basic control program, a program and various data for realizing the functions according to the present invention. Communication between the HDD 14 to be stored, a liquid crystal display for displaying and outputting information to the user, an input / output I / F 15 that is an interface with a device such as a mouse / keyboard that accepts input of information from the user, and other devices It is comprised with the computer which has hardware resources, such as communication I / F16 which is an interface to perform.

  Then, the program according to the present invention is read from the HDD 14, loaded into the RAM 12, and executed by the CPU 11, whereby each functional unit of the character processing device according to the present invention is realized by a computer. In this example, the morphological analysis means 2 constitutes a delimiting means, the word likelihood calculation means 4 constitutes a calculation means, and the conversion rule generation means 5 constitutes a first determination means, a second determination means, and a generation means. The conversion rule selection means 6 constitutes a selection means, and the conversion rule application means 7 constitutes an application means.

  Note that the program according to the present invention is provided to the practitioner of the present invention, for example, in a format in which an external storage medium such as a CD-ROM storing the program is distributed or distributed via a network. Each functional unit according to the present invention is not limited to a mode realized by a software configuration as in the present example, and may be configured by a dedicated hardware module. In addition, each functional unit according to the present invention is not limited to an aspect provided in one computer as in this example, and may be provided in a distributed manner in a plurality of computers.

It is a functional block diagram of the morphological analysis system concerning one embodiment of the present invention. It is a figure which illustrates the morphological analysis result concerning one embodiment of the present invention. It is a figure which shows the conversion rule production | generation process which concerns on one Embodiment of this invention. It is a figure which illustrates the division | segmentation candidate which concerns on one Embodiment of this invention. It is a figure which shows the conversion rule selection process which concerns on one Embodiment of this invention. It is a hardware block diagram of the character processing apparatus which concerns on one Embodiment of this invention.

Explanation of symbols

  1: text storage means, 2: morpheme analysis means, 3: morpheme analysis result storage means, 4: word likelihood calculation means, 5: conversion rule generation means, 6: conversion rule selection means, 7: conversion rule application means

Claims (6)

Calculating means for calculating word likelihood indicating the word likeness of a character string composed of one or more characters;
A concatenated character string obtained by concatenating one or more character strings is a first concatenated character string in which a first average value that is an average value of word likelihood of each character string in the concatenated character string is lower than a predetermined first threshold value. First determination means for determining whether or not,
The second average value, which is the average word likelihood of each character string in the concatenated character string in the concatenated character string in the first concatenated character string, exceeds a predetermined second threshold. Whether or not the second concatenated character string satisfies either the condition or the condition that the value obtained by subtracting the first average value from the second average value exceeds a predetermined third threshold value, or the condition that satisfies both of them. Second determination means for determining
Generating means for generating a conversion rule for converting a break position in the first linked character string into a break position in the corresponding second linked character string;
A character processing device comprising:   The conversion rule generated by the generating means is a conversion rule including one or more character strings located immediately before or immediately after the first concatenated character string in the sentence from which the first concatenated character string is extracted. The character processing device according to claim 1.   The character processing apparatus according to claim 1, wherein the generation unit generates a new conversion rule by generalizing the generated conversion rule. The character processing device includes a delimiter that delimits a sentence into a plurality of character strings,
The first determination means determines whether or not a concatenated character string obtained by concatenating one or more character strings delimited by the delimiter along a sequence in the sentence is a first concatenated character string. The character processing device according to any one of claims 1 to 3. The first determination means determines whether or not a concatenated character string obtained by concatenating one or more character strings included in a sentence divided into a plurality along a sequence in the sentence is a first concatenated character string;
The character processing device includes:
Applying means for applying a conversion rule to the whole sentence;
From the plurality of conversion rules generated by the generation means, each character in the sentence based on the delimiter position before application of the conversion rule from the average word likelihood of each character string in the sentence based on the delimiter position after application of the conversion rule A selection means for selecting a conversion rule in which a value obtained by subtracting an average value of word likelihoods of the column exceeds a fourth threshold;
The character processing device according to claim 1, comprising: On the computer,
A calculation function for calculating a word likelihood indicating the word likeness of a character string composed of one or more characters;
A concatenated character string obtained by concatenating one or more character strings is a first concatenated character string in which a first average value that is an average value of word likelihood of each character string in the concatenated character string is lower than a predetermined first threshold value. A first determination function for determining whether or not
The second average value, which is the average word likelihood of each character string in the concatenated character string in the concatenated character string in the first concatenated character string, exceeds a predetermined second threshold. Whether or not the second concatenated character string satisfies either the condition or the condition that the value obtained by subtracting the first average value from the second average value exceeds a predetermined third threshold value, or the condition that satisfies both of them. A second determination function for determining
A generation function for generating a conversion rule for converting a break position in the first linked character string into a break position in the corresponding second linked character string;
A program to make a computer realize.

Images