JP6619932B2 - Morphological analyzer and program - Google Patents

Description

  The present invention relates to a morpheme analyzer and a program for outputting a morpheme string corresponding to an inputted sentence.

  Text-to-speech conversion (Text-To-Speech, TTS) is a typical usage form of speech synthesis technology. Text-to-speech conversion is a process of synthesizing a speech waveform corresponding to input text. In the following, this series of processes is roughly divided into two processes: a process of analyzing input text to generate information on how to read the text, and a process of synthesizing a speech waveform from the information on how to read. Also, it is assumed that the input is a Japanese kanji kana mixed sentence.

  Hereinafter, a symbol used for expressing information on how to read is referred to as a speech synthesis symbol. There are various forms of symbols for speech synthesis. Here, we assume that the phonetic information that composes a series of speech and the prosodic information that is mainly expressed as a pose or voice pitch. . An example of such a symbol for speech synthesis is JEITA (Electronic Information Technology Industry Association) standard IT-4006 “Symbol for Japanese Text Speech Synthesis” (see Non-Patent Document 1). Although it is difficult to express even emotional expressions of speech with these symbols alone, it contains at least the information necessary to describe the linguistic features of normal speech.

  On the other hand, the process of synthesizing the speech waveform is performed so that the waveform according to the speech synthesis symbol is synthesized. Therefore, in order to realize accurate reading of the Japanese text, it is only necessary to create an accurate speech synthesis symbol corresponding to the Japanese kanji kana mixed text.

  The process of generating symbols for speech synthesis from arbitrary Japanese text is to divide Japanese Kanji kana mixed sentences into the smallest units that have meaning in linguistic expressions called morphemes, add readings for each morpheme, It can be realized by referring to the morpheme information appropriately, inserting prosodic boundaries such as poses as necessary, and connecting them. At this time, the reading of each morpheme is created and stored in advance as morpheme dictionary information (see Patent Document 1).

  However, the morpheme does not have to be as defined in linguistic terms, and may be a unit appropriately divided for performing a series of processing. For example, in order to more appropriately process the arrangement of morphemes, a phrase composed of a plurality of morphemes (such as a compound noun phrase) may be regarded as one morpheme for convenience. Therefore, in the following, a morpheme refers to a sequence of characters (character string) that is appropriately set as a minimum unit for processing from the viewpoint of its use, and all sentences are concatenated with this character string. It can be configured with.

  As described above, a process for dividing a sentence into morphemes is generally called a morpheme analysis process, and is used not only for speech synthesis processing but also for extraction of sentence components. Since the correctness of reading of the TTS system strongly depends on the accuracy of the morphological analysis, the TTS system requires highly accurate morphological analysis. On the other hand, unlike the case of extracting data from a large amount of text data on the World Wide Web (WWW), the TTS system normally does not need to process a large amount of text data in a short time.

  For example, even if the processing time of one sentence requires about 0.1 seconds, which delays the processing delay time in the TTS system by 0.1 seconds, the processing time is not a problem in many cases. That is, high-speed processing is unnecessary in comparison with a morphological analyzer for processing a large amount of text. On the other hand, considering the need for a TTS system on a mobile terminal such as a smartphone, it is more important to reduce the size of the system than to increase the speed of the system. That is, from this requirement, the morphological analyzer for the TTS system can be designed differently from the morphological analyzer for processing a large amount of text.

  First, as a morphological analysis method, a method based on the minimum cost method will be described below. In the morpheme analysis by the minimum cost method, first, an occurrence cost function reflecting the appearance frequency of each morpheme and a concatenated cost function representing ease of connection of consecutive morpheme are defined in advance. Then, an appropriate morpheme sequence is obtained by searching the morpheme registered in the morpheme dictionary for a morpheme sequence that matches the input text and minimizes the cost of the entire sentence. Usually, the occurrence cost function is defined so that the morpheme having the higher appearance frequency and the concatenated cost function have the smaller value as the morpheme sequence that is more easily connected.

  That is, when the morpheme sequence is M = (m1,..., Mn), the occurrence cost function is Ct (m), and the concatenation cost function is Cc (m (i−k + 1),..., Mi), the total cost ΣCt + ΣCc is minimum. The morpheme analysis process is performed by obtaining the morpheme string M, that is, argmin (ΣCt + ΣCc). Here, it is assumed that the concatenated cost function is determined by an array of k morphemes.

  When the cost function is defined in this way, the partial series constituting the optimal whole series in terms of cost are optimal in terms of cost even when only the partial series is viewed. Therefore, the subsequence that is not optimal in terms of cost does not need to be considered in the search because it is not a component of the optimal overall sequence. As described above, the optimum sequence search method that proceeds without considering a partial sequence that is not likely to form an optimum sequence is generally called dynamic programming, and can search for the optimum sequence efficiently.

  Among the components of the cost function, information on the occurrence cost can be held as the contents of the morpheme dictionary. On the other hand, the connection cost can be obtained by creating a table called a connection table in advance and using the values in the table. However, it is generally difficult to create and use a table of all morpheme combinations because there are many types of morphemes. Therefore, for example, attention is paid to only the morpheme part-of-speech type, and attention is paid to the morpheme class, and the connection table between classes is also used. In some cases, these functions are defined as having larger values. In that case, a morpheme sequence having the largest value of the entire sentence is searched.

  In the morpheme string search process in morpheme analysis, it is preferable to examine all possible morpheme sequences. Therefore, in normal morpheme analysis, in order to obtain a morpheme candidate, a partial character string starting from an arbitrary position in a sentence is used as a search key, and the morpheme registered in the morpheme dictionary is equal to the first partial character string of the key. The process of extracting all morphemes is repeated. Such a search is generally referred to as a common prefix search. As a data structure that expresses this relatively efficiently, trie is known.

  Here, a trie is a multi-tree structure for storing a plurality of character strings, and here, it is assumed that each character is stored as a tree branch in order from the first character of each character string. In the trie, a common prefix between character strings is shared on the tree structure, and therefore, all registered words that are prefixes of the character strings to be searched are arranged on one path of the tree structure. That is, the common prefix search can be realized by tracing the tree in the leaf direction along the search key from the root of the trie.

JP 2009-129258 A

"Symbol for Japanese text-to-speech synthesis" JEITA standard IT-4006, March 2010

  In general, Japanese text cannot be avoided to some degree. For example, kanji and kana, hiragana and katakana, new and old kanji in kanji, and substitution and misuse of kanji.

  If the comparison between the input character string performed by the morpheme analyzer and the notation character string registered in the morpheme dictionary (representing the morpheme) is based on perfect match, it is necessary to construct a morpheme dictionary that takes this fluctuation into account. . As such a method, Patent Document 1 refers to a method of writing mixed characters (in this case, for example, a method of writing only a difficult kanji such as outside a common kanji within one morpheme. ) Is disclosed as a method for realizing a morphological analysis of a document including) when a morpheme dictionary used in the morpheme analyzer is created, converting a difficult character included in the morpheme notation into kana and writing it as a morpheme. ing. According to this method, it is possible to deal with fluctuations in notation such as cross writing, but the number of words registered in the morpheme dictionary increases and the size of the morpheme dictionary increases.

  In addition, when comparing a notation character string registered in a morpheme dictionary and an input character string, a method of applying a more complicated comparison rule rather than a perfect match between characters is conceivable. However, this method can handle hiragana and katakana, new and old fonts, simple substitution and misuse, but it is difficult to write a rule description corresponding to a mixed writing case. For example, if rules for matching kanji and kana character strings are defined, rules for many kanji characters are required in practice, and the total number of rules is on the order of thousands to tens of thousands. As a result, the search for application rules in character comparison itself requires a structure like a trie, which makes the configuration of the apparatus more complicated.

  As a similar method, a search method using an intermediate representation can be considered. In this method, even if different characters are used, for example, hiragana and katakana, the relationship between old and new fonts, etc., it is a highly related character, or between characters that are similar in shape and misused or substituted For the above, an intermediate representation to which the same code is assigned is created, both the input and the morpheme in the morpheme dictionary are converted into the intermediate representation, and the morpheme analysis is performed on the intermediate representation.

  However, this method is also not easy to deal with cases such as mixed writing. Without the uniqueness of the intermediate representation, there are a plurality of possibilities for the morphological analysis target, making morphological analysis processing difficult. On the other hand, the uniqueness of the intermediate expression can be realized by using, for example, a kana reading (for example, “ABC” for “ABC”, etc.) as the intermediate expression, but the homonym and the like cannot be distinguished, and the morphological analysis result The possibility increases, and as a result, the accuracy of morphological analysis decreases.

  The present invention has been made in view of such circumstances, and while suppressing an increase in the size of the morpheme dictionary, it is possible to consider the morpheme notation hypothesis including intermediate expressions, and to avoid fluctuations in notation such as cross writing. Accordingly, an object of the present invention is to provide a morpheme analyzer and a program capable of efficiently and accurately performing morpheme analysis.

  (1) In order to achieve the above object, a morpheme analyzer of the present invention is a morpheme analyzer that outputs a morpheme analysis result corresponding to an input sentence, and extracts a partial character string from the input sentence. Using a partial character string generation unit to generate, an intermediate expression generation unit that generates, as an intermediate representation, one or more expressions that can be converted from the generated partial character string as an intermediate stage, and an morpheme dictionary A morpheme enumeration unit that enumerates the generated intermediate representation and a morpheme corresponding to the concatenation of the intermediate representations, and a morpheme sequence search that searches and outputs a morpheme sequence that satisfies a predetermined condition among the enumerated morphemes And a section.

  This makes it possible to consider the morpheme notation hypothesis including intermediate representations while suppressing the increase in the size of the morpheme dictionary, and to perform efficient and highly accurate morpheme analysis in response to fluctuations in notation such as mixed writing. it can.

  (2) In the morpheme analyzer of the present invention, at least one of the intermediate expression dictionary and the morpheme dictionary is a code in which character elements that can form a partial character string and intermediate expressions corresponding to the character elements are alternately arranged. It is characterized by storing columns. As a result, a set of character element sequences and corresponding intermediate representations can be stored in one area rather than separately, and the dictionary data structure can be simplified.

  (3) The morpheme analyzer of the present invention is characterized in that one or all of the intermediate representation dictionary and the morpheme dictionary are used as all or part of the other. Thereby, the amount of memory required for storage can be reduced, and morpheme analysis can be performed efficiently while suppressing an increase in the size of the morpheme dictionary.

  (4) Further, in the morpheme analyzer according to the present invention, one of the intermediate representation dictionary and the morpheme dictionary is a code string obtained by alternately arranging a character element and an intermediate expression corresponding to the character element as first dictionary data. And the other is a character element and an intermediate corresponding to the character element with respect to a character element string from the beginning of a part or all of the character element string stored as the first dictionary data to a predetermined number of character elements. It is characterized in that the storage order of expressions is stored as a code string alternately arranged in the reverse order to the first dictionary data.

  (5) The program of the present invention is a program for outputting a morphological analysis result corresponding to an input sentence by causing the computer of the morphological analysis apparatus having a dictionary to execute the partial character from the input sentence. A process of cutting out and generating a sequence, a process of generating one or more expressions that can be converted as an intermediate stage from the generated partial character string using an intermediate expression dictionary, and a morpheme dictionary using the morpheme dictionary A process of enumerating generated intermediate representations and morphemes corresponding to concatenation of the intermediate representations, and a process of searching for and outputting a morpheme string satisfying a predetermined condition among the enumerated morphemes. It is said.

  This makes it possible to consider the morpheme notation hypothesis including intermediate representations while suppressing the increase in the size of the morpheme dictionary, and to perform efficient and highly accurate morpheme analysis in response to fluctuations in notation such as mixed writing. it can.

  According to the present invention, it is possible to consider the morpheme notation hypothesis including intermediate representations while suppressing an increase in the size of the morpheme dictionary, and to efficiently and accurately perform morpheme analysis corresponding to notation fluctuations such as cross writing. It can be carried out.

It is a block diagram which shows the morphological analyzer of this invention. It is a figure which shows an example of the data structure of this invention. It is a figure which shows an example of the data structure of this invention. It is a flowchart which shows an example of the process by the morphological analyzer of this invention. It is a flowchart which shows an example of the process by the morphological analyzer of this invention. It is a flowchart which shows an example of the process by the morphological analyzer of this invention. It is a figure which shows an example of the data structure of this invention.

  Next, embodiments of the present invention will be described with reference to the drawings. In the following description, morphemes having different intermediate representations are treated as different morphemes even if the written character strings are the same.

[First Embodiment]
(Configuration of morphological analyzer)
FIG. 1 is a block diagram showing a morphological analyzer 100. The morpheme analyzer 100 is composed of a PC, for example, and outputs a morpheme string corresponding to the inputted sentence. The morpheme analyzer 100 converts a partial character string of an input character string into a predetermined intermediate representation once by the intermediate representation generation unit 120 in enumerating morpheme candidates in the morpheme analysis, and enumerates morphemes having the same intermediate representation as morpheme candidates. To do. The morpheme analyzer 100 includes a partial character string generation unit 110, an intermediate representation generation unit 120, a morpheme enumeration unit 130, and a morpheme string search unit 140.

  The partial character string generation unit 110 cuts out and generates a partial character string from the input sentence. The intermediate expression generation unit 120 uses the intermediate expression dictionary 125 to generate one or more expressions that can be converted from the generated partial character string as an intermediate expression. The intermediate expression is an expression that can be converted at an intermediate stage when a morphological analysis is performed on a character string. For example, the intermediate expression is a reading expressed in katakana for a character string. In this way, by using the intermediate representation, it is possible to consider the morpheme notation hypothesis including the intermediate representation while suppressing an increase in the size of the morpheme dictionary, and efficiently and accurately cope with fluctuations in notation such as cross writing. High morphological analysis can be performed.

  The intermediate representation generation unit 120 converts all partial character strings for the input sentence into an intermediate representation according to a predetermined conversion rule. The conversion rule is defined by the intermediate expression dictionary 125 in order to convert each character element and the morpheme included in the morpheme dictionary (a string of character elements constituting the morpheme) into an intermediate expression. If the partial character string is registered in the intermediate representation dictionary 125, the intermediate representation dictionary 125 is accessed, and the pair of the partial character string and the intermediate representation is sent to the morpheme enumeration unit 130. At that time, it may be sent together with corresponding position information in the input character string (from what character to what character in the input character string, etc.).

  The morpheme enumeration unit 130 uses the morpheme dictionary 135 to generate a character string that forms a morpheme corresponding to the generated intermediate representation and connection of the intermediate representations. The connection of intermediate expressions means an intermediate expression created by connecting a sequence of intermediate expressions so as to satisfy a predetermined condition. The predetermined condition is, for example, a predetermined number.

  The morpheme enumeration unit 130 accesses the morpheme dictionary 135, and if there is a morpheme whose intermediate representation matches, outputs the intermediate representation and morpheme information as morpheme candidate information and sends it to the morpheme string search unit 140. At this time, a partial character string in the input sentence may be output for use in processing of the morpheme string search unit 140 described later. If necessary, partial character strings may be concatenated and output so as to correspond to the intermediate representation.

  The morpheme information includes the corresponding position information for the input character string and the morpheme character string. As will be described later, when comparing a written character string and a morpheme character string in a morpheme string search, it is necessary to include information for restoring the written character string in the morpheme information. If the morpheme string search unit 140 has a structure that can refer to the input character string, only the corresponding position information described above may be used.

  As an example, consider using a katakana character string representation as an intermediate representation. Hereinafter, the minimum component for a written character string, such as an input character string, is referred to as a character element. In many cases, one character element corresponds to one character, but it may be preferable to treat a character string composed of two or more characters as one character element. For example, in a system in which a dakuten or semi-dakuten is coded as an independent character, a kana character and a dakuten or semi-dakuten sequence are not considered two-character elements, but two character sequences are considered one-character elements. However, in the trie structure, for example, the depth of the tree becomes shallower and more efficient processing can be performed.

  The morpheme string search unit 140 searches for a morpheme string that satisfies a predetermined condition among the listed morphemes. As a result, it is preferable to search for and output an optimal arrangement of morpheme candidates. For example, the morpheme sequence search unit 140 determines a list satisfying a predetermined condition for the morpheme candidates output by the morpheme enumeration unit 130 by a minimum cost method or the like, and combines it with the morpheme information of each morpheme as a morpheme analysis result. It is preferable to output.

  In the intermediate expression generation, a plurality of types of intermediate expressions may be generated from one character element string. For example, when the intermediate representation is a kana representation, a plurality of intermediate representations are generated if there are a plurality of kana representations (readings) for a character element or a character element string. Similarly, in the morpheme candidate enumeration, a plurality of morpheme candidates can be generated from one intermediate representation. For example, a plurality of homonyms are generated from one intermediate expression.

(Concrete example)
Hereinafter, as a specific example of performing morphological analysis for “recovery <Bankai>” (<> is an intermediate expression notation, and the same applies hereinafter), a case where the morpheme dictionary 135 does not have a morpheme of “Bankai” as a mixed expression. Think.

  When the character strings “BA <B>”, “N <N>”, “Recovery <Bankai>”, “Time <Cai>” are registered as morphemes, the conventional method uses the input character string “Ban times”. The morpheme candidates can be obtained only by connecting the three morphemes “ba”, “n”, and “times”. Note that “ba” and “n” are usually registered as particles, impression verbs, etc., respectively.

  On the other hand, in the intermediate expression generation unit 120, the intermediate expressions “ba”, “n”, and “chi” are first generated from “ba” “n” “times”. For this purpose, a conversion rule from a morpheme (a string of character elements constituting a morpheme) included in a character element or a morpheme dictionary needs to be defined. For example, a conversion rule for all character elements and a conversion rule from character element strings corresponding to all morphemes included in the morpheme dictionary may be provided.

  Next, in the morpheme enumeration unit 130, the morphemes corresponding to the intermediate expressions “ba”, “ban”, “bankai”, “n”, “nkai”, and “chi”, which are all possible connections of the continuous intermediate expressions, are obtained from the morpheme dictionary 135. Enumerate. As a result, “bankai” to “recovery” are also listed as morpheme candidates. Instead of considering all possible connections, for example, the maximum number of connections may be determined to reduce the number of morpheme dictionary references.

  In this way, once the intermediate representation is passed, it is possible to consider the possibility that the morpheme of “recovery” is included in the morpheme analysis process for the input of “bounce”.

(Reduction of the effect of processing via intermediate representation)
In the search for the morpheme string, for example, the edit distance from the notation character string may be added to the cost function to make it difficult to select the morpheme candidate in which the character element replacement has occurred. For example, if the insertion, deletion, and replacement of characters are each at distance 1, the editing distance between the morpheme character string “recovery” and the input character string “bump” is 2. A value obtained by multiplying the edit distance D by an appropriate weighting factor Wd and added to the original morpheme generation cost Ct (m) is expressed by the following equation (1).

Ct ′ (m) = Ct (m) + Wd · D (1)
By using the expression (1) as the occurrence cost in the morpheme analysis, the morpheme candidate generated by the replacement can be hardly included in the final morpheme analysis result. In this example, since the occurrence costs of morphemes “ba”, “n”, and “times” are generally considered to be relatively high (this corresponds to the low appearance frequency of the morphemes), As the analysis result, “recovery” generated via the intermediate expression is more easily selected than “ba”, “n”, and “recovery”.

  Further, the intermediate representation itself may be considered in the calculation of the occurrence cost and the connection cost. For example, the appearance probability of an intermediate representation is determined in advance from a large amount of text, and the occurrence probability for an occurrence cost function or an intermediate representation pair of adjacent morphemes that the occurrence cost decreases as the intermediate representation has a higher appearance probability. By using a concatenation cost function that reduces the concatenation cost as the appearance probability of the intermediate representation pair increases, avoid unnatural morphological analysis results from the intermediate representation. be able to.

[Second Embodiment]
The intermediate expression dictionary 125 used in the first embodiment has a data structure represented by a table of pairs of character element strings constituting the notation character string and corresponding intermediate expressions, but the same contents are represented in different expression forms. A data structure may be used.

  That is, at least one of the intermediate representation dictionary 125 and the morpheme dictionary 135 preferably stores a code string in which intermediate representations corresponding to character elements and character elements that can form a partial character string are alternately arranged. As a result, a set of character element sequences and corresponding intermediate representations can be stored in one area rather than separately, and the dictionary data structure can be simplified.

  For example, in the intermediate expression dictionary 125, “recovery <bankai>” can be expressed in the form of “recovery / ban / return / chi” (where “/” represents a delimiter between a notation character string and an intermediate expression notation). By using this representation, a set of character elementary sequences and corresponding intermediate representations can be stored in one area rather than separately, and the dictionary data structure can be simplified. The same applies to the representation form of the intermediate representation and the morpheme character string in the morpheme dictionary 135.

  Furthermore, if these are stored in the form of a trie, the common portion from the beginning is shared by each dictionary by the information encoded by arranging them alternately, and the amount of memory required for storage in the conversion table can be reduced. That is, all or part of one of the intermediate representation dictionary 125 and the morpheme dictionary 135 can be used as all or part of the other. Thereby, it is possible to efficiently perform morphological analysis while suppressing an increase in the size of the dictionary.

  2 and 3 are diagrams illustrating an example of the data structure. FIG. 2 shows an example of a dictionary trie for conversion from a notation character string to an intermediate representation notation, and FIG. 3 shows an example of a dictionary trie for conversion from an intermediate representation to a morpheme character string. ing.

  Here, “I / I”, “Ka / K”, “Ba / Ba”, “N / N”, “Grin / Ban”, “Grind / Bankai”, “Grind / Hiku”, “Tai / Kai”, “Turn / Mawas”, “Turn” / Mawal "conversion information is represented by a trial.

  Similarly, when a morpheme character string is included as one of the morpheme information in the morpheme dictionary 135, an intermediate representation and a morpheme character in the form of an expression in which character elements constituting the morpheme character string and intermediates corresponding to the character elements are alternately arranged. A set of columns can be stored.

(Access to dictionary)
When the expression by trial is used, it may be necessary to search for a plurality of paths for accessing the dictionary. As an example, access to a dictionary having a data structure in which character strings and intermediate representations according to this embodiment are alternately arranged will be described. 4 to 6 are flowcharts illustrating an example of processing of the morphological analyzer 100. Here, an example of a search flow in the case where character elements and intermediate representations are alternately arranged is shown. In the following processing, a node set is represented by {Ng}, {Nc}, {Ni}, etc.

  First, a root node is substituted for {Ng}, and {Ni} is an empty set (step S1). Then, it is determined whether or not there is an element in {Ng} (step S2). If there is an element in {Ng}, the process proceeds to step S3. If {Ng} is an empty set, the process proceeds to step S11. In step S3, one node is extracted from {Ng}, and the extracted node is set to n. The extracted node is deleted from the set. In the process of extracting a node, the extracted node is similarly deleted from the set to which it belongs. Then, all child nodes of the node n are substituted into the node set {Nc} (step S4).

  It is determined whether or not there is an element in {Nc} (step S5). If there is an element, the process proceeds to step S6, and if there is no element, the process returns to step S2. In step S6, one node is extracted from {Nc}, and that node is set as c. In step S7, it is determined whether or not the code linked to the branch from the node n to the node c matches the search target character element, and if it matches the search target character element, the process goes to step S8. move on. If not, the process proceeds to step S9. In step S8, node c is added to {Ng}, and the process returns to step S5.

  On the other hand, in step S9, it is determined whether or not the code linked to the branch from the node n to the node c is a delimiter. If it is a delimiter, the process proceeds to step S10, and if it is not a delimiter, the process returns to step S5. . In step S10, node c is added to {Ni}, and the process returns to step S5.

  In step S11, it is determined whether or not there is an element in {Ni}. If there is an element, the process proceeds to step S12. If {Ni} is an empty set, the process proceeds to step S21.

  In step S12, one node is extracted from {Ni}, and the extracted node is set to n. All child nodes of node n are substituted into {Nc} (step S13). It is determined whether or not there is an element in {Nc} (step S14). If there is an element, the process proceeds to step S15. If it is an empty set, the process returns to step S11.

  In step S15, one node is extracted from {Nc}, and that node is set as c. It is determined whether or not the node c is a leaf node (step S16). If it is a leaf node, the process proceeds to step S17. If it is not a leaf node, the process proceeds to step S18. In step S17, the character element / intermediate expression in the branch from the root node to node c is output as a search result, and the process returns to step S14.

  In step S18, it is determined whether or not the code linked to the branch from the node n to the node c is a delimiter. If it is a delimiter, the process proceeds to step S19. If not, the process proceeds to step S20. In step S19, node c is added to {Ng}, and the process returns to step S14. In step S20, node c is added to {Ni}, and the process returns to step S14. In step S21, when {Ng} is an empty set, the process ends. If it is not an empty set, the process returns to step S2.

[Third Embodiment]
In the first embodiment, the intermediate expression generation process uses the intermediate expression dictionary 125, and the morpheme enumeration process uses the morpheme dictionary 135, but the intermediate expression generation process may use the morpheme dictionary 135.

  If the morpheme information in the morpheme dictionary 135 includes a morpheme character string, the intermediate representation corresponding to the morpheme character string required by the intermediate representation generation unit 120 is obtained from the morpheme dictionary 135 in the morpheme enumeration unit 130. be able to.

  In addition, the conversion from the intermediate representation required by the morpheme enumeration unit 130 to the morpheme character string is performed on the notation character string that is also a morpheme character string in the conversion table from the notation character string to the intermediate representation that the intermediate representation dictionary 125 has. It can also be performed by searching for a notation character string having a predetermined intermediate expression and a sign in the table. Thereby, the whole dictionary size can be reduced.

[Fourth Embodiment]
One of the intermediate representation dictionary 125 and the morpheme dictionary 135 is stored as first dictionary data in a code string in which intermediate representations corresponding to character elements and character elements are alternately arranged, and the other is stored as first dictionary data. The sequence of storing intermediate representations corresponding to character elements and character elements in the reverse order of the first dictionary data for character element strings from the beginning of a part or all of the character element strings to a predetermined number of character primes. You may store with the code sequence put in order.

  As a result, a reverse code sequence can be used for the data structure in the lower part of the tree, and an increase in processing amount can be suppressed. For example, one of the trie used to convert a notation character string to an intermediate representation and the trie used to convert an intermediate representation to a morpheme character string (configuration preceded by an intermediate representation notation) is from the first to the second character prime , And the other try can be used for a search beyond that.

  In the conversion between the notation character string and the intermediate representation, the tree is traced from the root (upper side in the figure) to the leaf direction (downward in the figure) in the trie data structure. Therefore, the intermediate expression generation unit 120 may use a “grind / ban / times / chi” trie arranged in the order of character elements and intermediate expressions. Further, in the morpheme dictionary 135 accessed by the morpheme enumeration unit 130, it is sufficient to use “Ban / Grin / Kai / Time” trials arranged in the order of intermediate representation and character element.

  However, as a general trend, the number of branches of a tree decreases as it approaches the leaves of the tree, and therefore the amount of processing usually does not increase so much even if the tree is traced down on a trial basis. For example, when converting from an intermediate representation to a morpheme character string, when using a “trim / bang / times / chi” structure trie, it is necessary to skip the character element first to obtain the intermediate representation. When skipping, it is necessary to follow all branches at that location, but even when such processing is performed, the amount of processing does not increase so much compared to the upper part of the tree because there are fewer branches in the lower part of the tree. Therefore, one of the trie used for conversion from the notation character string to the intermediate representation and the trie used for conversion from the intermediate representation to the morpheme character string is a small trie, for example, from the first to the second character prime, and beyond The other trie may be used in the search. As a result, the overall size can be further reduced, and a trial search is required, but the increase in the processing amount is limited.

  For example, it is preferable in terms of throughput that the downward search on a trial basis is a shallow search in the depth direction. Therefore, in general, it is preferable to create a trie having a “grind / van / time / chi” structure in which a character element precedes an intermediate expression as in the conventional case. Further, in the configuration in which the other intermediate expression is preceded, a method of making a try up to the first character element of the morpheme character string is preferable. In the above example, a trie is created for the data that has been cut up to “Ban / Grin”.

  In the case of the above example, when converting from the intermediate representation to the morpheme character string, in the case of the above example, from the state where the “van / grind” leaf (rightmost in the representation in this document) is accessed, the next processing is performed. In addition, although it is necessary to access a node corresponding to “time” in “grinding / van / time / chi”, the character elements up to “times” which are information necessary to reach the node (that is, “ This is easy to access since the “rear” and its intermediate representation (“van”) have already been obtained. FIG. 7 is a diagram showing an example of the data structure, and shows a trie created up to the first written character corresponding to FIG.

  Alternatively, the location information for directly referencing the “time” node in the “ground / van / time / chi” trie without following the route of the trie is used as the leaf node of the “van / ground” trie. It is also possible to use a method of storing and directly accessing the intermediate node of the trie using this.

[Other Embodiments]
(Other examples of intermediate expressions)
In the above description, Katakana notation is used as the intermediate representation, but other representations can be used as the intermediate representation. For example, pronunciation notation by IPA (International Phonetic Alphabet) may be used.

  Alternatively, an intermediate expression may be used in which a set of characters composed only of kana and common kanji is used and kanji characters are written as much as possible. According to the above method, in the morpheme analysis, the occurrence cost function of the morpheme considers both the written string and the intermediate representation, and more accurately the morpheme that is easy to use the old font and the morpheme that is difficult to use like the proper noun. Can be distinguished. For example, “Sakurai” as a place name is rarely written as “Sakurai”, and the notation “Sakui” is likely to have a meaning as a personal name, but “Sakurai” is a place name. There is no possibility of being there.

  The morphological analysis based on the minimum cost method ultimately considers the arrangement of morphemes and makes a comprehensive decision. However, in the above method, “Sakurai” and “Sakurai” are often used, and the place names As “Sakurai” is registered in the morpheme dictionary, the place name “Sakui”, which is rarely used, is not registered in the morpheme dictionary. it can. Thus, for example, when a morpheme such as “city” or “town” immediately follows a place name, using a concatenation cost function defined so that the connection cost between the two is smaller, the place name is not the person name. Morphemes can be selected. However, if the intermediate representation is too abstract, the number of morpheme possibilities that should be taken into account during processing increases.

  This can be a problem, especially with proper nouns with many morphemes of the same sound. For example, in the above example, if katakana notation is used for the intermediate expression, the intermediate expression for “Sakurai” becomes “Sakurai”, and it is necessary to consider the possibility of more morphemes such as “Sakurai”. In the above method, this possibility is eliminated by searching for the optimal cost solution by reflecting the number of character substitutions in the occurrence cost, but a lower abstraction such as “Sakurai” is used as the intermediate representation. By using it, while the response to the notation fluctuation is limited, the number of possibilities to be considered in the subsequent search is reduced, and more efficient processing is possible in terms of processing amount.

(Example without delimiter)
In the second and fourth embodiments described above, the delimiter is used in the code string expression in which the character elements and the intermediate expressions are alternately arranged. However, the arrangement is not limited to this. For example, when a code representing a character element and a code representing an intermediate representation are completely independent at the code position (code value), a delimiter is not necessary.

(program)
The above operation can be realized by causing a CPU mounted on the morphological analyzer 100 to execute a program, for example. This program can be distributed in a state of being recorded on a recording medium. In addition, such a program is transmitted by a computer as a transmission device via a transmission medium such as a communication network including a public telephone line, a dedicated telephone line, a cable TV line, a wireless communication line, etc. constituting the network. Can be distributed by receiving the received signal.

100 Morphological Analyzer 110 Partial Character String Generation Unit 120 Intermediate Representation Generation Unit 125 Intermediate Representation Dictionary 130 Morphological Enumeration Unit 135 Morphological Dictionary 140 Morphological Sequence Search Unit

Claims (4)

A morpheme analyzer that outputs a morpheme analysis result corresponding to an input character string,
A partial character string generation unit that generates a cut-out partial character string from an input character string that is a kanji kana mixed sentence;
Using the intermediate expression dictionary that stores the character element string that can be a partial character string of the kanji kana mixed sentence and the intermediate expression pair as the reading expression corresponding to the character element string in association with each other, the generated An intermediate representation generation unit that generates one or more reading notations that can be converted from the partial character string as an intermediate stage, as an intermediate representation;
A morpheme enumeration that enumerates the generated intermediate representations and the morpheme candidates corresponding to the concatenation of the intermediate representations using a morpheme dictionary that stores a set of the intermediate representations and character element strings that can be morpheme candidates in association with each other. And
A morpheme string search unit that searches for and outputs a morpheme string that satisfies a predetermined condition among the listed morpheme candidates,
The predetermined condition is a scale defined by a combination of a scale corresponding to each appearance frequency of the enumerated morpheme candidates and a scale corresponding to ease of connection when the enumerated morpheme candidates are continuous. optimization der of is,
At least one of the intermediate expression dictionary and the morpheme dictionary is stored as dictionary data in a code string in which intermediate representations corresponding to the character elements and the character elements that can form a partial character string are alternately arranged, and from the character element strings A morpheme analyzer that is used for conversion to the intermediate representation or conversion from the intermediate representation to the morpheme string .   The morpheme analyzer according to claim 1, wherein all or part of one of the intermediate representation dictionary and the morpheme dictionary is used as all or part of the other. One of the intermediate expression dictionary and the morpheme dictionary is stored as first dictionary data in a code string in which intermediate elements corresponding to the character elements and the character elements are alternately arranged, and from the character element string to the intermediate expression. Used for conversion,
The other is the storage of the character element and the intermediate representation corresponding to the character element with respect to the character element string from the beginning of a part or all of the character element string stored as the first dictionary data to a predetermined number of character elements. wherein the order of the first dictionary data stored in the code string arranged alternately in reverse order, morphological analysis apparatus according to claim 2, wherein the be used in actual conversion from the intermediate representation to the morphemes. A program for outputting a morphological analysis result corresponding to an input character string by causing a computer of a morphological analyzer having a dictionary to execute the program,
Processing to cut out and generate a partial character string from an input character string that is a kanji kana mixed sentence;
Using the intermediate expression dictionary that stores the character element string that can be a partial character string of the kanji kana mixed sentence and the intermediate expression pair as the reading expression corresponding to the character element string in association with each other, the generated Processing to generate one or more reading notations that can be converted from the partial character string as an intermediate stage as an intermediate representation;
A process of enumerating the generated intermediate representation and morpheme candidates corresponding to the concatenation of the intermediate representations using a morpheme dictionary that associates and stores the set of intermediate representations and character element strings that can be morpheme candidates; ,
A process of searching for and outputting a morpheme string satisfying a predetermined condition among the listed morpheme candidates,
The predetermined condition is a scale defined by a combination of a scale corresponding to each appearance frequency of the enumerated morpheme candidates and a scale corresponding to ease of connection when the enumerated morpheme candidates are continuous. optimization der of is,
At least one of the intermediate expression dictionary and the morpheme dictionary is stored as dictionary data in a code string in which intermediate representations corresponding to the character elements and the character elements that can form a partial character string are alternately arranged, and from the character element strings program characterized be used in actual conversion to the morpheme string from the conversion or the intermediate representation to the intermediate representation.

Images