JP2015194801A - Dictionary device, morpheme analysis device, data structure, method and program of morpheme analysis - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a dictionary device, a morpheme analysis device, data structure, a method and a program of morpheme analysis capable of reducing data size of a dictionary, and improving efficiency of processing in morpheme analysis.SOLUTION: A dictionary device 120 for morpheme analysis which stores data on character strings by structure based on a trie stores partial character strings obtained by sectioning data on the character strings and information on the partial character strings by alternately aligning them along the character strings. Thus, efficient data representation becomes possible by using the information regarding the partial character strings, and an amount of data regarding the character strings to be stored for the morpheme analysis can be reduced. In addition, since information regarding partial character strings which are not adopted can be discriminated at an early stage, processing of the morpheme analysis can be efficiently performed. In addition, since the information regarding the partial character strings can be easily referred to, morpheme analysis processing considering constraints of the information regarding the partial character strings can be realized.

Description

  The present invention relates to a dictionary device for morphological analysis, a morphological analysis device, a data structure, a morphological analysis method, and a program for storing character string data in a structure based on a trie.

  Text-to-speech conversion (Text-To-Speech, TTS) is a typical usage form of speech synthesis technology. Text-to-speech conversion is a process of synthesizing a speech waveform corresponding to input text. In the following, this series of processes is roughly divided into two processes: a process of analyzing input text to generate information on how to read the text, and a process of synthesizing a speech waveform from the information on how to read. Also, it is assumed that the input is a Japanese kanji kana mixed sentence.

  Hereinafter, a symbol used for expressing information on how to read is referred to as a speech synthesis symbol. There are various forms of symbols for speech synthesis. Here, we assume that the phonetic information that composes a series of speech and the prosodic information that is mainly expressed as a pose or voice pitch. . An example of such a symbol for speech synthesis is JEITA (Electronic Information Technology Industry Association) standard IT-4006 “Symbol for Japanese Text Speech Synthesis” (see Non-Patent Document 1). Although it is difficult to express even emotional expressions of speech with these symbols alone, it contains at least the information necessary to describe the linguistic features of normal speech.

  On the other hand, the process of synthesizing the speech waveform is performed so that the waveform according to the speech synthesis symbol is synthesized. Therefore, in order to realize accurate reading of the Japanese text, it is only necessary to create an accurate speech synthesis symbol corresponding to the Japanese kanji kana mixed text.

  The process of generating symbols for speech synthesis from arbitrary Japanese text is to divide Japanese Kanji kana mixed sentences into the smallest units that have meaning in linguistic expressions called morphemes, add readings for each morpheme, It can be realized by referring to the morpheme information appropriately, inserting prosodic boundaries such as poses as necessary, and connecting them. At this time, the reading of each morpheme is created and stored in advance as morpheme dictionary information.

  However, the morpheme need not be as defined in linguistic terms, and may be a unit that is appropriately delimited for performing a series of processing. For example, in order to more appropriately process the arrangement of morphemes, a phrase composed of a plurality of morphemes (such as a compound noun phrase) may be regarded as one morpheme for convenience. Therefore, in the following, a morpheme refers to a sequence of characters (character string) that is appropriately set as a minimum unit for processing from the viewpoint of its use, and all sentences are concatenated with this character string. It can be configured with. A process of dividing a sentence into morphemes is generally called a morpheme analysis process, and is used not only for speech synthesis processing but also for extraction of sentence components.

  As a morphological analysis method, a method based on the minimum cost method will be described below. In the morpheme analysis by the minimum cost method, first, an occurrence cost function reflecting the appearance frequency of each morpheme and a concatenated cost function representing ease of connection of consecutive morpheme are defined in advance. Then, an appropriate morpheme sequence is obtained by searching the morpheme registered in the morpheme dictionary for a morpheme sequence that matches the input text and minimizes the cost of the entire sentence. Usually, the occurrence cost function is defined so that the morpheme having the higher appearance frequency and the concatenated cost function have the smaller value as the morpheme sequence that is more easily connected.

  That is, when the morpheme sequence is M = (m1,..., Mn), the generation cost function is Ct (m), and the concatenation cost function is Cc (m (i−k + 1),..., Mi), the total cost Σ Ct + Σ A morpheme analysis process is performed by obtaining a morpheme string M that minimizes Cc, that is, argmin (Σ Ct + Σ Cc). Here, it is assumed that the concatenated cost function is determined by an array of k morphemes.

  When the cost function is defined in this way, the partial series constituting the optimal whole series in terms of cost are optimal in terms of cost even when only the partial series is viewed. Therefore, the subsequence that is not optimal in terms of cost does not need to be considered in the search because it is not a component of the optimal overall sequence. In this way, the optimum sequence search method that proceeds without considering a partial sequence that does not have the possibility of forming an optimum sequence is generally called dynamic programming, and the optimum sequence can be searched efficiently.

  Among the components of the cost function, information on the occurrence cost can be held as the contents of the morpheme dictionary. On the other hand, the connection cost can be obtained by creating a table called a connection table in advance and using the values in the table. However, since it is difficult to create a table of all morpheme sequence combinations, for example, a table that focuses only on the morpheme part-of-speech type is also used. In some cases, these functions are defined as having larger values. In that case, a morpheme sequence having the largest value of the entire sentence is searched.

  In the morpheme string search process in morpheme analysis, it is preferable to examine all possible morpheme sequences. Therefore, in normal morpheme analysis, in order to obtain a morpheme candidate, a partial character string starting from an arbitrary position in a sentence is used as a search key, and the morpheme registered in the morpheme dictionary is equal to the first partial character string of the key. The process of extracting all morphemes is repeated. Such a search is generally referred to as a common prefix search. As a data structure that expresses this relatively efficiently, trie and Patricia tree are known.

  Here, a trie is a multi-tree structure for storing a plurality of character strings, and here, it is assumed that each character is stored as a tree branch in order from the first character of each character string. In the trie, a common prefix between character strings is shared on the tree structure, and therefore, all registered words that are prefixes of the character strings to be searched are arranged on one path of the tree structure. That is, the common prefix search can be realized by tracing the tree in the leaf direction along the search key from the root of the trie.

  The Patricia tree means a node obtained by further combining a node having only one child with the child node in the above-described trie. By this combination, not only one character but also a plurality of consecutive characters may be stored in one branch.

  It is known that the trial can be realized with a small amount of memory by LOUDS (Level-Order Unary Degree Sequence) which is one of succinct data structures. (See Non-Patent Document 2)

"Symbol for Japanese text-to-speech synthesis" JEITA standard IT-4006, March 2010 "Space-efficient static trees and graphs" Jacobson, G, Foundations of Computer Science, 1989., 30th Annual Symposium, pp. 549-554, October 1989

  In Japanese and many other languages, there is a strong relationship between writing and reading information. However, the conventional morpheme analysis system treats the reading information as additional information of each morpheme, and does not use the relationship between the character notation and the reading information in data representation. As a result, for example, the reading information is encoded completely independently for each morpheme, and there is a problem that the size of the morpheme dictionary increases. Although it is possible to compress the size of the reading information by a method such as considering the appearance frequency of symbols describing the reading information, such as Huffman coding, in this case, the process of decoding the compressed reading information Is required, and the amount of processing required increases.

  Also, when using a morpheme dictionary based on a tree structure, the reading information is linked only to the leaf of the tree, so the partial reading information up to the traced part is traced from the root to the leaf node. In the case of using an algorithm that aborts the process in the middle of accessing the tree structure dictionary, reading information cannot be considered as a criterion for aborting.

  The present invention has been made in view of such circumstances, a dictionary device, a morpheme analyzer, a data structure, a morpheme analysis method, and a dictionary device that can reduce the data size of the dictionary and increase the efficiency of processing during morpheme analysis. The purpose is to provide a program.

  (1) In order to achieve the above object, the dictionary device of the present invention is a dictionary device for morphological analysis that stores character string data in a structure based on a trie, and is a partial character string obtained by dividing character string data The information on the partial character string is stored alternately along the character string.

  As a result, as for the information related to the partial character string, the common portion at the head of the information is shared on the tree structure similarly to the partial character string, so that the data size related to the character string stored for morphological analysis can be reduced. In addition, since information regarding partial character strings that are not adopted can be determined at an early stage of the common prefix search, morphological analysis can be performed efficiently. In addition, since information related to the partial character string can be easily referred to, it is possible to realize morpheme analysis processing in consideration of restrictions on information related to the partial character string.

  (2) Moreover, the dictionary apparatus of this invention is characterized by the information regarding the said partial character string including the information regarding the reading of a partial character string. As a result, the data size of the dictionary can be reduced by using information relating to reading of the character string. In addition, it is possible to efficiently perform morphological analysis using information related to reading.

  (3) Moreover, the morpheme analyzer of the present invention is a morpheme analyzer that outputs information on the morpheme strings constituting the morpheme sequence based on the character string, and includes the above dictionary device and the input character string. The present invention relates to a morpheme that cuts out a partial character string, collates the extracted partial character string for each partial character string in the order of the character string, and matches the first partial character string of the cut out partial character string. And a collation unit that outputs information candidates. As a result, information on partial character strings that are not adopted can be determined at an early stage, and morphological analysis can be performed efficiently.

  (4) The morpheme analyzer of the present invention is a morpheme analyzer that outputs information on a morpheme sequence that constitutes a character string based on the character string, and includes the above dictionary device and the input character string. The present invention relates to a morpheme that cuts out a partial character string, collates the extracted partial character string for each partial character string in the order of the character string, and matches the first partial character string of the cut out partial character string. A collation unit that outputs information candidates, and refers to information related to reading of some characters constituting the input character string as constraints, and candidates that satisfy the constraints among candidates output as a result of the collation And a constraint reference unit for outputting. This makes it possible to easily refer to information relating to morpheme reading during the morpheme analysis process, and to realize a morpheme analysis process in consideration of restrictions on reading information.

  (5) Further, the data structure of the present invention is a data structure of a dictionary for morpheme analysis configured based on a trie in a storage unit in a computer, and includes a partial character string obtained by dividing character string data and the part Information on the character string is stored alternately along the character string. As a result, the character string data can be reduced by efficient data representation. Also, the morphological analysis process can be performed efficiently.

  (6) Further, the method of the present invention is a morpheme analysis method for outputting information on a morpheme sequence constituting a character string based on the character string, and cutting out a partial character string from the input character string; The step of collating the extracted partial character string with the character string data having the data structure described above for each partial character string in the order of the character string is performed using a computer. As a result, information on partial character strings that are not adopted can be determined at an early stage, and morphological analysis can be performed efficiently.

  (7) Further, the method of the present invention is a morpheme analysis method for outputting information on a morpheme sequence constituting a character string based on the character string, the step of cutting out a partial character string from the input character string; The character string data, wherein the partial character string extracted for each partial character string in the order of the character string has the data structure described above, and the information on the partial character string includes information on reading of the partial character string And a step of referring to reading information for some characters constituting the input character string as a constraint, and outputting a candidate satisfying the constraint among candidates output as a result of the collation, and Is executed using a computer. This makes it possible to easily refer to information relating to morpheme reading during the morpheme analysis process, and to realize a morpheme analysis process in consideration of restrictions on reading information.

  (8) Further, the program of the present invention is a morpheme analysis program that outputs information on morpheme strings that constitute a character string based on the character string, and a process of cutting out a partial character string from the input character string; , Causing a computer to execute a series of processes including the process of collating the extracted partial character string with the data of the character string having the data structure described above for each partial character string in the order of the character string Yes. As a result, information on partial character strings that are not adopted can be determined at an early stage, and morphological analysis can be performed efficiently.

  (9) Further, the program of the present invention is a morpheme analysis program that outputs information on morpheme strings that constitute a character string based on the character string, and a process of cutting out a partial character string from the input character string; The character string data, wherein the partial character string extracted for each partial character string in the order of the character string has the data structure described above, and the information on the partial character string includes information on reading of the partial character string And a process of referring to reading information for some characters constituting the input character string as a constraint, and outputting a candidate satisfying the constraint among candidates output as a result of the collation The computer is caused to execute a series of processes including. This makes it possible to easily refer to information relating to morpheme reading during the morpheme analysis process, and to realize a morpheme analysis process in consideration of restrictions on reading information.

  According to the present invention, efficient data representation is possible, and the data size of a character string stored for morphological analysis can be reduced. Further, since information on partial character strings that are not adopted can be determined at an early stage, morphological analysis processing can be performed efficiently. In addition, since information related to the partial character string can be easily referred to, it is possible to realize morpheme analysis processing in consideration of restrictions on information related to the partial character string.

It is a block diagram which shows the morphological analyzer of this invention. It is a figure which shows an example of the conventional data structure. It is a figure which shows an example of the data structure of this invention. It is a flowchart which shows operation | movement of the morphological analyzer of this invention. It is a flowchart which shows operation | movement of the morphological analyzer of this invention. It is a flowchart which shows operation | movement of the morphological analyzer of this invention. It is a figure which shows an example of the conventional data structure. It is a figure which shows an example of the data structure of this invention. It is a figure which shows an example of the data structure of this invention.

  Next, embodiments of the present invention will be described with reference to the drawings. In the following description, morphemes with different readings even if the notation character strings are the same are treated as different morphemes.

[First Embodiment]
(Configuration of morphological analyzer)
FIG. 1 is a block diagram showing a morphological analyzer 100. As shown in FIG. 1, the morphological analysis apparatus 100 includes a collation unit 110, a dictionary device 120, a constraint reference unit 130, a determination unit 140, and a connection table storage unit 150. Determine and output suitable information.

  The collation unit 110 cuts out a partial character string from the input character string (kanji-kana mixed text), collates the partial character string cut out for each partial character string in the order of the character string, and cuts out the partial character string. In addition, information candidates relating to morphemes that match the partial character string at the head of the partial character string are output. Thereby, the information regarding the morpheme which is not adopted can be determined at an early stage, and the process of morpheme analysis can be performed efficiently. The data in the dictionary device 120 is collated in the order along the stored character string.

  The dictionary device 120 stores character string data and information related thereto. The dictionary device 120 stores partial character strings obtained by dividing character string data and information related to the partial character strings alternately arranged along the character strings. In general, character string data is divided into a plurality of partial character strings. The data stored in the dictionary device 120 has a tri-based structure. That is, for a plurality of character string data with common data from one end (the beginning or the end) to the middle, the common portion data is represented by a single data, and the non-common portion from the middle to the other end A character string is branched by a plurality of data.

  Thus, by using the relationship between characters and pronunciation, efficient data expression is possible, and morphological analysis with reading information can be realized with a smaller data size. In addition, since information regarding partial character strings that are not adopted can be determined at an early stage, morphological analysis processing can be performed efficiently.

  The information related to the partial character string preferably includes information related to the reading of the partial character string. As a result, the data size of the dictionary can be reduced by using information relating to reading of the character string. In addition, it is possible to efficiently perform morphological analysis using information related to reading.

  The constraint reference unit 130 refers to information related to reading of some characters constituting the input character string as a constraint, and outputs candidates satisfying the constraint among candidates output as a result of collation. By using the data structure of the dictionary device 120, it is possible to easily refer to information related to morpheme reading during the morpheme analysis process, and thus it is possible to realize a morpheme analysis process that takes into account restrictions on reading information.

  The determining unit 140 refers to the concatenation table, calculates the cost of the morpheme candidate sequence, determines the most suitable sequence among the candidate sequences, and outputs it as the morpheme sequence of the morpheme analysis result. The concatenation table storage unit 150 stores a concatenation table representing costs corresponding to partial character strings and consecutive readings.

(Data structure of dictionary device)
The dictionary device 120 embeds pronunciation information in the index. FIG. 2 is a diagram illustrating an example of a conventional data structure. FIG. 3 shows the data structure of the present invention.

  As shown in FIG. 2, in the conventional method in which pronunciation information is not included in the index, information related to reading is linked to a node representing the morpheme end. In this case, the letters “kai” in the index are shared in the trie structure, but the common parts of “kaisha” and “kaigi” related to reading are not directly shared.

  On the other hand, in the example shown in FIG. 3, the character string “Kai / Kai / Company / Sha” is used for the index of the morpheme “Company (Kaisha)”. Here, the character “/” is a delimiter that separates the written character from the reading information. Thus, by using a trie (including a Patricia tree) in the dictionary structure, partial information including pronunciation information is shared by the same node in the tree structure, so that the overall size can be suppressed. For example, if the morpheme dictionary also contains a morpheme called “meeting (kaigi)”, the “/ kai / kai /” part of “kai / kai / sha / sha” and “kai / kai / discussion / gi” is included in the index. Shared.

(Specific data structure based on trial)
When the morphological analysis process is performed using the dictionary device 120 configured by a trie, the tree is traced from the root node of the trie toward the leaf node as a common prefix search, and the morphemes on the nodes are output as candidates.

  However, when the morphological analysis dictionary structure including the reading information in the index is used in the morphological analysis apparatus 100, the reading information in the section surrounded by the leaf direction “/” in the index of the dictionary is ignored in the search, and its child What is necessary is just to change to a node. However, when there are a plurality of types of pronunciation information for a certain character, there are a plurality of possibilities at the node corresponding to the pronunciation information, and there are a plurality of child nodes that can be search results. In that case, it is necessary to consider all those child nodes as candidates.

  In the conventional method, the morpheme candidates listed by the common prefix search exist only on one path of the trie. In the present invention, however, morpheme candidates exist on a plurality of different paths in this way. This may need to be taken into account in the search. Specifically, it is necessary to create a list for holding a plurality of lists of nodes from the root to the leaf.

(Operation of morphological analyzer)
4 to 6 are flowcharts illustrating an example of the operation of the matching unit 110 in the morphological analyzer 100, respectively. FIG. 4 shows the processing of the entire verification unit. As shown in FIG. 4, first, the input character string is divided into partial character strings so as to enumerate morpheme candidates that may be included in the input character string as a result of the common prefix search. These are listed (step S101). The enumeration of partial character strings here may be repeated, for example, from the first character to the last character, from the second character to the last character. Next, one partial character string is set as a search key (step S102). Then, subroutine G (root node, 0) is called to collate the data stored in the dictionary with the search key (step S103).

  FIG. 5 shows processing of a subroutine (G (N, k)) for collating a character string with a search key. G (N, k) means collation of the character string between the node N of the dictionary data and the kth character of the search key. As shown in FIG. 5, all N child nodes are listed and stored in the node list L (step S201), and an initial value 0 is set to i (step S202).

  Next, it is determined whether the character linked to the branch to L [i] matches the kth character of the search key (step S203). If they match, G (L [i], k + 1) is called (step S204), and the process proceeds to step S207.

  If they do not match, it is determined whether or not the character linked to the branch to L [i] is a delimiter (step S205). If the character is a delimiter, subroutine P (L [i], k) is called (step S206), and the process proceeds to step S207. On the other hand, if the character is not a delimiter, the process proceeds to step S207.

  Then, i is increased by 1 (step S207), and it is determined whether i is less than the size of L [i] (step S208). If it is less, the process proceeds to step S203. If i is greater than or equal to L [i], the subroutine ends and the process returns to the original process.

  FIG. 6 shows processing of a reading information collation subroutine (P (N, k)). Subroutine P (N, k) means collation of reading between node N of the dictionary data and the kth character of the search key. As shown in FIG. 6, all N child nodes are listed and stored in the node list L (step S301), and an initial value 0 is set to i (step S302).

  Next, it is determined whether L [i] is a morpheme end (step S303). If it is a morpheme powder, a character string from the root to L [i] is output as a result (step S304), and the process proceeds to step S308.

  If it is not a morpheme end, it is determined whether or not the character linked to the branch to L [i] is a delimiter (step S305). If the character is a delimiter, subroutine G (L [i], k) is called (step S306), and the process proceeds to step S308. On the other hand, if the character is not a delimiter, subroutine P (L [i], k) is called (step S307), and the process proceeds to step S308.

  Then, i is incremented by 1 (step S308), and it is determined whether i is less than the size of L [i] (step S309). If it is less, the process proceeds to step S303. If i is greater than or equal to L [i], the subroutine ends and the process returns to the original process. The above processing can be performed by causing a computer to execute processing.

(Example of morphological analysis processing)
An example of processing will be described below. FIG. 7 is a diagram illustrating an example of a conventional data structure. FIG. 8 shows an example of the data structure of the present invention.

  When the morphological analysis of the sentence “go to Akihabara” is performed, a common prefix search is first performed using “go to Akihabara” as a key. At this time, if three words “Aki”, “Akiba”, and “Akihabara” are registered in the dictionary, these are listed as morpheme candidates.

  However, when an index that does not include conventional reading information as shown in FIG. 7 is used, these three words are present on the same path when viewed from the root node. In other words, only a single path on the trie needs to be considered in the common prefix search.

  On the other hand, in the method of the present invention as shown in FIG. 8, since “Akihabara” and “Akihabara” have different readings of the “leaf” portion, the word “Akiha” does not exist on the path to “Akihabara”. . Up to “Autumn / Aki / Leaf /” are on the same path, and thereafter, information is stored in different branches.

  In this process, the parent node needs to be able to enumerate all its child nodes at high speed. In the conventional morphological analysis, it is only necessary to determine at high speed whether or not there is a child node connected to a branch to which a specific character is linked, and this part is different from the conventional one.

  However, it is relatively easy to realize such a data structure. For example, a parent node has only a link to one child node among a plurality of child nodes, and a list for all sibling nodes starting with the child node. You can build a structure. Alternatively, LOUDS, which is a data structure used for implementing memory-saving tries, enumerates child nodes when tracing trees in the leaf direction, even if the conventional method does not include reading information in the index. A practical system using LOUDS for the data structure satisfies the above-mentioned conditions required by the present invention.

(Processing when constraints are entered)
In the conventional dictionary structure, morpheme information is linked to a trie node. For this reason, when performing a constrained search for reading information, it is necessary to perform processing such as enumerating all morpheme candidates by common prefix search, then examining each morpheme information, and discarding morpheme candidates that do not satisfy the reading constraint. Become.

  On the other hand, when the morpheme dictionary structure of the present invention is constructed with reading information, instead of simply skipping the pronunciation information, the trie search is performed to determine whether the pronunciation information satisfies the restrictions on the given reading information. You can check in stages. Then, morphemes that do not satisfy the restriction condition of the reading information can be excluded from the morpheme candidate enumeration targets at an earlier time point. Thereby, the number of morpheme candidates to be enumerated decreases, and the processing amount and memory usage amount can be reduced. In this case, when the processing based on the flowchart shown in FIG. 6 is performed, it is checked whether or not the reading information constraint condition is satisfied immediately before S303, and if satisfied, the process proceeds to S303, and if not, the process proceeds to S308. do it.

(Other processing examples)
FIG. 9 shows an example of the data structure of the present invention. For example, in the case where three words “upper (we)”, “upper (aga ru)” and “upper (novo le)” are registered in the dictionary, the text to be analyzed is “up”, When “NOVO” is specified as the reading of “UP”, in the conventional dictionary structure, as a result of the common prefix search for “UP”, these three words are obtained as candidates. Therefore, a process of discarding morphemes other than “Noboru” is required.

  On the other hand, in the method of the present invention, in the search, “up / novo / ru / *” (where “*” indicates all character strings composed of characters other than “/”) is used as a key. Prefix search may be performed, and only “noboru” with the index “up / novo / ru / le” is obtained as a result of the common prefix search.

  In this processing, in order to take into account fluctuations in reading, it is possible to use another selection criterion instead of the complete matching of reading information as described above. For example, it is possible to set a constraint condition that allows ambiguity such as enumerating morphemes whose edit distance is less than or equal to a certain value (more than a certain number of matching characters) as the search result of the morpheme dictionary. It is.

  In the above description, the delimiter symbol “/” is inserted between the partial character string constituting the notation character string and the corresponding phonetic symbol, but it is a rule that phonetic information is given for each character of the notation character string. In the case where the delimiter “/” is not written between the character of the written character string and the pronunciation information, it is possible.

  Alternatively, instead of defining a delimiter, each node stores 1-bit information indicating whether the character stored in the immediately preceding branch is a character representing a written character string or a character representing reading information, You may use this. Thereby, the size of the dictionary can be further suppressed.

  In the description so far, the reading information is exemplified as a character string including only katakana characters, but the present invention is not limited to this. For example, a symbol including prosodic information such as JEITA IT-4006 may be used. For cases where reading information is regularly determined from written characters, a method of reducing the length of reading information by defining a code having the meaning of “determined regularly” as one of the reading information may be used. it can. As an example of reading that is regularly determined, there is a kanji that has only one type of reading.

  In the method of the present invention, information other than reading information may be stored. For example, in a morphological analysis system in which the morpheme occurrence cost is defined as the sum of the occurrence costs of each character in the character notation, a symbol representing the character occurrence cost is defined, and a set of characters and this symbol is arranged to indicate a trie index. By configuring the information, the information about the morpheme generation cost can be directly embedded in the trie structure itself.

  In this structure, since the cost of occurrence is obtained in the process of following a trie, enumeration of candidates taking into account the cost of occurrence of morpheme is performed, such as discarding candidates with high morpheme generation cost in the process of enumerating candidates for morpheme. be able to. The information to be stored may be a combination of various information such as a combination of reading information and a character generation cost.

  Further, a dictionary configuration in which all written character strings are reversed in the front-rear order is also possible. For example, a morpheme dictionary in which character strings are reversed in front and back is constructed, and in the morphological analysis of sentences, there is a method of enumerating morpheme candidates from the beginning of sentences in reverse order, that is, from the end of the original sentence toward the beginning. Conceivable. In this case, the optimal morpheme sequence finally obtained is determined from the beginning of the sentence toward the end of the sentence.However, in many devices, the morpheme analysis result is also output from the beginning of the sentence toward the end of the sentence. Unlike the case of determining from the end of the sentence to the beginning of the sentence, there is an advantage that it is not necessary to temporarily store the morphological analysis result.

100 morphological analyzer 110 collation unit 120 dictionary device 130 constraint reference unit 140 determination unit 150 connection table storage unit

Claims (9)

A dictionary device for morphological analysis that stores character string data in a tri-based structure,
A dictionary device characterized by storing partial character strings obtained by dividing character string data and information on the partial character strings alternately arranged along the character strings.   2. The dictionary apparatus according to claim 1, wherein the information on the partial character string includes information on reading of the partial character string. A morpheme analyzer that outputs information on a morpheme sequence that constitutes a character string based on a character string,
The dictionary device according to claim 1 or 2,
A partial character string is cut out from the input character string, and the extracted partial character string is collated with the dictionary device for each partial character string in the order of the character string, and the first partial character of the cut out partial character string A morpheme analyzer comprising: a collation unit that outputs information candidates relating to morphemes that match the columns. A morpheme analyzer that outputs information on a morpheme sequence that constitutes a character string based on a character string,
A dictionary device according to claim 2;
A partial character string is cut out from the input character string, and the extracted partial character string is collated with the dictionary device for each partial character string in the order of the character string, and the first partial character of the cut out partial character string A collation unit that outputs candidate information about morphemes that match the columns;
A constraint reference unit that refers to information about reading of some characters constituting the input character string as a constraint, and outputs a candidate that satisfies the constraint among candidates output as a result of the collation. A morphological analyzer characterized by the above. A data structure of a dictionary for morphological analysis configured based on a trie in a storage unit in a computer,
A data structure characterized in that partial character strings obtained by dividing character string data and information on the partial character strings are stored alternately arranged along the character string. A morpheme analysis method for outputting information on a morpheme sequence constituting a character string based on a character string,
Cutting out a substring from the input string;
The step of collating the extracted partial character string with the character string data having the data structure according to claim 5 is performed using a computer. Method. A morpheme analysis method for outputting information on a morpheme sequence constituting a character string based on a character string,
Cutting out a substring from the input string;
6. The character string having the data structure according to claim 5, wherein the partial character string cut out for each partial character string in the order of the character string, wherein the information on the partial character string includes information on reading of the partial character string Checking against the data of
Using a computer to refer to reading information for some characters constituting the input character string as a constraint, and to output candidates that satisfy the constraint among candidates output as a result of the matching A method characterized by performing. A morpheme analysis program that outputs information on morpheme sequences that make up a character string,
A process of cutting a substring from the input string,
6. A process for causing a computer to execute a series of processes including a process of collating the extracted partial character string for each partial character string in the order of the character string with data of a character string having a data structure according to claim 5. A featured program. A morpheme analysis program that outputs information on morpheme sequences that make up a character string,
A process of cutting a substring from the input string,
6. The character string having the data structure according to claim 5, wherein the partial character string cut out for each partial character string in the order of the character string, wherein the information on the partial character string includes information on reading of the partial character string Processing to match the data of
A series of processes including a process of referring to reading information for some characters constituting the input character string as a constraint, and outputting candidates satisfying the constraint among candidates output as a result of the collation A program that causes a computer to execute.

Images