JP2014106707A - Word division device, data structure of dictionary for word division, word division method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem of inability to divide a sentence into two or more words at a high speed.SOLUTION: A sentence can be divided into two or more words at a high speed with the use of a word division device including: a dictionary for word division which can store one or more words and one or more division data which is a set of a word and two or more divided words which is a result of division of the word; a first division unit which acquires a word coinciding with a character string which is longest from a pointer of a sentence, which is the beginning of an accepted sentence, from the dictionary for word division, executes divided word acquisition processing for acquiring two or more divided words instead of a coinciding word if there are two or more divided words corresponding to the acquired word; executes the divided word acquisition processing up to a word containing the last letter of the sentence after moving the pointer of the sentence to a next letter of the coinciding word, and acquires a first division result which is a group of two or more words obtained from division of the sentence; and an output unit for outputting the first division result.

Description

  The present invention relates to a word dividing device that divides a sentence into two or more words.

  Conventionally, there is a technology that increases the recognition accuracy of natural language processing by calculating the n-gram probabilities of words with high accuracy by effectively using the first corpus that has already been divided into words and the second corpus that has not been divided into words. (See Patent Document 1).

  Conventionally, there has been a morphological analysis system that can be easily customized by a user (see Patent Document 2). In this system, the character string input to the text input unit is divided into a plurality of morphemes by the general-purpose morpheme analysis unit with reference to the morpheme analysis dictionary. Next, the pattern matching engine performs conversion processing on a plurality of morphemes divided by the general-purpose morpheme analysis unit with reference to a pattern instructing division or combination of morphemes described by the user in the pattern file. . The plurality of morphemes converted by the pattern matching engine are output from the output generation unit as analysis results.

  Furthermore, there has conventionally been a technique for dividing a word string such as a sentence or a compound word into a sequence of correct words (see Patent Document 3).

JP 2006-3295A (first page, FIG. 1 etc.) Japanese Patent Laid-Open No. 10-40252 (first page, FIG. 1 etc.) JP-A-7-262191 (first page, FIG. 1 etc.)

  However, in the prior art, a sentence cannot be divided into two or more words at high speed.

  The word segmentation device according to the first aspect of the present invention is a word segmentation dictionary that can store one or more words and one or more segmentation information that is a set of two or more segmented words that is a result of segmenting the words. Using a word receiving dictionary that accepts a sentence having one or more characters and a word segmentation dictionary, obtains the maximum length word that matches the character string constituting the sentence accepted by the accepting unit from the word segmentation dictionary; A first division unit that obtains two or more divided words corresponding to the obtained word and obtains a first division result that is a set of two or more words obtained by dividing the sentence; A word segmenting device comprising an output unit for outputting.

  With this configuration, since the sentence can be divided into two or more words by a very simple process, the sentence can be divided into two or more words at high speed.

  In addition, the word dividing device according to the second aspect of the invention obtains two or more words by dividing the sentence received by the receiving unit with an algorithm different from that of the first dividing unit with respect to the first invention. Using the second division unit, a second division result acquisition unit that acquires a second division result that is a set of two or more divided words and whether the first division result and the second division result are different are determined. If the determination unit determines that the first division result and the second division result are different from each other, the character string in the sentence corresponding to the portion where the first division result and the second division result are different is acquired. A division information acquisition unit that acquires two or more words included in the second division result corresponding to different locations, and that constitutes division information having the acquired character string and the acquired two or more words; A dictionary registration unit for storing the division information in the word division dictionary. A dividing device.

  With this configuration, the word division dictionary can be enriched.

  The word segmentation device according to the third aspect of the present invention is a second segmentation method for segmenting a sentence received by the reception unit by using an algorithm different from that of the first division unit and acquiring two or more words. A word segmentation device further comprising a segmentation unit.

  With this configuration, the word division dictionary can be enriched.

  Further, in the word dividing device according to the fourth aspect of the present invention, in contrast to the second or third aspect, the second dividing unit divides the sentence by an algorithm of morphological analysis using the Viterbi algorithm to obtain two or more words. Is a word segmentation device for acquiring.

  With this configuration, it is possible to register division information with high accuracy in the word division dictionary.

  According to the word dividing device of the present invention, a sentence can be divided into two or more words at high speed.

Block diagram of word segmentation apparatus 1 according to Embodiment 1 A flowchart for explaining the operation of the word segmentation apparatus 1 The figure which shows the dictionary 11 for the word division | segmentation Block diagram of word segmentation apparatus 2 according to Embodiment 2 A flowchart for explaining the operation of the word segmentation device 2 Block diagram of the word dividing device 3 The figure which shows the experimental result of the word division | segmentation apparatus 1 The figure which shows the experimental result of the word division | segmentation apparatus 1 Overview of the computer system in the above embodiment Block diagram of a computer system according to the embodiment Another block diagram of word segmentation apparatus 2 in the second embodiment

  Hereinafter, embodiments of a word division device and the like will be described with reference to the drawings. In addition, since the component which attached | subjected the same code | symbol in embodiment performs the same operation | movement, description may be abbreviate | omitted again.

(Embodiment 1)
In the present embodiment, a word dividing device 1 that divides a sentence into two or more words will be described.

  FIG. 1 is a block diagram of a word segmentation apparatus 1 in the present embodiment. The word division device 1 includes a word division dictionary 11, a reception unit 12, a first division unit 13, and an output unit 14.

  The word division dictionary 11 can store one or more words and one or more pieces of division information. The division information is a set of a word and two or more divided words. The divided word is a result of dividing the word. The division information is, for example, “free format: free / format” or “is not: is / is / is”. “Free format” of the division information “free format: free / format” is a word, and “free” and “format” of “free / format” are divided words. In addition, “has not”: “has not”: “has not” is “word”, and “ha”, “no”, “no”, and “ha” are respectively segmented words. Note that the word may be considered to include any term having meaning such as a morpheme or a collocation. A divided word can also be said to be a word.

  In the word division dictionary 11, it is preferable that one or more words and one or more pieces of division information are held in the same file or the same database. However, the one or more words and the one or more pieces of division information may be held in separate files or separate databases. That is, the specific data structure of the word division dictionary 11 is not limited. The word division dictionary 11 only needs to hold one or more words and one or more pieces of division information.

  The word segmentation dictionary 11 is preferably a non-volatile recording medium, but can also be realized by a volatile recording medium. The process of storing words and division information in the word division dictionary 11 does not matter. For example, words and division information may be stored in the word division dictionary 11 via a recording medium, and words and division information transmitted via a communication line or the like are stored in the word division dictionary 11. Alternatively, a word or division information input via an input device may be stored in the word division dictionary 11.

  The accepting unit 12 accepts a sentence having one or more characters. The sentence may be an incomplete sentence. That is, the sentence may be a multiple word or the like. Moreover, the language of a sentence is not ask | required. The sentence is usually a sentence in a language that is not divided, such as Japanese, Chinese, Korean, and Mongolian. However, the sentence may be a language such as English that is not shared. The sentence may be, for example, a character string indicating a URL or a character string indicating a file name. In addition, reception means reception of information input from an input device such as a keyboard, mouse, touch panel, reception of information transmitted via a wired or wireless communication line, an optical disk, a magnetic disk, a semiconductor memory, etc. This is a concept including reception of information read from the recording medium.

  The sentence input means may be anything such as a keyboard, mouse or menu screen. The receiving unit 12 can be realized by a device driver for input means such as a keyboard, control software for a menu screen, or the like.

The first dividing unit 13 divides the sentence received by the receiving unit 12 and acquires a first division result that is a set of two or more words.
More specifically, the first dividing unit 13 acquires, from the word dividing dictionary, the maximum length word that matches the character string constituting the sentence received by the receiving unit 12 using the word dividing dictionary. Two or more divided words corresponding to the acquired word are acquired, and a first division result which is a set of two or more words obtained by dividing the sentence is acquired. This process will be described in further detail as follows. The 1st division part 13 acquires one or more character strings which constitute the sentence which acceptance part 12 received using the dictionary for word division. Then, the first division unit 13 obtains the maximum length word that matches the one or more character strings from the word division dictionary. And the 1st division part 13 acquires two or more division words corresponding to a word for every one or more words acquired from the word division dictionary, and obtains two or more words obtained by dividing a sentence. The first division result that is a set is acquired.

  More specifically, the first dividing unit 13 performs processing as follows, for example. First, the first dividing unit 13 performs a first process of acquiring, from the word dividing dictionary 11, a word that matches the maximum length character string from the sentence pointer at the head of the sentence received by the receiving unit 12. When the first dividing unit 13 has two or more divided words corresponding to the acquired word, the first dividing unit 13 performs a second process of acquiring two or more divided words instead of the matching words. The first process and the second process are referred to as a divided word acquisition process. Then, the first dividing unit 13 moves the sentence pointer to the character next to the matching word. And the 1st division part 13 performs said division | segmentation word acquisition process to the word containing the last character of a sentence. As a result, the first division unit 13 can acquire a first division result that is a set of two or more words obtained by dividing a sentence. When the word acquired in the first process is not a word included in the division information, the first dividing unit 13 holds the word acquired in the first process as it is. The first division result is a set of two or more words, but has a data structure in which the two or more breaks can be determined.

Moreover, the method of the 1st process which the 1st division part 13 performs is not ask | required. For the first processing performed by the first dividing unit 13, a known technique such as a so-called longest match method (also referred to as a longest match or maximum match) can be used. The longest match method is described in “Iwanami Shoten, Iwanami Lecture, Software Science 15 Natural Language Processing, pages 126-127, Makoto Nagao”.

Further, the first dividing unit 13 determines the character string of the maximum length from the sentence pointer, and the character string from the character indicated by the sentence pointer to the last character of the sentence (assuming that it is the Nth character from the sentence pointer). It is determined whether or not a word that matches (this is referred to as a character string A) exists in the word segmentation dictionary 11, and if it exists, the character string A is acquired, and if it does not exist, a pointer to the sentence It is determined whether or not a word that matches the character string (referred to as character string B) from the character indicated by the sentence pointer to the (N-1) th character exists in the word segmentation dictionary 11; If it exists, the character string B is acquired. If it does not exist, the character string is reduced by one character at a time, and the word with the maximum character string in the character string starting with the character indicated by the sentence pointer is replaced with the word. Search from the partitioning dictionary 11. That is, the first dividing unit 13 searches the word dividing dictionary 11 in order while reducing the character by character from the longest character string of the unprocessed character string in the sentence, and obtains the longest character string from the pointer p. You may do it.
The first dividing unit 13 has a known technique “trie” as a data structure for detecting the maximum length character string from the sentence. Since the trial is described in the following (1) to (3), a detailed description is omitted.
(1) Takuyuki Tokunaga, “Technology that supports Japanese input”, pages 89-99 (2) Internet web page, URL
"Http://www.slideshare.net/higashiyama/ss-8738479"
(3) Internet Web page, URL
"Http://nanika.osonae.com/DArray/dary.html"
"

  The first dividing unit 13 can be usually realized by an MPU, a memory, or the like. The processing procedure of the first dividing unit 13 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The output unit 14 outputs the first division result acquired by the first division unit 13. Here, output refers to display on a display, projection using a projector, printing with a printer, sound output, transmission to an external device, storage in a recording medium, and output to other processing devices or other programs. It is a concept that includes delivery of processing results. When handing over the processing result to another program, the word segmentation device 1 and the other program realize, for example, a speech recognition device, a machine translation device, or the like. That is, the first division result obtained by dividing the sentence can be used for, for example, speech recognition processing, machine translation processing, and the like.

  The output unit 14 may be considered as including or not including an output device such as a display or a speaker. The output unit 14 can be realized by output device driver software, or output device driver software and an output device.

  Next, operation | movement of the word division | segmentation apparatus 1 is demonstrated using the flowchart of FIG.

  (Step S201) The reception unit 12 determines whether a sentence has been received. If a sentence is accepted, the process goes to step S202, and if no sentence is accepted, the process returns to step S201.

  (Step S202) The first dividing unit 13 sets the sentence pointer p to 1. The sentence pointer p indicates the position of the beginning of word acquisition in the sentence.

  (Step S203) The first division unit 13 searches for a word that is present in the word division dictionary 11 and matches the maximum length character string from characters corresponding to p in the sentence. Then, the first division unit 13 acquires a word that is a maximum length character string from the word division dictionary 11.

  (Step S204) The first division unit 13 determines whether or not the word acquired in step S203 is a word included in the division information. If it is a word included in the division information, go to step S205, and if it is not a word included in the division information, go to step S206.

  (Step S205) The first dividing unit 13 acquires two or more divided words corresponding to the word acquired in step S203 from the word dividing dictionary 11. Then, the first dividing unit 13 adds two or more divided words to the buffer. Note that the initial value of the buffer is NULL. The first dividing unit 13 puts a delimiter character in each of the two or more divided words and adds the two or more divided words to the buffer. The delimiter may be anything such as “/”, “(space)”, “,”, etc. Go to step S207.

  (Step S206) The first dividing unit 13 adds the word acquired in step S203 to the buffer. The first dividing unit 13 places a delimiter between the word acquired in step S203 and the previous or / and subsequent word.

  (Step S207) The first dividing unit 13 advances the pointer p by the maximum character string length.

  (Step S208) The first dividing unit 13 determines whether or not all the dividing processes have been completed. If all the division processes are completed, the process goes to step S209, and if not completed, the process returns to step S203. When the pointer p is the position next to the last character of the sentence, it can be said that all the division processes have been completed.

  (Step S209) The output unit 14 outputs two or more words in the buffer. The process returns to step S201.

In the flowchart of FIG. 2, the process is terminated by powering off or a process termination interrupt.
Further, in the flowchart of FIG. 2, the processing is started from the beginning of the accepted sentence, and the processing is sequentially performed until the end of the sentence. However, for example, the process may be started from the end of the accepted sentence, and the process may be performed in the previous direction after the sentence. In other words, in step S202, the first dividing unit 13 sets the sentence pointer p to the end of the sentence, and in step S207, the pointer p may return to the front of the sentence by the maximum character string length. good. In such a case, in step S203, the first dividing unit 13 is a word existing in the word dividing dictionary 11, and advances the pointer forward from the character corresponding to p in the sentence to obtain the maximum length character string. Search for matching words. Then, the first division unit 13 acquires a word that is a maximum length character string from the word division dictionary 11.

  Hereinafter, a specific operation of the word segmentation apparatus 1 in the present embodiment will be described.

  FIG. 3 shows the word division dictionary 11. The records constituting the word division dictionary 11 have “ID”, “word”, and “division word”. The record of the word division dictionary 11 may have other information such as part of speech and appearance probability. The records in the word division dictionary 11 are classified into words or division information.

  A record classified as a word has an attribute “divided word” value of NULL (“−” in FIG. 3). Moreover, the record classified into a word is a record of "ID = 5,6,8,9,10,11,12,13" of FIG. 3, for example. Further, the record classified as the division information has a division word having the attribute “division word” value of 2 or more. Here, the divided words in the attribute “divided word” are separated by a delimiter “/”. Further, the record classified as the division information is, for example, a record of “ID = 1, 2, 3, 4, 7” in FIG. Note that the record of the word division dictionary 11 may have “a flag indicating whether it is a word or division information” as an attribute value.

(Specific example 1)
In this situation, it is assumed that the reception unit 12 has received the sentence “Masao is a solid person”. Next, the first dividing unit 13 sets the sentence pointer p to 1. That is, the pointer p is set at the “positive” position of the sentence.

  Next, the first division unit 13 searches for the word “Matsuo” that is present in the word division dictionary 11 and matches the maximum character string from the character “Correct” corresponding to p in the sentence. And get.

  Next, the first division unit 13 determines whether or not the acquired word “Matsuo” is a word included in the division information. That is, the first division unit 13 determines that the division information corresponding to the word “Matsuo” is “− (NULL)”.

  Then, the first dividing unit 13 adds the acquired word “Matsuo” to the buffer.

  Next, the first dividing unit 13 calculates the character string length “2” of the word “Matsuo”. Next, the first dividing unit 13 advances the pointer p by “2” corresponding to the maximum length of the character string, and sets the pointer p to the position “ha” of the sentence.

  Next, the first division unit 13 determines that the division process has not been completed yet.

  Next, the first division unit 13 searches for a word “ha” that is present in the word division dictionary 11 and matches the maximum length character string from the character “ha” corresponding to p in the sentence. And get.

  Next, the first division unit 13 determines whether or not the acquired word “ha” is a word included in the division information. That is, the first division unit 13 determines that the division information corresponding to the word “ha” is “− (NULL)”.

  Then, the first dividing unit 13 adds the acquired word “ha” to the buffer. The first dividing unit 13 adds a delimiter “/” in front of the word “ha” and appends it to the buffer. Then, “Masao / Ha” is stored in the current buffer.

  Next, the first dividing unit 13 calculates the character string length “1” of the word “ha”. Next, the first dividing unit 13 advances the pointer p by “1” corresponding to the maximum length of the character string, and sets the pointer p to the position of “shi” of the sentence.

  Next, the first division unit 13 determines that the division process has not been completed yet.

  Next, the first dividing unit 13 is a word that exists in the word dividing dictionary 11, and from the character “shi” corresponding to p in the sentence, the word “solid” that matches the maximum length character string is obtained. Search and get.

  Next, the first division unit 13 determines whether or not the acquired word “firm person” is a word included in the division information. That is, since the divided word corresponding to the word “firm person” is not NULL, the first dividing unit 13 determines that the word “firm person” is a word included in the division information.

  Then, the first division unit 13 acquires the division information “firm / person” corresponding to the word “firm person” from the word division dictionary 11.

  Then, the first dividing unit 13 adds the delimiter “/” and the acquired word “solid / person” to the buffer. In the current buffer, “Masao / Ha / Ken / People” is stored.

  Next, the first dividing unit 13 calculates the character string length “5” of the word “firm person”. Next, the first dividing unit 13 advances the pointer p by “5” corresponding to the maximum length of the character string, and sets the pointer p to the position of “DA” in the sentence.

  Next, the first division unit 13 determines that the division process has not been completed yet.

  Next, the first division unit 13 searches for a word “DA” that is a word existing in the word division dictionary 11 and matches the maximum length character string from the character “DA” corresponding to p in the sentence. And get.

  Next, the first division unit 13 determines whether or not the acquired word “DA” is a word included in the division information. That is, the first division unit 13 determines that the division information corresponding to the word “DA” is “− (NULL)”.

  Then, the first division unit 13 adds the delimiter “/” and the acquired word “da” to the buffer. In the current buffer, “Masao / Ha / So / Person” is stored.

  Next, the first dividing unit 13 calculates the character string length “1” of the word “DA”. Next, the first dividing unit 13 advances the pointer p by “1” corresponding to the maximum length of the character string, and sets the pointer p to a position next to “DA” in the sentence.

  Next, the first division unit 13 determines that the division process has been completed.

  Then, the output unit 14 outputs two or more divided word strings “Masao / Ha / So / I / D” in the buffer.

(Specific example 2)
It is assumed that the reception unit 12 has received the sentence “Although it is said that it is still a child”. Next, the first dividing unit 13 sets the sentence pointer p to 1. That is, the pointer p is set at the position “s” in the sentence.

  Next, the first division unit 13 is a word that exists in the word division dictionary 11, and the word “So” corresponding to p in the sentence is matched with the maximum length character string “Yes. Search for and get "".

  Next, the first dividing unit 13 determines whether or not the acquired word “even if so” is a word included in the division information. That is, since the divided word corresponding to the word “Yes” is not NULL, the first dividing unit 13 determines that the word “No, yes” is a word included in the division information.

  Then, the first division unit 13 acquires the division information “so / ha / inte / mo” corresponding to the word “so yes” from the word division dictionary 11.

  Then, the first dividing unit 13 adds the acquired word “so / ha / inte / mo” to the buffer. In the current buffer, “yes / ha / te / mo” is stored.

  Next, the first dividing unit 13 calculates the character string length “7” of the word “even if so”. Next, the first dividing unit 13 advances the pointer p by “7” corresponding to the maximum length of the character string, and sets the pointer p to the position “ma” of the sentence.

  Next, the first division unit 13 determines that the division process has not been completed yet.

  Next, the first division unit 13 searches for the word “still” that matches the maximum length character string from the character “ma” corresponding to p in the sentence that is present in the word division dictionary 11. And get.

  Next, the first division unit 13 determines whether or not the acquired word “still” is a word included in the division information. That is, the first division unit 13 determines that the division information corresponding to the word “ha” is “− (NULL)”.

  Then, the first dividing unit 13 appends the delimiter “/” and the acquired word “still” to the buffer. In the current buffer, “yes / ha / te / mo / no” is stored.

  Next, the first dividing unit 13 calculates the character string length “2” of the word “still”. Next, the first division unit 13 advances the pointer p by “2” corresponding to the maximum length of the character string, and sets the pointer p to the position of the “child” of the sentence.

  Next, the first division unit 13 determines that the division process has not been completed yet.

  Next, the first division unit 13 searches for a word “child” that is in the word division dictionary 11 and matches the maximum character string from the character “child” corresponding to p in the sentence. And get.

  Next, the first division unit 13 determines whether or not the acquired word “child” is a word included in the division information. That is, the first division unit 13 determines that the division information corresponding to the word “ha” is “− (NULL)”.

  Then, the first dividing unit 13 adds the delimiter “/” and the acquired word “child” to the buffer. In the current buffer, “Yes / I / I / Mo / Still / Child” is stored.

  Next, the first dividing unit 13 calculates the character string length “2” of the word “still”. Next, the first dividing unit 13 advances the pointer p by “2” corresponding to the maximum length of the character string, and sets the pointer p to the position next to “don” in the sentence.

  Next, the first division unit 13 determines that the division process has been completed.

  Then, the output unit 14 outputs the two or more divided word strings “so / ha / de / mo / yana / child” in the buffer.

  As described above, according to the present embodiment, a sentence can be divided into two or more words by a very simple process. Therefore, the sentence can be divided into words very quickly.

In the present embodiment, the algorithm by which the first dividing unit 13 acquires the word that is the maximum length character string from the word dividing dictionary 11 does not matter.
Further, in the present embodiment, instead of the second division unit 21, a second division result storage unit 26 that is a set of one or more second division results may be used. In such a case, the determination unit 23 determines whether the first division result and the second division result stored in the second division result storage unit 26 are different. In such a case, the word division device 2 includes the word division dictionary 11, the reception unit 12, the first division unit 13, the output unit 14, the determination unit 23, the division information acquisition unit 24, the dictionary registration unit 25, and the second division. A result storage unit 26 is provided. FIG. 11 shows a block diagram of the word dividing device 2 in such a case.
In FIG. 11, the division information acquisition unit 24 acquires the second division result from the second division result storage unit 26 via the determination unit 23.
It should be noted that the set of second division results in the second division result storage unit 26 is a large amount of data greater than or equal to a certain amount, and is preferably word-divisionally created data created manually. The sentence to be divided is the same for the first division result and the second division result stored in the second division result storage unit 26.

  Furthermore, the processing in the present embodiment may be realized by software. Then, this software may be distributed by software download or the like. Further, this software may be recorded and distributed on a recording medium such as a CD-ROM. This also applies to other embodiments in this specification. Note that the software that implements the word segmentation apparatus in the present embodiment is the following program. That is, this program stores a word division dictionary having one or more words and one or more pieces of division information that is a set of two or more divided words that are the result of dividing the words. A word receiving dictionary that accepts a sentence having one or more characters, and a word that matches a maximum-length character string from a sentence pointer at the head of the sentence accepted by the accepting part. If there are two or more divided words corresponding to the acquired word, a divided word acquisition process is performed in which the two or more divided words are obtained instead of the matching words, and the sentence pointer is After moving to the next character of the matching word, the divided word acquisition process is performed up to the word including the last character of the sentence, and the first division result which is a set of two or more words obtained by dividing the sentence is acquired. The first minute to And parts, is a program, to function as an output unit for outputting the first result of division.

(Embodiment 2)
In the present embodiment, a word division apparatus 1 that can enhance a word division dictionary while performing sentence division processing will be described.

  FIG. 4 is a block diagram of word segmentation apparatus 2 in the present embodiment. The word division device 2 includes a word division dictionary 11, a reception unit 12, a first division unit 13, an output unit 14, a second division unit 21, a second division result acquisition unit 22, a determination unit 23, a division information acquisition unit 24, A dictionary registration unit 25 is provided.

  The second dividing unit 21 divides the sentence received by the receiving unit 12 using an algorithm different from that of the first dividing unit 13 and acquires two or more words. These two or more words are also referred to as a second division result.

  It is preferable that the second dividing unit 21 has been confirmed to have a certain level of accuracy in the process of dividing a sentence and obtaining two or more words. For example, the second dividing unit 21 acquires two or more words by dividing a sentence by an algorithm of morphological analysis using a Viterbi algorithm.

  The second division result acquisition unit 22 acquires a second division result that is a set of two or more words acquired by the second division unit 21. The second division result acquisition unit 22 is a process that only acquires the second division result from the second division unit 21.

  Further, the second division unit 21 may pass the second division result to the determination unit 23 described later. In such a case, the second division result acquisition unit 22 does not perform any processing, but it may be considered that the second division result acquisition unit 22 has passed the determination result to the determination unit 23.

  The determination unit 23 determines whether or not the first division result and the second division result are different. The first division result is a set of two or more words acquired by the first division unit 13.

  The division information acquisition unit 24 configures division information when the determination unit 23 determines that the first division result and the second division result are different.

  The division information is a character string in a sentence corresponding to a portion where the first division result and the second division result are different, and two or more delimited words corresponding to the character string, and is included in the second division result Two or more words. That is, the division information acquisition unit 24 first identifies a location where the first division result and the second division result are different. Next, the division information acquisition unit 24 acquires a character string in the sentence corresponding to the part from the sentences received by the reception unit 12. Next, the division information acquisition unit 24 acquires two or more division words corresponding to the character string from the second division result. And the division | segmentation information acquisition part 24 comprises the division | segmentation information which has the character string in a sentence, and two or more division | segmentation words. In addition, the character string in a sentence is the word which comprises division | segmentation information.

  The dictionary registration unit 25 accumulates the division information acquired by the division information acquisition unit 24 in the word division dictionary 11.

  The second division unit 21, the second division result acquisition unit 22, the determination unit 23, the division information acquisition unit 24, and the dictionary registration unit 25 can be usually realized by an MPU, a memory, or the like. The processing procedure of the second division unit 21 and the like is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  Next, operation | movement of the word division | segmentation apparatus 2 is demonstrated using the flowchart of FIG. In the flowchart of FIG. 5, the description of the same steps as those in the flowchart of FIG. 2 is omitted.

  (Step S501) The second dividing unit 21 performs division processing on the sentence received by the receiving unit 12, and acquires two or more words. These two or more words are the result of the second division.

  (Step S502) The second division result acquisition unit 22 acquires the second division result.

  (Step S503) The determination unit 23 acquires the first division result.

  (Step S504) The determination unit 23 substitutes 1 for counters i and j. The counter i is a counter for divided words included in the first division result, and the counter j is a counter for divided words included in the second division result.

  (Step S505) The determination unit 23 determines whether or not the j-th divided word is present in the second division result. If the jth divided word exists, the process goes to step S506, and if the jth divided word does not exist, the process returns to step S201.

  (Step S506) The determination unit 23 determines whether or not the i-th divided word in the first division result matches the j-th divided word in the second division result. If they match, go to step S511; otherwise, go to step S507.

  (Step S507) The division information acquisition unit 24 selects from the second division result until the last character of the division word in the first division result and the division word in the second division result match. Get two or more split words. The two or more divided words are divided words that are continuous from the jth divided word.

  (Step S508) The division information acquisition unit 24 configures division information. That is, the division information acquisition unit 24 deletes a delimiter from one or more division words acquired in step S507, and acquires a word. And the division | segmentation information acquisition part 24 comprises division information using the said word and two or more division | segmentation words acquired by step S507. Note that the division information acquisition unit 24 may acquire a word from a sentence received by the reception unit 12 instead of deleting a delimiter from one or more division words acquired in step S507.

  (Step S509) The dictionary registration unit 25 registers the division information configured in step S508 in the word division dictionary 11.

  (Step S510) The determination unit 23 advances the counters i and j to the divided word that matches the last character in step S507.

  (Step S511) The determination unit 23 advances the counters i and j by one each. The process returns to step S505.

  In the flowchart of FIG. 5, the output unit 14 outputs the first division result even when the first division result and the second division result are different. However, when the first division result and the second division result are different, the output unit 14 may output the second division result. The output unit 14 outputs the second division result before the word division dictionary 11 is enhanced to satisfy a predetermined condition, and after the output, the output unit 14 outputs the first division result. Also good.

  Further, in the flowchart of FIG. 5, the processing is ended by powering off or interruption of processing end.

  Hereinafter, a specific operation of the word segmentation apparatus 2 in the present embodiment will be described.

  FIG. 3 shows the word division dictionary 11.

  In this situation, it is assumed that the reception unit 12 has received the sentence “Is there a mistake?” Next, the first dividing unit 13 sets the sentence pointer p to 1. That is, the pointer p is set at a position “between” of the sentences.

  Next, the first division unit 13 searches for a word “wrong” that is a word existing in the word division dictionary 11 and matches the maximum character string from the character “between” corresponding to p in the sentence. And get.

  Next, the first division unit 13 determines whether or not the acquired word “wrong” is a word included in the division information. That is, the first division unit 13 determines that the division information corresponding to the word “wrong” is “− (NULL)”.

  Then, the first division unit 13 adds the acquired word “wrong” to the buffer.

  Next, the first dividing unit 13 calculates the character string length “3” of the word “wrong”. Next, the first dividing unit 13 advances the pointer p by “3” corresponding to the maximum length of the character string, and sets the pointer p to the position “ha” of the sentence.

  Next, the first division unit 13 determines that the division process has not been completed yet.

  Next, the first division unit 13 is a word existing in the word division dictionary 11, and from the character “ha” corresponding to p in the sentence, the word “hana” that matches the maximum length character string. Search and get.

  Next, the first division unit 13 determines whether or not the acquired word “Hana” is a word included in the division information. That is, the first division unit 13 determines that the division information corresponding to the word “Hana” is “− (NULL)”.

  Then, the first dividing unit 13 adds the acquired word “Hana” to the buffer. The first dividing unit 13 adds a delimiter character “/” before the word “Hana” and appends it to the buffer. The current buffer stores “wrong / hana”.

  Next, the first dividing unit 13 calculates the character string length “2” of the word “Hana”. Next, the first dividing unit 13 advances the pointer p by “2” corresponding to the maximum length of the character string, and sets the pointer p to the position “i” of the sentence.

  Next, the first division unit 13 determines that the division process has not been completed yet.

  Next, the first division unit 13 searches the word “Ika” that is a word existing in the word division dictionary 11 and matches the maximum length character string from the character “I” corresponding to p in the sentence. And get.

  Next, the first division unit 13 determines whether or not the acquired word “squid” is a word included in the division information. That is, the first division unit 13 determines that the division information corresponding to the word “squid” is “− (NULL)”.

  Then, the first division unit 13 adds the delimiter “/” and the acquired word “Ika” to the buffer. The current buffer stores “wrong / hana / ika”.

  Next, the first dividing unit 13 calculates the character string length “2” of the word “squid”. Next, the first dividing unit 13 advances the pointer p by “2” corresponding to the maximum length of the character string, and sets the pointer p to a position next to “GA” in the sentence.

  Next, the first division unit 13 determines that the division process has been completed.

  Then, the output unit 14 outputs two or more divided word strings “wrong / hana / ika” in the buffer.

  Next, the second dividing unit 21 performs a dividing process on the sentence “Is there an error” received by the receiving unit 12 and acquires two or more words “incorrect / has / not /”? To do.

  Next, the second division result acquisition unit 22 acquires the second division result “wrong / has / no /”. In addition, the determination unit 23 acquires the first division result “wrong / hana / ika”.

  Next, the determination unit 23 substitutes 1 for counters i and j.

  Next, the determination unit 23 determines that the first divided word exists in the second division result. Next, the determination unit 23 determines that the first divided word in the first division result matches the first divided word in the second division result. Then, determination unit 23 increments counters i and j by one each.

  Next, the determination unit 23 determines that the second divided word exists in the second division result. Next, the determination unit 23 determines that the second divided word “Hana” in the first division result does not match the second divided word “Hana” in the second division result. To do.

  Next, the division information acquisition unit 24 selects 2 from the second division result until the last character of the division word in the first division result matches the division word in the second division result. The above divided words are acquired. That is, the division information acquisition unit 24 acquires “ha / no /” in the second division result.

  Next, the division information acquisition unit 24 removes the delimiter from “ha / no / ka” in the second division result, and acquires the word “has no”. Then, the division information acquisition unit 24 configures the division information by using the word “has no” and two or more division words “ha / no /?”.

  Next, the dictionary registration unit 25 registers the configured division information in the word division dictionary 11. This division information is information having the word “has no” and two or more division words “ha no / no”.

  Next, the determination unit 23 advances the counters i and j to the divided word that matches the last character. That is, the determination unit 23 advances the counter i by 2 and advances the counter j by 3.

  Next, the determination unit 23 determines that the fifth divided word does not exist in the second division result. Then, the process ends.

  As described above, according to the present embodiment, it is possible to enrich the word division dictionary while performing sentence division processing.

  According to the present embodiment, the word dividing device has the second dividing unit 21. However, the second dividing unit 21 may exist in a device outside the word dividing device. FIG. 6 shows a block diagram of the word dividing device 3 in such a case. Here, it is assumed that the word division device 3 receives the second division result from the word division device 4 including the second division unit 21. That is, in such a case, for example, the second division result acquisition unit 22 receives the second division result that is a set of two or more words acquired by the second division unit 21 from the word division device 4.

  Furthermore, the software that implements the word segmentation apparatus in the present embodiment is the following program. That is, this program stores a word division dictionary having one or more words and one or more pieces of division information that is a set of two or more divided words that are the result of dividing the words. A word receiving dictionary that accepts a sentence having one or more characters, and a word that matches a maximum-length character string from a sentence pointer at the head of the sentence accepted by the accepting part. If there are two or more divided words corresponding to the acquired word, a divided word acquisition process is performed in which the two or more divided words are obtained instead of the matching words, and the sentence pointer is After moving to the next character of the matching word, the divided word acquisition process is performed up to the word including the last character of the sentence, and the first division result which is a set of two or more words obtained by dividing the sentence is acquired. The first minute to And parts, is a program, to function as an output unit for outputting the first result of division.

  In the above program, the computer divides the sentence received by the accepting unit using a second dividing unit that obtains two or more words by dividing the sentence by an algorithm different from the first dividing unit. A second division result acquisition unit that acquires a second division result that is a set of words, a determination unit that determines whether the first division result and the second division result are different, and the first division result When the determination unit determines that the second division result is different from the second division result, a character string in a sentence corresponding to a location where the first division result and the second division result are different is obtained, and the different location is obtained. Two or more words included in the corresponding second division result, a division information acquisition unit that forms division information including the acquired character string and the acquired two or more words; Divide information for word division As dictionary registration section for storing in the book, it is preferable to further function.

  In the above program, the computer further functions as a computer further comprising a second dividing unit that obtains two or more words by dividing the sentence received by the receiving unit using an algorithm different from the first dividing unit. That is preferred.

In the above program, it is preferable that the second division unit functions as a unit that divides a sentence and acquires two or more words by an algorithm of morphological analysis using a Viterbi algorithm.
(Experiment 1)

Hereinafter, the result of Experiment 1 using the word segmentation apparatus 1 will be described. The software that implements the word segmentation device 1 is named “MA-2”. As another word segmentation device, “MeCab 0.98”, which is a known technique, was used. “MeCab 0.98” is described in “http://mecab.googlecode.com/svn/trunk/mecab/doc/index.html”. In addition, as another word dividing device, the word dividing device developed by the applicant, the word dividing device “MA-1” using the Viterbi algorithm was also used. In FIG. 7, UTF-8 Japanese text 388.5MB was input to the above three devices, and the processing speed (KB / sec) of each device was measured. “MA-2”, which is the word segmentation device 1, was 4.3 times faster than “MeCab 0.98” and 7.5 times faster than “WebMA2 (Version 3.7.0)”. In addition, according to “MA-2” which is the word dividing device 1, it can be understood that one year of newspaper can be analyzed in about 30 seconds.
(Experiment 2)

  Next, the result of Experiment 2 using the word segmentation apparatus 1 “MA-2” will be described. The result of Experiment 2 is shown in FIG. In Experiment 2, “JUMAN 6.0”, “MeCab 0.98”, “KyTea 0.3.0”, and “ChaSen 2.3.3”, which are known techniques, were used as other word segmentation devices. "JUMAN 6.0" is "http://nlp.ist.i.kyoto-u.ac.jp/index.php?cmd=read&page=JUMAN&alias%5B%5D=%E6%97%A5%E6%9C%AC % E8% AA% 9E% E5% BD% A2% E6% 85% 8B% E7% B4% A0% E8% A7% A3% E6% 9E% 90% E3% 82% B7% E3% 82% B9% E3 % 83% 86% E3% 83% A0JUMAN ”,“ KyTea 0.3.0 ”is“ http://www.phontron.com/kytea/index-en.html ”,“ ChaSen 2.3.3 ”is“ http: / /chasen.naist.jp/hiki/ChaSen/ ”. In this experiment, the processing time was measured when 80,000 web texts were input to the above five devices and analyzed by each device (see FIG. 8). It can be seen that the processing speed of the word segmentation device 1 “MA-2” is extremely faster than others. Note that the algorithm and model of the word segmentation apparatus 1 are “depth-first search + collocation” shown in FIG.

  FIG. 9 shows the external appearance of a computer that executes the program described in this specification to realize the word segmentation device according to various embodiments described above. The above-described embodiments can be realized by computer hardware and a computer program executed thereon. FIG. 9 is an overview diagram of the computer system 300, and FIG. 10 is a block diagram of the system 300.

  In FIG. 9, a computer system 300 includes a computer 301 including a CD-ROM drive, a keyboard 302, a mouse 303, and a monitor 304.

  10, in addition to the CD-ROM drive 3012, the computer 301 includes an MPU 3013, a bus 3014 connected to the MPU 3013 and the CD-ROM drive 3012, a ROM 3015 for storing a program such as a bootup program, and an MPU 3013. And a RAM 3016 for temporarily storing instructions of the application program and providing a temporary storage space, and a hard disk 3017 for storing the application program, the system program, and data. Although not shown here, the computer 301 may further include a network card that provides connection to a LAN.

  A program that causes the computer system 300 to execute the function of the word segmentation device of the above-described embodiment may be stored in the CD-ROM 3101, inserted into the CD-ROM drive 3012, and further transferred to the hard disk 3017. Alternatively, the program may be transmitted to the computer 301 via a network (not shown) and stored in the hard disk 3017. The program is loaded into the RAM 3016 at the time of execution. The program may be loaded directly from the CD-ROM 3101 or the network.

  The program does not necessarily include an operating system (OS), a third-party program, or the like that causes the computer 301 to execute the functions of the word segmentation device according to the above-described embodiment. The program only needs to include an instruction portion that calls an appropriate function (module) in a controlled manner and obtains a desired result. How the computer system 300 operates is well known and will not be described in detail.

  In the above program, in a transmission step for transmitting information, a reception step for receiving information, etc., processing performed by hardware, for example, processing performed by a modem or an interface card in the transmission step (only performed by hardware). Not included) is not included.

  Further, the computer that executes the program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

  In each of the above embodiments, it is needless to say that two or more communication means (terminal information transmission unit, terminal information reception unit, etc.) existing in one device may be physically realized by one medium. .

  In each of the above embodiments, each process (each function) may be realized by centralized processing by a single device (system), or by distributed processing by a plurality of devices. May be.

  The present invention is not limited to the above-described embodiments, and various modifications are possible, and it goes without saying that these are also included in the scope of the present invention.

  As described above, the word segmentation device according to the present invention has an effect that a sentence can be segmented into two or more words at high speed, and is useful as a word segmentation device or the like.

1, 2, 3, 4 Word segmentation device 11 Word segmentation dictionary 12 Reception unit 13 First segmentation unit 14 Output unit 21 Second segmentation unit 22 Second segmentation result acquisition unit 23 Judgment unit 24 Division information acquisition unit 25 Dictionary registration unit

Claims (7)

A word division dictionary that can store one or more words and one or more pieces of division information that is a set of two or more divided words that is a result of dividing the word and the word;
A reception unit for receiving a sentence having one or more characters;
Using the word segmentation dictionary, the maximum length word that matches the character string constituting the sentence accepted by the accepting unit is acquired from the word segmentation dictionary, and two or more segments corresponding to the acquired word A first dividing unit that acquires a word and acquires a first division result that is a set of two or more words obtained by dividing the sentence;
A word segmentation device comprising: an output unit that outputs the first segmentation result. The sentence received by the accepting unit is a set of two or more words divided by using a second dividing unit that obtains two or more words by dividing the sentence by an algorithm different from the first dividing unit. A second split result acquisition unit for acquiring a split result,
A determination unit that determines whether the first division result and the second division result are different;
When the determination unit determines that the first division result and the second division result are different, a character string in a sentence corresponding to a portion where the first division result and the second division result are different is acquired. The division information which acquires the 2 or more words contained in the said 2nd division result corresponding to the said different location, and comprises the division information which has the said word which is the acquired character string, and the said 2 or more acquired words An acquisition unit;
The word division device according to claim 1, further comprising a dictionary registration unit that accumulates the division information in the word division dictionary.   The word division device according to claim 2, further comprising a second division unit that obtains two or more words by dividing the sentence received by the reception unit using an algorithm different from the first division unit. The second dividing unit is
The word segmentation device according to claim 2 or 3, wherein a sentence is divided and two or more words are obtained by a morphological analysis algorithm using a Viterbi algorithm. A data structure of a dictionary for word division having one or more words and one or more pieces of division information that is a set of two or more divided words that is a result of dividing the words and the words,
A reception unit for receiving a sentence having one or more characters;
When a word matching the maximum length character string is acquired from the word segmentation dictionary from the sentence pointer at the head of the sentence received by the reception unit, and there are two or more divided words corresponding to the acquired word Performs divided word acquisition processing for acquiring the two or more divided words instead of the matching words, moves the pointer of the sentence to the next character of the matching words, and then executes the divided word acquisition processing. A first division unit that performs a process up to a word including the last character of and obtains a first division result that is a set of two or more words obtained by dividing the sentence;
The data structure of the word division | segmentation dictionary utilized for the word division | segmentation apparatus provided with the output part which outputs said 1st division | segmentation result. On the recording medium,
Storing a word division dictionary that can store one or more words and one or more pieces of division information that is a set of two or more divided words that is a result of dividing the word and the word;
A word division method realized by a reception unit, a first division unit, and an output unit,
An accepting step in which the accepting unit accepts a sentence having one or more characters;
The first division unit uses the word division dictionary to obtain the maximum length word that matches the character string constituting the sentence accepted in the acceptance step from the word division dictionary, and obtains the word A first division step of obtaining two or more divided words corresponding to the word and obtaining a first division result that is a set of two or more words obtained by dividing the sentence;
A word division method, wherein the output unit includes an output step of outputting the first division result. On the recording medium,
Storing a word division dictionary that can store one or more words and one or more pieces of division information that is a set of two or more divided words that is a result of dividing the word and the word;
Computer
A reception unit for receiving a sentence having one or more characters;
Using the word segmentation dictionary, the maximum length word that matches the character string constituting the sentence accepted by the accepting unit is acquired from the word segmentation dictionary, and two or more segments corresponding to the acquired word A first dividing unit that acquires a word and acquires a first division result that is a set of two or more words obtained by dividing the sentence;
A program for functioning as an output unit for outputting the first division result.