JP3904025B2 - Character string dividing device and recording medium - Google Patents

Description

  The present invention relates to the field of sentence analysis of languages such as Japanese and Chinese that do not have word-by-word separation.

  A language such as Japanese or Chinese is different from a Western language such as English, and when expressed in characters, the word breaks are unclear from the character string because the words are not separated. Therefore, in Japanese sentence analysis, first, it is necessary to perform a dividing process for separating words and phrases from a continuous character string. Various methods for this division have been proposed (see, for example, Patent Document 1).

A well-known division method is a “character type cutting method”. This character type cutting method is a method of dividing words and phrases at high speed by dividing the character string in advance at the transition between hiragana and other character types in the character string. However, if a character string is divided at the transition of all character types, there is a risk of over-division that would otherwise divide too much into a portion that is not a word break, so as a method to prevent this over-division For example, a dictionary using a phrase dictionary is proposed. A phrase dictionary is a dictionary in which characters that are not at the break of a phrase are registered even when the character type changes, and when the input character string is divided at the change of the character type, it is over-divided by comparing with the words contained in this phrase dictionary Have bailout.
Japanese Patent Laid-Open No. 06-096115

  However, since the words included in the phrase dictionary have an indefinite length, enormous operations are required to compare the match / mismatch between each word in the phrase dictionary and the input character string. There was a problem of causing a drop.

  SUMMARY OF THE INVENTION An object of the present invention is to solve the above-mentioned problem by determining whether or not a character string can be divided by only a simple table search.

  A general sentence has a structure in which words are continuous, and when an inputted sentence is divided into words, the word may be divided at the breaks of the words. If you break the sentence into words without knowing the meaning of the input sentence, it may be used instead of being divided where there is a character string included in the word that may be used. You can divide a sentence without error by dividing it at a character string that is not included in any of the words.

  The present invention is intended to efficiently perform this process. The present invention relates to a division prohibition code storage means for storing a plurality of types of two character strings and a code indicating whether or not a division point is between these two characters, and indicates one character among the input character strings. A pointer, a determination means for determining whether or not a hiragana continues immediately after the pointer indicates, and a character indicated by the pointer and a character immediately after the character when the determination means determines that the hiragana continues Refers to the division prohibition code storage means between the character indicated by the pointer and the character immediately after this character when the determination means does not determine that the hiragana character continues, and the means for advancing the pointer without dividing the interval Means for detecting the code; and means for advancing the pointer by dividing the character indicated by the pointer and the character immediately after the character according to the detected code; A character string dividing apparatus having.

  In addition, the present invention provides a computer that stores a plurality of types of two character strings and a division prohibition code storage unit that associates and stores a code indicating whether or not a division point is between the two characters. A pointer that points to one character, a determination unit that determines whether or not a hiragana character follows immediately after the pointer indicates, and a character that the pointer indicates and a character that immediately follows this character when the determination unit determines that a hiragana character continues. Means for advancing the pointer without using the character as a dividing point, and the character indicating the pointer and the character immediately after the character when the determining means does not determine that the hiragana character continues. Referring to the storage means, the means for detecting the code, and the character indicated by the pointer and the character immediately after the character are divided according to the detected code, and the poi Means for advancing the data, a recording medium recording a program to be functionally.

  In addition, even if the code searched with two characters extracted from the sentence indicates that the two characters should not be divided, there may be a break of words in the sentence (that is, this division table shows the sentence This is a sufficient condition to divide for each word, but it is not a necessary condition), but the final division is performed at the stage of morpheme analysis using a word dictionary or a connectability dictionary, and this morpheme analysis is performed quickly If this is used as a pre-division, a reliable division point can be found at high speed.

  According to the present invention, rough but reliable word division can be performed with simple processing.

  A morphological analysis system of the present invention will be described with reference to the drawings. This morphological analysis system performs simple word division on the input character string using a division table in advance, and performs accurate word division using a word dictionary on the character divided character string ( By performing morphological analysis), working time is reduced while performing highly accurate analysis.

  First, the division table will be described. FIG. 1 is a diagram showing an outline of a division table. The division table is a two-dimensional array table in which characters (including numbers, alphabets, kana characters, and kanji characters) used in an input sentence are addresses of respective matrices (vertical and horizontal). The portion surrounded by a thick frame in this division table is a virtual assignment of the character corresponding to the address. Actually, the value calculated from the character code corresponding to the illustrated character (for example, in the character set) And what number character is). "..." in the table indicates that a part of the table is omitted.

  Each column designated by the row address and the column address is 1 bit, and the meaning of this bit is as follows.

  0 (reset): The partial character string composed of the character at the row address and the character at the column address is surely divided into words between the two characters.

  1 (set): In a partial character string composed of characters of a row address and a character of a column address, the two characters may be character strings in the same word, and if divided between them, there is a possibility of excessive division.

  In order to create a partition table in which the bits in each column are set / reset by the above rules, the following partition table creation process is performed. A word is read from the word dictionary 5 (see FIG. 3) for performing morphological analysis, and a two-character partial character string continuous from the word is extracted. The bit of the column designated by using the first character and the second character of the partial character string as the row address and the column address is set.

  For example, the word “solid writing” will be explained. From this word, three types of partial character strings “solid”, “ta writing”, and “writing” can be extracted. Since these partial character strings are included in one word “solid writing”, when this partial character string appears in the input sentence, the word (for example, “solid writing”) is divided by dividing the input sentence between them. There is a possibility of being divided along the way. Therefore, when two characters “BET” are extracted from the sentence, the bit in the column designated with “B” as the row address and “TA” as the column address is set so that the sentence is not divided between these characters. . In the same way, for “ta” and “writing”, the bits in the column designated as “ta” as the row address and “call” as the column address are set, “call” as the row address and “ki” as the column address. Set the bit for the specified field. By performing this process for all words registered in the word dictionary, the division table is a column in which bits are set and bits are reset for all partial character strings that may occur in one word. For partial character strings that are designated by a row address and a column address, it is guaranteed that a word will not be divided in the middle even if a sentence is divided between those characters.

  For the row address and column address of the division table, the character code may be used as it is, or each character may be converted into a simpler code (number or the like) than the character code and used as an address.

  The pre-division device 2 (see FIG. 3) using this division table extracts two characters at a time from the beginning of the input sentence, refers to the division table, and resets the bit in the column designated by the two characters. Only in the case where the character has been divided, a high-speed pre-division is realized by dividing the character (inserting a division mark).

  The reason why the pre-division processing is high-speed is that the number of characters is determined to be two characters, not comparison of unequal length character strings, and the processing performed with these two characters is only a table search. By dividing the sentence input by this processing to a certain extent, the morphological analysis in the subsequent stage only needs to perform analysis for each divided block, and the processing can be greatly reduced. .

  If the language to be analyzed is Japanese, and the characters that can be input are character sets defined in JIS-X-0208, the character set is composed of 8836 characters. Therefore, the partition table is a two-dimensional 8836 × 8836. It becomes a table. This two-dimensional array can be used as it is in consideration of the storage capacity occupied by the table and the speed of referring to the table, but a method using a hash table is also conceivable.

  Further, the determination using the table may be limited to only the first level characters having high usage frequency. In this case, in the pre-division apparatus described later, if other characters (second level or the like) are included in either of the two characters, it may be determined collectively that “no pre-division”. In other words, since the filing frequency is low, the overall processing speed is hardly lowered even if the division processing is performed by the morphological analysis device without performing preliminary division.

  The prior division of the input sentence will be specifically described with a simple example. A case where words are divided for the following two example sentences will be described.

A Thermal oxidation of the embedded silicon layer B Prevention of fluctuation of threshold voltage over time Here, the division table includes “threshold”, “threshold voltage”, “pierce” and “embedding” as shown in FIG. It is assumed that the bit is set in a partial character string extracted from the words “buried metal” “silicon layer” “thermal oxidation” “temporal” “variation” “prevention”.

When the above example sentence is pre-divided with reference to this division table,
A: The embedded silicon layer is thermally oxidized to prevent the fluctuation of the threshold voltage over time. Here, “|” is a mark of a dividing point. In the example sentence A, since the bit of “Dashi” and “Hot” is not set in the above division table, it is determined that division is possible at this portion, and a division mark is inserted. Also, in the example sentence B, since the “change” and “motion prevention” bits are not set in the division table, it is determined that division is possible at this portion, and a division mark is inserted. Note that the reason why the division mark is not set between “time-lapsed” and “threshold” is that the character division method is adopted as the basic division method, and the hiragana characters are not divided. is there. Note that the hiragana characters may be divided without using the character type cutting method. In this way, it is possible to pre-divide an input character string with extremely high accuracy by a simple table search.

  FIG. 3 is a functional block diagram of the sentence analysis apparatus using the division table. The character string 1 to be subjected to morphological analysis is in a language such as Japanese where there is no word-by-word segmentation. In the case of Japanese, a character string described as a kanji-kana mixed sentence is targeted. This character string is first input to the pre-dividing device 2. The pre-dividing device 2 pre-divides this character string using the division table 3.

  The pre-splitting device 2 takes out two characters sequentially from the beginning of the input character string, and determines whether the character string can be split between the two characters with reference to the partition table 3. Then, the decision result based on this division table is determined in consideration of the following decision rule to determine whether or not to finally divide.

  “If the last character of a two-character partial character string is hiragana, it may be over-divided, so it will not be divided even if the bit is reset.” This is because most of the “adjunct words” such as auxiliary verbs are hiragana, but it is preferable that the adjunct words are analyzed together with the previous word by the morphological analyzer 4.

  If it is determined by the above determination that the image can be divided, a division mark is inserted between them.

  Then, the character string that has been pre-divided is input to the morphological analyzer 4.

  The morpheme analyzer 4 includes a word dictionary 5 and a connectability dictionary 6 that are also used to create the partition table. The morpheme analyzer 4 performs morpheme analysis on each block that has been pre-divided by the pre-divider 2 and applies a word to each word. Attribute information such as part of speech is read out, and finally the analysis result is determined based on the possibility of connection of the whole sentence.

  The algorithm of morphological analysis will be described with reference to the example of morphological analysis in FIG.

  (1) A pointer indicating a position (between characters) in the input sentence is prepared. As an initial state, the pointer is placed at position 0 (left side of the first character). A virtual node called the beginning of the sentence is also placed.

  (2) Search the dictionary for words starting from the pointer position. From the position 0 in the example of FIG. 4, “this (conjunction)” and “ko (suffix: pieces)” are searched.

  (3) A link is established between each pair of a word ending at the pointer position (a head node in the case of position 0) and a word starting from the pointer position, if any of them can be connected. The word which cannot be connected with any word among the words which start from a pointer position is excluded. In the case of position 0 in the figure, “this” is linked and “ko” is excluded.

  (4) The right side from the pointer position is examined in order, and the pointer is moved to a position where a word ending there exists. In the example shown in the figure, the position next to position 0 moves to position 2, and the next position moves to position 3.

  (5) Repeat steps (2), (3) and (4) until the pointer reaches the end of the sentence. In the case of the end of the sentence, a virtual node called the end of the sentence is set, the possibility of connection between the word ending at the end of the sentence and the end of the sentence is examined, and only possible ones are linked to the end of the sentence to end the process.

  (6) Finally, the path from the sentence start node to the sentence end node (sequence of nodes and links) is the morphological analysis result for the input sentence.

  Although the basic morphological analysis has been completed by the above processing, as is apparent from the drawing, there are many unsuitable solutions as they are. Therefore, it is necessary to select only plausible solutions according to some priority rule.

  Generally used as a priority rule is to prioritize a solution composed of as long a word as possible or a solution with as few words as possible. Although there is no logical basis for this idea, as in the example shown, a considerable part of the invalid solution contains unnecessarily short words. it can.

  The final morpheme analysis result is determined according to the above priority rules and output to the next stage. The next stage is, for example, a syntax / semantic analyzer.

  When the morphological analysis of the example sentence in FIG. 2 is performed, each of the divided blocks “embedded”, “silicon layer”, “thermally oxidized”, “time-lapse threshold voltage”, “fluctuation”, and “prevention” are described above. Morphological analysis may be performed, and only the possibility of connection is confirmed between consecutive blocks.

The document analysis apparatus described above is often realized by a personal computer and software. Hereinafter, the hardware of the personal computer that executes the function of the sentence analysis apparatus of the above embodiment and the operation of the processing unit will be described.

  FIG. 5 is a diagram showing a configuration of a personal computer that executes the function of the sentence analysis device, and a diagram showing storage areas of the RAM 21 a and the hard disk drive 22. In FIG. 2A, a personal computer main body 20 is composed of a main board 21 equipped with a CPU and a memory, a built-in peripheral device such as a hard disk 22, a CD-ROM drive 23, a floppy disk drive 24, a modem 25, and the like. . The personal computer main body 20 is connected to a monitor 30, a keyboard 31, a mouse 32, and the like. The application program for executing the function of the sentence analysis apparatus is normally stored in the hard disk 22 and is read into the RAM of the main board 21 when executing the processing operation. The application program (including a division table, a word dictionary, and a connectability dictionary) is supplied by a CD-ROM 33 set in the CD-ROM drive 23, a floppy disk 34 set in the floppy disk drive 24, and the like. Installed on the hard disk 22. Alternatively, the operation program may be downloaded from the server computer 40 connected via the modem 25.

  In FIG. 5B, when this personal computer is activated, the system program 210 is read from the hard disk drive 22 into the RAM 21a. When the sentence analysis program (application program) is started, an application area 211 is secured for execution of the application program, in which the sentence analysis program 212 and the division table 213 are read and the work area 214 is Secured. In the work area 214, an area for storing a sentence to be analyzed, a word dictionary of the hard disk 22, a cache area of a connectability dictionary, and the like are set.

  In FIG. 2C, the hard disk drive 22 stores a system program 220 and a text analysis program 221 that is an application program. The system program 220 is read into the RAM 21a when the personal computer is activated, and the sentence analysis program 221 is read into the RAM 21a when the program is activated. Further, the division table 222, the word dictionary 223, and the connection possibility dictionary 224 are stored together with the application program. The division table 222 is read into the RAM 21a together with the program when the sentence analysis program is started. However, the word dictionary 223 and the concatenation possibility dictionary 224 are large in size and are referred to while being stored on the hard disk 22, and one of them. Are loaded into the RAM 21a as a cache.

  FIG. 6 is a flowchart showing the pre-division operation of the personal computer. First, a pointer is placed on the first character (on the left side) of the input character string (s1). In s2, it is determined whether the pointer has reached the end of the input character string, and if it has reached the end, the operation is terminated. If it has not reached the end, the operation after s3 is executed. Therefore, at first, the process proceeds from s1 to s3.

  In s3, it is determined whether the character next to the character with the current pointer is hiragana. In the case of hiragana, the division point is not set between the character with the pointer and the next character (s5), the pointer is advanced one step (s8), and the process returns to s2. That is, in this process, the character type cutting method is adopted as a basic division method, and since the above rule "Do not divide when the next character in the partial character string is hiragana" is adopted. This is because there is no division in front of hiragana.

  On the other hand, if the next character of the pointer is not hiragana, the division table is referred to by the character with the pointer and the partial character string of the next character, and the bit in the column is read (s4). If the read bit is 1, there is a possibility of excessive division, so this portion is not divided (s5), and the pointer is advanced by one (s8). When the read bit is 0, it is possible to divide at this position, so a mark is inserted between the character with the pointer and the next character as a dividing point (s7), and the pointer is advanced by 1 (s8). The above operation is repeated until the pointer reaches the end of the sentence (s2).

  FIG. 7 is a flowchart showing a processing operation when the division table is created. First, the area of the partition table is set, and the bits in all the columns are initialized to 0 (s11). Then, the first word is read from the word dictionary (s12). A pointer is placed on the first character of the read word (s14). The division table column is searched using the character with the pointer and the next character as an address, and the bit of the column is set to 1 (s17). Then, the pointer is shifted to the right by one (s18). The processing from s15 is repeated until the pointer reaches the last character of the word. When the pointer reaches the last character of the word, the next word is read from the dictionary (s16), and the operations after s13 are repeated. This is executed until the word to be read is completed (s13), and the division table is completed.

  This division table creation operation may be performed in advance according to the creation of the word dictionary, and the division table may be supplied together with the sentence analysis program. Further, even if the same division table can be created by other procedures such as simple copying, a medium storing the division table, a sentence analysis device using the division table, and a sentence analysis method are also claimed in this application. It is included in the range.

The figure which shows the example of the division | segmentation table used for this invention The figure which shows the specific example of a division | segmentation table and pre-division processing Functional block diagram of a sentence analysis apparatus according to an embodiment of the present invention Diagram showing an example of morphological analysis Block diagram of a personal computer that implements the sentence analysis device Flow chart showing operation of the personal computer Flow chart showing partition table creation operation

Explanation of symbols

1 ... input character string, 2 ... pre-dividing device, 3 ... division table,
4 ... Morphological analyzer, 5 ... Word dictionary, 6 ... Concatenation possibility dictionary,
7 ... Word information output 21 ... Main board, 21a ... RAM, 22 ... Hard disk drive,
23 ... CD-ROM drive, 33 ... CD-ROM,
24 ... floppy disk drive, 34 ... floppy disk,
31 ... Keyboard, 32 ... Mouse

Claims (2)

A division prohibition code storage means for storing a plurality of types of two character strings and a code indicating whether or not a division point is set between the two characters;
A pointer that points to one character in the input string;
A judging means for judging whether or not the hiragana follows immediately after the pointer indicates;
Means for advancing the pointer without using a dividing point between the character indicated by the pointer and the character immediately after the character when it is determined that the hiragana character continues by the determining means;
Means for detecting the code by referring to the division prohibition code storage means with the character indicated by the pointer and the character immediately after the character when the determination means does not determine that the hiragana character continues;
Means for advancing the pointer by dividing the character indicated by the pointer and the character immediately after the character according to the detected code;
A character string dividing device having Computer
A division prohibition code storage means for storing a plurality of types of two character strings and a code indicating whether or not a division point is set between the two characters;
A pointer that points to one character in the input string;
A judging means for judging whether or not the hiragana follows immediately after the pointer indicates;
Means for advancing the pointer without using a dividing point between the character indicated by the pointer and the character immediately after the character when the determination unit determines that the hiragana character continues;
Means for detecting the code by referring to the division prohibition code storage means with the character indicated by the pointer and the character immediately after the character when the determination means does not determine that the hiragana character continues;
Means for advancing the pointer by dividing the character indicated by the pointer and the character immediately after the character according to the detected code;
A recording medium that records a program that allows it to function.

Images