JPH07141347A - Method for segmenting japanese character string - Google Patents

Abstract

PURPOSE:To increase the processing speed and to reduce a program size by segmenting a Japanese character string by a character string starting with a KANJI (Chinese character) or a KATAKANA (square form of Japanese syllabary) and ending with a HIRAGANA (cursive form a Japanese syllabary) or a punctuation mark. CONSTITUTION:When a program is started, a variable is first decleared (S1), and a single line (single end of paragraph) is read out of a file and sent to an array L (S2). Then, a segmenting position is decided (S4), and it is decided whether a variable KPOS=0 is satisfied or not (S5). That is, it is decided whether the array L starts with a KANJI or a KATAKANA. When KPOS=0 is satisfied, a character string including 0 through HPOS is substituted for an array X (S6), and a line feed code is added to the array X and sent to a reading file (S7). It is decided whether an array L [i] is equal to a punctuation mark or not (S12). If not, the characters following (i+1) characters are substituted again for the array L (S13). If an array SL [i] is decided as a punctuation mark in S12, the characters following (i+2) charactersare substituted again for the array L (S14).

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a Japanese character string segmentation method suitable for applications such as Japanese spell check, machine translation, and automatic keyword extraction.

[0002]

2. Description of the Related Art In English (English), words are divided by white space characters such as spaces and tabs. Therefore, a character string is divided into words for the purpose of spell checking.

However, in the case of Japanese, it is difficult to classify the character string because the character string (Japanese sentence) is not classified by the white space character. Therefore, conventionally, a character string has been classified according to the change of the character type such as kanji or hiragana, or it has been classified into a character string such as a subject or a predicate based on an analysis result of a sentence structure.

[0004]

However, in the method of segmenting a character string at the point where the character type changes, the segmentation becomes too fine, and more character strings containing only hiragana are segmented. However, it cannot be determined whether it is a futuristic kana, an adjective, or an adverb. Also,
It is not possible to judge whether a character string consisting only of hiragana is suitable as a sending kana by itself. Furthermore, there is a problem that the subject and the predicate can hardly be discriminated.

On the other hand, in the method of dividing the character string from the analysis result of the text structure, a relatively good division result can be obtained, but the size of the program for analyzing the text structure becomes large, so that the processing speed is slow. Moreover, there is a problem that it cannot be incorporated in a small device.

The present invention has been made in view of the above problems, and character strings can be divided to such an extent that the meanings can be independently discriminated, and the processing speed is high and the apparatus can be incorporated into a small device. The purpose is to make the program size compact.

[0007]

In order to achieve this object, the Japanese character string segmentation method of the present invention is to segment a Japanese character string by a character string starting with a kanji or katakana or ending with a hiragana or punctuation mark. Is configured.

[0008]

[Function] In the Japanese character string segmentation method having the above configuration, the Japanese character string is segmented only by a character string starting with a kanji or katakana and ending with a hiragana or punctuation mark.
The processing speed is fast and the program size can be made compact. Further, it is possible to divide the character string to such an extent that the meaning can be independently discriminated without unnecessary unnecessary fine division.

[0009]

Embodiments of the present invention will be described below with reference to the drawings. The embodiment exemplifies a case in which the present invention is applied to a program that divides a character string read from a file and outputs the divided character string to another file. However, in the actual program, the program will be created assuming that 1-byte characters and 2-byte characters are mixed, but in order to simplify the explanation, the contents of the file to be read are double-byte characters. Will be described as a single character and double-byte characters will be treated as one character.

FIG. 1 is a flowchart showing an embodiment of a Japanese character string segmentation method according to the present invention.

In FIG. 1, when the program starts, variables are first declared (step S1). The variables here are i, KPOS, H used in the program.
POS, KFNEW, KFOLD, HFNEW, HFO
There are seven variables of LD, and 0 is assigned to each variable at the same time as the declaration.

Next, one line (one paragraph) is read from the file into the array L (step S2). Array L, array L
Character codes for one line are stored in order from [0]. As a result of reading one line from the file in step S2, if the file end is reached, the program is terminated, and if it is not the file end, the process proceeds to the next step S4 to perform the delimiter discrimination processing. It should be noted that in step S2, the following description will be made assuming that one row of "This processing is performed very quickly. Therefore, the waiting time of the user can be minimized."

In step S4, a process for discriminating the delimiter position is performed as shown in FIG.

In FIG. 2, first, it is determined whether the array L [i] is a kanji or katakana (step S4).
1). If it is a kanji or katakana, the process proceeds to step S43 and 1 is substituted for KFNEW. Further, step S4
If it is determined that KFOLD is 0 in step 4, KPOS
The value of i is substituted into (step S45), and step S4
Go to 6. If KFOLD is 1, the process immediately proceeds to step S46 (step S44).

In step S44, KFOLD is 0 and K
When FNEW is 1, it means that a new character string of kanji or katakana has begun, so that the position i is stored in KPOS in step S45.

When it is determined in step S41 that the array L [i] is not a kanji character or katakana, 0 is assigned to KFNEW and then the process proceeds to step S46 (step S4).
2).

Next, it is determined whether the array L [i] is a hiragana or a punctuation mark (step S46). If it is neither a hiragana character nor a punctuation mark, the process proceeds to step S48 and 0 is assigned to HFNEW. Further, in step S49, K
If it is determined that FOLD is 1 and that the array L [i] is a punctuation mark in step S50, the HPO
The value of i-1 is substituted for S (step S51), and the process proceeds to step S53. KFOLD is 1 in step S49
If it is determined that the array L [i] is not a punctuation mark in step S50, the value of i is assigned to HPOS (step S52), and the process proceeds to step S53.
If it is determined in step S49 that KFOLD is 0, the process immediately proceeds to step S53. If it is determined in step S46 that the array L [i] is a hiragana or punctuation mark, 1 is assigned to HFNEW, and then the process proceeds to step S53 (step S47).

In step S49, HHOLD is 1, and
When HFNEW is 0, it means that the character string ends with a hiragana or a punctuation mark, so that the position is stored in the HPOS in step S51 or step S52. In step S51, i
The reason for substituting the value of -1 is that the position immediately before the punctuation mark is stored.

In step S53, KPOS ≧ 0, HP
It is determined whether or not OS ≧ 1 and KPOS <HPOS are satisfied by a logical product. This means that it is determined whether or not there is a character string starting with a kanji or katakana or ending with a hiragana or punctuation mark in one line (array L) read from the file in step S2. In the example shown above, the logical product is first established when the character strings up to “this process” are scanned while incrementing i. That is, KPOS =
2, HPOS = 4, 2 <4 holds as a logical product. In addition,
Note that the beginning of the array L is the array L [0], which starts from 0, so that KPOS = 3, HPOS = 5, and 3 <5 are not satisfied.

When the logical product is not established in step S53, the value of KFNEW is substituted into KFOLD and the value of HFNEW is substituted into HFOLD in step S54, and i is further incremented before returning to step S41. When the logical product is established in step S53, the next step S5
Move to.

In step S5, it is determined whether KPOS = 0 holds. This means to determine whether the array L starts with Kanji or Katakana. When KPOS = 0, a character string from 0 to HPOS is assigned to the array X (step S6), a line feed code is attached to the array X and output to the write file (step S7).

When KPOS is not 0, the character strings from 0 to KPOS-1 are substituted into the array X (step S
8) Then, add a line feed code to the array X and output to the write file (step S9). Then, KPOS is added to the array Y.
To the character string from HPOS (step S1
0), the line feed code is added to the array Y and output to the write file (step S11).

In the above-described example, since the logical product of step S53 is established at the time when the character strings up to "this process" are scanned and the process proceeds to step S5, 0 to 1
(That is, KPOS-1) the character string "this" up to the array X
(Step S8), add a line feed code, and output to a write file (step S9). Then 2
(That is, KPOS) to 4 (HPOS) character strings "processed" are assigned to the array Y (step S10), a line feed code is added and output to a write file (step S11).

As can be seen from the fact that the character string "this" described here is output in step S9, even if it is delimited by "a character string starting with a kanji or katakana or ending with a hiragana or punctuation mark", it is before and after the character string. One of the features of the present invention is that character strings other than the "character strings that start with a kanji or katakana and end with a hiragana or punctuation mark" existing in 3) can be delimited. This means that if there is a pronoun such as "that" or "this" before "a character string that begins with a kanji or katakana and ends with a hiragana or punctuation mark" or if there is an adverb such as "immediately" It means that it can be effectively separated.

Step S7, Step S9, Step S
When the output to the writing file is completed in 11, preparations are made for performing the next delimiter determination processing in step S12 and subsequent steps.

In step S12, it is determined whether or not the array L [i] is a punctuation mark, and if it is determined that the array L [i] is not a punctuation mark, the array L is replaced with i + 1 characters or more (step S12). S13). If it is determined in step S12 that the array L [i] is a punctuation mark, i + 2 characters and subsequent characters are substituted into the array L again (step S14). The reason why the numbers to be added are different between step S13 and step S14 is that they are deleted so that no punctuation marks are included in the delimited character string. It is also possible not to delete the punctuation marks.

Next, in step S15, i, KP
OS, HPOS, KFNEW, KFOLD, HFNE
0 is assigned to the seven variables W and HHOLD, and it is then determined whether or not the scanning of the array L is completed and the end of the line is reached (step S16). If it is not the end of line, step S4
The division determination process (step S4) and subsequent steps are repeated for the array L that has been replaced. If it is at the end of the line, the process returns to step S2, a new line is read from the file into the array L, and the delimiter determination process (step S4) and subsequent steps are repeated. These repeated operations are performed until the file end is detected in step S3.

In the input example shown above, "This processing is performed very quickly. Therefore, the waiting time of the user can be minimized." Is output in steps S7, S9, and S11. As a result, the following is output to the export file.

This process is very fast and therefore minimizes user latency

The present invention has been described above with reference to the embodiments.
Various modifications are possible according to the technical idea of the present invention. For example, in the above-described embodiment, the case where the punctuation marks are “,” and “.” Has been described, but “,” and “.” Can be used as the punctuation marks.
Also, parentheses such as "" () [] [] can be added to the delimiters. Also,
Although the punctuation marks are not output in the above-mentioned embodiment, the punctuation marks are useful as information of the sentence structure,
You can also output punctuation.

Further, in the above-described embodiment, the example of outputting to the writing file in step S7, step S9 and step S11 has been described, but instead of outputting to the writing file, the user is asked whether or not to edit. Or pass it as a data value to the translator.

In addition, including the division of a Japanese character string by "a character string that starts with a kanji or katakana and ends with a hiragana or punctuation mark,""starts with a kanji or katakana and is followed by hiragana or punctuation marks, The present invention can be further derived so that the division position of the Japanese character string is determined by the “character string ending with katakana”. The present invention does not prohibit the combination or combination with other analysis methods.

Further, in the above-described embodiment, the delimited character strings are output as they are. However, after delimited, the pronouns and adverbs are extracted by the prefix match search and the particle and auxiliary verb are extracted by the suffix match search. It is also possible to output after performing analysis processing such as extraction.

[0034]

As described above, according to the Japanese character string segmentation method of the present invention, a Japanese character string is segmented by a character string starting with a kanji or katakana and ending with a hiragana or a punctuation mark, so that the processing speed is high. In addition, the program size can be made compact. In addition, it is possible to divide the character string to such an extent that the meaning can be independently determined without unnecessarily finely dividing.

[Brief description of drawings]

FIG. 1 is a flowchart showing an embodiment of a Japanese character string segmentation method according to the present invention.

FIG. 2 is a flowchart showing an embodiment of a Japanese character string segmentation method according to the present invention.

[Explanation of symbols]

 Steps S1 to S16 Steps S41 to S54

Claims (1)

[Claims] 1. A method for classifying a Japanese character string, wherein a Japanese character string is classified by a character string starting with a kanji or katakana and ending with a hiragana or a punctuation mark.