JP2010102559A - Apparatus and method for data compression and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus and a method for data compression and a program, for improving compressibility when compressing text data including Kanjis (Chinese characters). <P>SOLUTION: The data compression apparatus 1 for performing the data compression to the text data including Kanjis is used. The data compression apparatus 1 includes a Kanji separation part 6 for separating the Kanji included in the text data from characters other than the Kanji included in the text data, and a Kanji data generation part 7 for generating Kanji list data for specifying the separated Kanji and Kanji position data for specifying the position of the Kanji in the text data. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a data compression apparatus, a data compression method, and a program for compressing text data.

  Conventionally, data compression has been performed on digitized data for efficient data transfer and effective use of a data storage area. Data compression is roughly classified into lossless compression and lossy compression. Of these, in the lossy compression, as the name implies, a part of the original data is lost, and therefore it is impossible to completely restore the original data from the compressed data. However, the irreversible compression is superior in that a very high compression ratio can be achieved compared to the lossless compression.

  For example, in audio data, image data, etc., except for some data, even if leveling sounds outside the human audible range or indistinguishable color change, there is little data lost, These data can be used as information even if they cannot be completely restored. Therefore, irreversible compression is mainly used for image data, audio data, and the like, coupled with the fact that a high compression ratio is required because of the large data size (for example, Patent Document 1). To 3).

  On the other hand, when irreversible compression is performed on text data such as program data and document data, the text is discrete data in units of characters and is vulnerable to noise, so that restoration becomes extremely difficult. In this case, the program data has a problem that the program does not operate normally after restoration. In addition, the document data is garbled after restoration, which causes a problem that it cannot be used as information.

  As described above, it is difficult to perform lossy compression on text data such as program data and document data, and lossless compression is mainly used for compression of text data. However, improvement of the compression rate is also demanded for text data, and various compression methods have been conventionally proposed (see, for example, Patent Documents 4 to 6).

  For example, Patent Document 4 discloses a data compression technique that reduces the amount of data when document data is transmitted as an e-mail. In the data compression technique of Patent Document 4, data compression is performed by conversion of full-width hiragana to half-width katakana, conversion of full-width alphanumeric characters to half-width alphanumeric characters, conversion of specific character strings to specific symbols, and the like.

  Patent Document 5 discloses a data compression technique for performing reversible compression such as LZH or ZIP for text data and irreversible compression for image data when compressing document data to which an image is added. is doing. According to the data compression technique of Patent Document 5, it is possible to improve the compression rate for document data to which an image is added.

Further, Patent Document 6 discloses a data compression technique for compressing document data for printing sent from a computer device to a printer. In the data compression technique of Patent Document 6, first, text data is converted into dot map data, and then the obtained dot map data is compressed, so that the compression rate is improved.
JP-T-2001-507195 JP-T-2004-523933 JP 2005-500709 A Japanese Patent Laying-Open No. 2003-196268 (FIG. 5) JP 2005-182735 A JP 2006-120145 A

  However, in the data compression technique disclosed in Patent Document 4, data compression is not performed on kanji characters included in document data unless a corresponding symbol is set. In the data compression technique disclosed in Patent Document 4, there is a limit in improving the compression rate of text data.

  In the data compression technique disclosed in Patent Document 5, text data is only compressed by a general lossless compression algorithm such as LZH or ZIP. Therefore, it is difficult to improve the compression rate of text data, especially for document data including kanji.

  Furthermore, in the data compression technique disclosed in Patent Document 6, text data is converted into dot map data and then compressed. However, in order to improve the compression ratio, it is necessary to lower the quality of the text. For this reason, particularly when the text data includes Chinese characters, it is extremely difficult to improve the compression rate.

  An object of the present invention is to provide a data compression apparatus, a data compression method, and a program capable of solving the above problems and improving the compression rate when compressing text data including Chinese characters.

In order to achieve the above object, a data compression apparatus according to the present invention is a data compression apparatus that performs data compression on text data including Chinese characters,
A kanji separating unit that separates the kanji included in the text data from characters other than the kanji included in the text data;
And a kanji data generation unit that generates kanji list data for specifying the separated kanji and kanji position data for specifying the position of the kanji in the text data.

In order to achieve the above object, a data compression method according to the present invention is a data compression method for performing data compression on text data including kanji,
(A) separating the Chinese characters included in the text data from characters other than the Chinese characters included in the text data;
(B) generating kanji list data for specifying the separated kanji and generating kanji position data for specifying the position of the kanji in the text data.

Furthermore, in order to achieve the above object, a program according to the present invention is a program for performing data compression on text data including kanji using a computer device,
In the computer device,
(A) separating the Chinese characters included in the text data from characters other than the Chinese characters included in the text data;
(B) generating kanji list data for specifying the separated kanji and generating kanji position data for specifying the position of the kanji in the text data.

  With the above features, according to the data compression device, data compression method, and program of the present invention, the compression rate when compressing text data including Chinese characters can be improved.

(Embodiment)
Hereinafter, a data compression apparatus, a data compression method, and a program according to an embodiment of the present invention will be described with reference to FIGS. Initially, the structure of the data compression apparatus in this Embodiment is demonstrated using FIGS. 1-3.

  FIG. 1 is a block diagram showing a configuration of a data compression apparatus according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a result of processing performed by the data compression device according to the embodiment of the present invention. FIG. 3 is a diagram illustrating an example of data generated by the data compression apparatus according to the present embodiment.

  The data compression apparatus 1 shown in FIG. 1 performs data compression on text data 12 (see FIG. 2) including Chinese characters. As shown in FIG. 1, the data compression apparatus 1 includes a kanji processing unit 5 having a kanji separation unit 6 and a kanji data generation unit 7.

  The kanji separating unit 6 separates kanji characters included in the text data 12 to be subjected to data compression from characters (kana and alphanumeric characters) other than kanji characters also included in the text data. The kanji data generation unit 7 generates kanji list data for specifying the separated kanji and kanji position data for specifying the position of the kanji in the text data 12 (see FIG. 3).

  In this embodiment, kanji characters that are difficult to compress are separated from kana and alphanumeric characters and converted into kanji list data and kanji position data. As shown in FIG. 3, the kanji list data can be composed of kanji codes created by combining alphanumeric characters, and digital compression of the kanji list data is easy. Further, as shown in FIG. 3, the kanji position data only needs to indicate the location in the document, and can be composed of alphanumeric characters, so that digital compression of the kanji position data is easy. For this reason, according to the present embodiment, it is possible to compress text data including Chinese characters at a high compression rate.

  Next, the configuration of the data compression device 1 will be described more specifically. As shown in FIG. 1, in this embodiment, the data compression apparatus 1 includes a data reading unit 2, a morpheme analysis unit 3, a dictionary 4, an auxiliary compression unit 9, a data compression unit 10, in addition to the kanji processing unit 5. And a data output unit 11. The kanji processing unit 5 further includes a kanji-kana conversion unit 8 in addition to the kanji separation unit 6 and the kanji data generation unit 7.

  The data reading unit 2 reads text data input from the outside and outputs it to the morphological analysis unit 3. As the text data to be input, for example, Japanese text data 12 including kanji as shown in FIG. In practice, the text data 12 is input in a coded state.

  In the present embodiment, the text data is input to the data reading unit 2 via an input device such as a keyboard and a mouse connected to the data compression device 1 or from an external computer device via a network. In this case, as the data reading unit 2, an interface for connecting the outside and the data compression apparatus 1 is used. The text data may be provided in a state stored in a recording medium. In this case, a reading device is used as the data reading unit 2.

  The morpheme analysis unit 3 performs morpheme analysis with reference to the dictionary 4 and decomposes the text data in units of words. The dictionary 4 stores a word list with information such as part of speech, kanji notation, and reading. In the present embodiment, the morphological analysis algorithm is not particularly limited, and the morphological analysis unit 3 may perform a morphological analysis based on a rule or may perform a morphological analysis based on a probabilistic language model. .

  When the morpheme analysis is completed, the morpheme analysis unit 3 outputs data 13 (see FIG. 2) indicating the result of the morpheme analysis to the kanji processing unit 5. In the present embodiment, the morphological analysis unit 3 adds a reading of each Chinese character (not shown in FIG. 2) to the data 13 in order to enable conversion processing in the Chinese character kana conversion unit 8 described later. Yes.

  As described above, the kanji separating unit 6 separates kanji included in the text data 12 from characters other than kanji. At this time, in this embodiment, the kanji separating unit 6 separates kanji based on the character code. In addition, the kanji separation unit 6 extracts a corresponding kanji code for each separated kanji, and further extracts information for specifying the position of the kanji (kanji position information). In the present embodiment, for example, JIS code, shift JIS code, Unicode, etc. are used as the Kanji code. In addition, as the kanji position information, for example, a number in the case of sequentially numbering the first character as “1” is used.

  The kanji data generation unit 7 uses the kanji code extracted by the kanji separation unit 6 to generate kanji list data shown in FIG. Furthermore, the kanji data generation unit 7 also generates kanji position data shown in FIG. 3 using the kanji position information extracted by the kanji separation unit 6.

  The kanji / kana conversion unit 8 generates the kana conversion text data 14 (see FIG. 2). The kana conversion text data 14 is data composed only of characters other than the kanji obtained by converting the kanji included in the text data 12 (see FIG. 2) into kana. In the present embodiment, the kanji / kana conversion unit 8 determines whether the character code is a kanji code for each character included in the text data 12. Then, the kanji / kana conversion unit 8 converts a character determined to be a kanji code into a corresponding kana by using the result of morphological analysis. Thereby, kana conversion text data 14 is generated.

  As shown in FIG. 2, the kana conversion text data 14 is composed of characters other than Kanji and is data in which the same character type is continuous. Therefore, the kana converted text data 14 is more compressed than the data obtained by removing only the kanji from the text data 12 (for example, “GA / RU / TO / DE ///” (see data 13)). The target common area increases. In the present embodiment, the kana conversion text data 14 is created to further improve the compression rate.

  The auxiliary compression unit 9 reduces the types of character codes included in the kana conversion text data 14 and compresses the data amount of the kana conversion text data 14. Specifically, the auxiliary compression unit 9 converts plosives, muddy sounds, and prompt sounds into clear-sound characters (for example, pa → haga → ka, tsu → tsu), katakana hiragana, , Convert katakana and hiragana to romaji. In FIG. 2, data 15 indicates data obtained by auxiliary compression. In the present embodiment, the compression by the auxiliary compression unit 9 may be performed as necessary, and may be performed in a manner that is not performed.

  The data compression unit 10 receives the kanji list data, the kanji position data, and the kana converted text data 14 (data 15 when the auxiliary compression is performed by the auxiliary compression unit 9) shown in FIG. Specifically, the data compression unit 10 performs digital compression of data using, for example, a reversible compression algorithm such as LZH format, LZ (Lempel-Ziv) format, or ZIP format.

  As described above, actually, the kanji list data, the kanji position data, and the kana conversion text data are composed of alphanumeric characters. Therefore, the data compressed by the data compression unit 10 is compressed at a higher compression rate than when compressed by the techniques disclosed in Patent Documents 4 to 6 in the background art column. According to the present embodiment, Japanese text data 12 (see FIG. 2) including kanji is compressed at a high compression rate.

  Furthermore, in the present embodiment, the data compression unit 10 discards the kanji list data and the kanji position data in accordance with an instruction from the outside, specifically, an instruction from the operator of the data compression apparatus 1, and performs kana conversion. Digital compression can be performed only on the text data 14. In this case, it is impossible to restore the compressed data to the original text data 12, but it is not necessary to perform reversible compression and is effective when only a high compression rate is required.

  The compressed data is output to the outside by the data output unit 11. Specifically, the compressed data is output to another computer device or a recording medium connected to the data compression device 1 via a network.

  Next, a data compression method according to the embodiment of the present invention will be described with reference to FIG. FIG. 4 is a flowchart showing a flow of processing performed in the data compression method according to the embodiment of the present invention. The data compression method in the present embodiment can be implemented by operating the data compression apparatus 1 shown in FIG. Therefore, hereinafter, the description of the data compression method according to the present embodiment will be made together with the description of the operation of the data compression apparatus shown in FIG. 1, with appropriate reference to FIGS.

  First, as shown in FIG. 4, when text data 12 (see FIG. 2) is input to the data compression apparatus 1, the data reading unit 2 reads the input text data 12 (step S1) and reads the read text. Data 12 is input to the morphological analyzer 3.

  Next, when the text data 12 is input, the morpheme analysis unit 3 performs morpheme analysis on the dictionary 4 while referring to the dictionary 4, and generates data 13 to which the reading of each kanji is added (step). S2). The morpheme analysis unit 3 inputs the generated data 13 to the kanji processing unit 5.

  Subsequently, the kanji processing unit 5 determines whether or not kanji is included in the text data 12 (step S3). If the kanji is not included, step S5 is executed by the auxiliary compression unit 9. On the other hand, when kanji is included, the kanji processing unit 5 performs kanji processing (step S4).

  Specifically, in step S4, first, the kanji separation unit 6 separates kanji included in the text data 12 from characters other than kanji (kana and alphanumeric characters), and kanji codes corresponding to the separated kanji characters. And kanji position information is extracted. Next, in step S4, the kanji data generation unit 7 generates kanji list data and kanji position data using the extracted kanji code and kanji position information (see FIG. 3). Further, in step S4, the kanji / kana conversion unit 8 generates the kana converted text data 14.

  Next, the auxiliary compression unit 9 executes auxiliary compression for reducing the amount of data for the kana converted text data 14 (step S5). In addition, when step S4 is not performed, the auxiliary compression part 9 performs auxiliary compression with respect to the data 13 produced | generated by the morphological analysis.

  Subsequently, the data compression unit 10 performs data compression on the kanji list data, the kanji position data, and the kana converted text data 14 after the auxiliary compression (step S6). When step S4 is not executed, the data compression unit 10 performs data compression on the data 13 after the auxiliary compression.

  Thereafter, the data output unit 11 outputs the data compressed in step S6 to the outside (step S7). When step S7 ends, the data compression apparatus 1 ends the process. Thus, if the data compression method in this Embodiment is performed, a Chinese character will be compressed separately from characters other than a Chinese character, and the text data 12 containing a Chinese character will be compressed with a high compression rate.

  Moreover, the program in this Embodiment is a program which makes a computer apparatus embody step S1-S7 shown in FIG. Therefore, the data compression apparatus 1 can be realized by installing this program in a computer and executing this program. In this case, a central processing unit (CPU) of the computer functions as the morphological analysis unit 3, the kanji processing unit 5, the auxiliary compression unit 9, the data compression unit 10, and the data output unit 11, and performs the processes of steps S1 to S7.

  In the present embodiment, the dictionary 4 can be realized by storing a data file constituting the dictionary 4 in a storage device such as a hard disk provided in the computer device. The dictionary 4 can also be realized by mounting a recording medium in which a data file is stored in a reading device connected to a computer device. Furthermore, the dictionary 4 may be provided by a computer device different from the computer device in which the program is installed.

FIG. 1 is a block diagram showing a configuration of a data compression apparatus according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a result of processing performed by the data compression device according to the embodiment of the present invention. FIG. 3 is a diagram illustrating an example of data generated by the data compression apparatus according to the present embodiment. FIG. 4 is a flowchart showing a flow of processing performed in the data compression method according to the embodiment of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Data compression apparatus 2 Data reading part 3 Morphological analysis part 4 Dictionary 5 Kanji processing part 6 Kanji separation part 7 Kanji data generation part 8 Kanji kana conversion part 9 Auxiliary compression part 10 Data compression part 11 Data output part 12 Text data 13 Morphological analysis Data after 14 Kana conversion text data 15 Data after auxiliary compression

Claims (15)

A data compression device that performs data compression on text data including kanji,
A kanji separating unit that separates the kanji included in the text data from characters other than the kanji included in the text data;
A data compression apparatus comprising: a kanji list data for specifying the separated kanji characters; and a kanji data generation unit for generating kanji position data for specifying a position of the kanji in the text data. A kanji conversion unit that converts kanji contained in the text data into kana and generates kana conversion text data composed only of characters other than kanji.
The data compression apparatus according to claim 1.   The data compression apparatus according to claim 1 or 2, further comprising a data compression unit that performs digital compression on the kanji list data, the kanji position data, and the kana converted text data.   The data according to claim 3, wherein the data compression unit discards the kanji list data and the kanji position data and performs digital compression only on the kana converted text data when instructed from outside. Compression device. Morphological analysis is performed on the text data, and further includes a morphological analysis unit that decomposes the text data into words.
The data compression apparatus according to claim 2, wherein the kanji / kana conversion unit converts the kanji contained in the text data to kana using the result of the morphological analysis to generate the kana conversion text data. A data compression method for performing data compression on text data including kanji,
(A) separating the Chinese characters included in the text data from characters other than the Chinese characters included in the text data;
And (b) generating kanji list data for specifying the separated kanji and kanji position data for specifying the position of the kanji in the text data. 7. The data compression method according to claim 6, further comprising the step of (c) converting kanji included in the text data into kana and generating kana converted text data composed only of characters other than kanji. The data compression method according to claim 6 or 7, further comprising: (d) performing digital compression on the kanji list data, the kanji position data, and the kana converted text data.   The step of (d), when instructed from outside, discards the kanji list data and the kanji position data, and performs digital compression only on the kana converted text data. Data compression method. (E) further comprising performing a morphological analysis on the text data and decomposing the text data into words;
The step (e) is executed before the step (a) is executed,
In the step of (a), using the result of the morphological analysis obtained in the step of (e), kanji included in the text data is converted into kana, and the kana converted text data is generated. Item 8. The data compression method according to Item 7. A program for performing data compression on text data including kanji using a computer device,
In the computer device,
(A) separating the Chinese characters included in the text data from characters other than the Chinese characters included in the text data;
(B) generating a kanji list data specifying the separated kanji and generating kanji position data specifying a position of the kanji in the text data. The program according to claim 11, further comprising: (c) causing the computer device to further execute a step of converting kanji included in the text data into kana and generating kana converted text data including only characters other than kanji. . The program according to claim 11 or 12, further causing the computer apparatus to execute a step of performing digital compression on the kanji list data, the kanji position data, and the kana converted text data.   The step of (d), when there is an instruction from the outside, discards the kanji list data and the kanji position data and performs digital compression only on the kana converted text data. Program. (E) performing a morphological analysis on the text data, and causing the computer device to further execute a step of decomposing the text data into words, prior to the step (a),
In the step of (a), using the result of the morphological analysis obtained in the step of (e), kanji included in the text data is converted into kana, and the kana converted text data is generated. Item 13. The program according to item 12.

Images