JP2007129683A - Compressed data transmission method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To register, in a slide dictionary, a long character string of high appearance frequency as an initial value. <P>SOLUTION: Character strings in sample data are sorted into groups of character strings having leading three characters ("ABC") equal to each other. A character string of highest appearance frequency (most frequently appearing character string) is extracted from the sample data one by one for each group. The most frequently appearing character string extracted from each of the groups is registered in a dictionary as an initial value in an appearance frequency order. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a data compression method using a slide dictionary, and more particularly to a data compression method and a compressed data transmission method suitable for compression of a text document, a program, and the like.

  There are lossless and lossy data compression methods. Lossless is a lossless compression method that can completely restore compressed data, and is mainly used for compression of text documents and programs. Lossy is an irreversible compression method that cannot completely restore compressed data, and is used for compression of images, sounds, moving pictures, and the like. Data compression is used for the purpose of reducing the amount of data transmitted in data communication.

  As one of the lossless compression methods, a data compression method using a slide dictionary is known. This data compression method searches for a maximum matching partial sequence from data sequences that have appeared in the past, and outputs the partial sequence position and matching length as a code. Is stored in the dictionary. Since the detection range of this dictionary changes (slides) during compression, it is generally called a slide dictionary.

In recent years, the spread of mobile terminals has been remarkable. In mobile terminal services, data with a small capacity is frequently communicated. For example, the data amount of the HTTP request (uplink data) from the mobile terminal to the server is only about 1 KB (kilobytes). In addition, in data exchanged between a mobile terminal and a server, or in wireless communication using an RFID tag, data with similar contents such as header information is exchanged many times during a series of data transmission / reception. The tendency is strong.
In a conventional data compression method using a slide dictionary, a partial sequence that appeared in the past is registered (learned) in the dictionary. Generally, about 8 KB of data is required to complete a dictionary. Therefore, if the amount of data is small, registration (learning) sufficient for compression cannot be performed, so that a sufficient compression rate cannot be obtained.

  In order to solve this problem, the applicant previously registered a frequent character as an initial value in advance in the dictionary before compression, and performed compression by matching the initial value in the dictionary with the data to be compressed. A data compression method for improving the efficiency has been proposed (Japanese Patent Laid-Open No. 5-24177). In this data compression method, since the character string registered as an initial value in the dictionary can be compressed even when it is first appeared, the compression efficiency is improved.

FIG. 22 is a diagram for explaining an initial value registration method in a dictionary disclosed in the above-mentioned Japanese Patent Laid-Open No. 5-24177.
FIG. 22A shows the type of character string existing in the sample data for initial value creation in a tree structure. The characters “a”, “b”, “c”, “d” of each node of the tree structure shown in FIG. 5A indicate characters in the sample data, and the numbers in the rectangle below the characters indicate the characters of those characters. Indicates the frequency of occurrence within each character string.

  Referring to the tree structure of FIG. 8A, when a character string having an appearance frequency of a predetermined threshold value 2 or higher is extracted, “aaa”, “abc”, “ac”, “bb”, “bb” shown in FIG. There are five “cc” and “d”. Then, these five character strings are registered as initial values in the dictionary 1001 (see (c) in the figure).

As described above, it is possible to improve the data compression efficiency by registering a character string having a high appearance frequency in the dictionary in advance based on the sample data.
Further, the present applicant has proposed a data compression method of SLC (Super Lossless Data Compression) as shown in FIG. 21 (Japanese Patent No. 3541930, US Pat. No. 6,320,522 B1).

  In the SLC method, a hash table is used as the dictionary 2001, and an arbitrary number of characters (three characters in this example) at the beginning of the already-existing character string in the compressed data 2000 is converted into a hash value by the hash function 2002, and the hash The value and the length of the existing character string (character string length) are registered in the dictionary 2001. A serial number starting from 1 is assigned to the character string of the compressed data 2000 as an appearance position. By sliding the sliding window 2005, a character string that repeatedly appears is checked, and a character string that matches the already-existing character string is encoded into a code of (appearance position, length). Here, the appearance position is the appearance position of the existing character string registered in the dictionary 2001, and is read from the dictionary 2001 using the hash value as a key.

FIG. 21 shows an example in which the compressed data 2000 is “compression & decompression...”, And the character string “compression” that appears for the second time in this character string is encoded to the code (1, 11). Has been. The hash value of the first three characters “com” of “compression” is i, and the appearance position (= 1) corresponding to the hash value i is read from the dictionary 2001.
Japanese Patent Laid-Open No. 5-24177 Japanese Patent No. 3541930 (US Pat. No. 6,320,522 B1)

  The prior art shown in FIG. 22 attempts to improve the data compression efficiency by registering a character string frequently appearing in the dictionary as an initial value before compressing the data. However, the appearance frequency is simply higher than a predetermined threshold value. Since the character string (short text) is registered in the dictionary, there is a problem that the size of the initial value becomes large.

  The prior art shown in FIG. 21 converts an arbitrary number of leading characters into a hash value in order to detect an already-existing character string registered in the dictionary at a high speed, and the hash value together with the appearance position of the already-existing character string. I am trying to register. However, the dictionary (hash table) has a configuration in which only one appearance position information can be registered for the same hash value. Some initial value character strings may accidentally take the same hash value even though they are different character strings. Therefore, when such a hash value collision occurs, the initial value registered in the dictionary is overwritten by the initial value having the same hash value that appears later, and the initial value registered earlier is not utilized. There was a problem.

  An object of the present invention is to generate an initial value for slide type dictionary registration with high compression efficiency. Another object of the present invention is to make it possible to extract initial values that do not cause hash value collision when registering initial values in a hash table used as a slide dictionary.

  Another object of the present invention is to provide a compressed data transmission method using the dictionary generated as described above.

  The compressed data transmission method of the present invention is executed by a computer for encoding data using a dictionary in which initial values generated from sample data are registered in advance, and transmitting the encoded data between a server and a client. This method is assumed.

  In the compressed data transmission method of the present invention, the character string in the sample data is classified into a group of character strings in which the first n characters or the hash value of the first n characters are the same, and the most frequent character string is extracted from each group. When the server stores a dictionary that detects and registers the most frequently used character string as the initial value, the server transmits the dictionary information to the client, and transmits and receives data between the server and the client. Is transmitted and received after the data is compressed with the dictionary information.

The dictionary information is, for example, a means (for example, a hash table) that stores an initial value to be registered in the dictionary, or position information of the initial value and the initial value in the dictionary.
In the compressed data transmission method of the present invention, for example, the server holds dictionary information for each type of service to be provided, and when providing a service to a client, the server uses a dictionary corresponding to the service to The data to be transmitted may be compressed.
In the compressed data transmission method of the present invention, for example, when the server holds a plurality of dictionary information and transmits data to the client, the dictionary information having the best compression efficiency of the data is used. The data may be compressed.

  According to the compressed data transmission method of the present invention, a relatively small amount of data can be efficiently compressed and transmitted / received between the server and the client.

  According to the present invention, after character strings appearing in sample data are classified into groups of character strings having the same first n characters, the most frequent character strings are detected from each group, and the detected most frequent character strings are dictionaryd. Therefore, an initial value with good compression efficiency can be registered within a limited initial value size. In addition, a larger number of character strings having a long matching length that increase the compression rate can be registered in the dictionary as initial values.

  In addition, after character strings appearing in the sample data are classified into groups of character strings having the same hash value of the first n characters, the most frequent character strings are detected from each group, and the detected most frequent character strings are stored in a dictionary (hash Therefore, the initial value registered in the dictionary can be prevented from being overwritten by another initial value. Also, more initial values can be efficiently registered in a limited size area of the dictionary.

As a result, by using a dictionary in which initial values are registered in advance as described above, even relatively small data to be compressed can be efficiently compressed.
In addition, transmission data can be compressed using an appropriate dictionary among the plurality of dictionaries generated as described above.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[Principle of the present invention (1)]
FIG. 1 is a diagram for explaining a method of encoding compressed data according to the present invention.

  In the present invention, a long character string frequently appearing in sample data is prepared in advance as an initial value. Then, assuming that the initial value virtually appears before the character string to be compressed (compressed data), the compressed data is compressed and decompressed with reference to the initial value.

By the way, in this specification, the slide dictionary may be simply referred to as a dictionary for simplification.
In the example shown in FIG. 1, it is assumed that the character string (initial value) “compression” appears virtually before the character string “compression & decomptession ...” to be compressed. The leading position of the compressed data is set to “1”. FIG. 1 shows a learned hash table 14.

  In the hash table 14, the appearance position of the initial n characters of the initial value (the first 3 characters in the example of FIG. 1) is registered, as in the SLC method described above. In the figure, for convenience of explanation, a hash value is described in association with data in the hash table 14, but the hash value is data indicating the line to be accessed in the hash table. There is no need to register a hash value.

  In the example shown in FIG. 1, the initial initial value is “compression”. Since the hash value of the first three characters “com” of “compression” is “i”, the start position of “compression” is registered in the “i + 1” line of the hash table 14. In this registration, “−32” which is the appearance position (position of the first character) of “compression” is registered in the “i + 1” line of the hash table 14.

  In the example shown in FIG. 1, the first 11 characters of the compression target character string “compression & decomptession...” Are the same character string as the initial value “compression”. Therefore, in the compression target character string, the first 11 characters “compression” are immediately encoded into a code of (−32, 11).

  As described above, since compression using a slide dictionary can encode a long repetition with a single code, a character string having a high appearance frequency (frequent character string) is registered in advance in the slide dictionary as an initial value. The possibility of encoding the data to be compressed at a high compression rate is increased.

  A slide dictionary using a hash table includes a memory (hereinafter referred to as a dictionary buffer for convenience) for registering a new character string that appears when an initial value and data to be compressed are encoded, and a character registered in the memory. It consists of a hash table that holds the start position of the column. Learning of the slide dictionary using the hash table is performed by two processes: registration of a character string in the dictionary buffer and registration of a start position of the character string registered in the dictionary buffer in the hash table.

  When the initial value is registered in the slide dictionary, as schematically shown in FIG. 2, a character string (long frequent character string) with good compression efficiency is initialized from among a plurality of sample data 21-1 to 21-n. It is necessary to generate them as values and register these character strings in the dictionary 24 in order of appearance frequency.

  In the slide dictionary, normally, only the appearance position of the first character string is held in the hash table. This is to save memory usage. Also in the present invention, only the appearance position of the first n characters (n is a positive integer) of the initial value is held in the hash table of the slide dictionary.

Next, the procedure of the initial value generation / registration method in the present invention will be described.
1. A plurality of sample data is prepared, and character strings appearing in the sample data are classified into groups of character strings having the same first n characters.
2. The most frequently occurring character string (character string having the highest appearance frequency) is detected from each group classified in the procedure 1.
3. The most frequently appearing character strings of each group obtained in the procedure 2 are registered as initial values in the slide dictionary in the order of frequency.

  According to the above steps 1 to 3, the initial value character string in which the most frequently appearing character strings in the character string having the same first n characters are arranged in a line in order of frequency is registered in the slide dictionary. If the initial size of the slide dictionary is limited in steps 1 to 3, the most frequently used character string is continuously registered in the slide dictionary until the upper limit of the size is reached.

  By the way, in this specification, before starting compression of data to be compressed using a slide dictionary, a series of one or more most frequently appearing character strings registered in advance in the slide dictionary is referred to as “initial value characters” for convenience. In some cases, “initial value” is described instead of “column”. Therefore, the initial value may refer to an initial value character string.

FIG. 3 shows an example in which the initial value generation method of the present invention is applied to character strings having the same first three characters.
In the figure, the classification of character strings created by the procedure 1 and having the first three characters “ABC” is shown in a tree structure. In this example, as a character string whose first three characters are “ABC”, a character string whose subsequent character strings following “ABC” are “BBBC”, “BCDC”,..., “CDEF” is shown. . In the tree structure, the number below the character arranged at each node is the appearance frequency of the character. In this example, the character string in which the subsequent character string of “ABC” is “BCDC”, that is, the character string “ABCBCDC” appears eight times, and has the highest appearance frequency. Therefore, the character string “ABCBCDC” is detected as the longest character string among the most frequent character strings from the character string group whose first three characters are “ABC”. Similarly, the longest character string among the most frequently appearing character strings can be detected for character strings whose leading three characters are not “ABC”.

As described above, in the present invention, the longest character string is extracted from the character strings that start with the same first character string (number of characters n), and the extracted most frequent character strings are stored in the dictionary in the order of appearance frequency. Pre-register as initial value.
[Principle of the present invention (2)]
When using a hash table as a dictionary, as shown in FIG. 4, in the process of sequentially learning the position information of the hash value, the character string in which the position information is already registered in the hash table is already included in the character string. If it exists, that is, if there is a different character string having the same hash value by chance, the existing position information is overwritten by the position information of the newly appearing character string. In the example of FIG. 4, the position information of “compression” is overwritten from “1” to “k”.

  Therefore, in the present invention, when a character string corresponding to a certain hash value is registered in the stage of registering the initial value in the hash table, the character string corresponding to the hash value is never registered in the hash table.

  In the hash table used in the present invention, the hash value is used as a value indicating a row in the hash table. That is, in the hash table, one piece of position information is registered for one hash value. Since it is configured, it is necessary to detect the most frequent character string for each hash value.

  The hash value is calculated based on the first n characters registered as the initial value. For this reason, in the present invention, the most frequent character string is detected from character strings having the same first n characters, and the most frequent character string is set as an initial value.

FIG. 5 is a diagram showing a method of generating an initial value (most frequently appearing character string) from character strings having the same first three characters by the above method.
In the tree structure shown in the figure, the hash values of the first three characters “ABC” and “XYZ” have the same value (= 100). For this reason, the most frequently occurring character string is detected from the character strings having the first three characters “ABC” or “XYZ”, and registered as an initial value in the hash table. In the example shown in FIG. 5, “XYZCDEF” appears eight times, which is the most frequent character string and the longest character string.

Similarly, for the character string having the first three characters whose hash value is other than 100, the most frequent character string and the longest character string are detected and registered as an initial value in the hash value.
By sequentially registering the initial values in the hash table as described above, overwriting of the position information of the initial values registered in the hash table is prevented. This is because only one piece of position information is registered for one hash value in the hash table.

When the same method is used to register the first n characters of the initial value in the hash table, it is possible to prevent overwriting of the initial position information.
[First embodiment]
The first embodiment of the present invention is a system including an information processing device for creating an initial value, an information processing device on the data compression side, and an information processing device on the data restoration side.

FIG. 6 is a block diagram showing a system configuration of an information processing apparatus that creates an initial value by the data compression method of the present invention.
The information processing apparatus 100 for creating an initial value includes a compression program 110, sample data 120, and an initial value creating program 130. These are stored in a memory or an external storage device (storage) included in the information processing apparatus 100.

  The compression program 110 is a program for compressing and restoring data, and includes a hash table 111, an initial value (initial value character string) 112, and a hash function 113 therein. The hash table 111 is a dictionary that stores the position information at the position in the table represented by the hash value of the first n characters of the character string registered as the initial value. The data structure of the hash table 111 is the same as that of the hash table 14 of FIG. The initial value 112 is a character string used for data compression / decompression, and is created by the initial value creation program 130. The hash function 113 calculates a hash value corresponding to the first three characters of the initial value 112.

  The compression program 110 inputs and holds the initial value 112 from the initial value creation program 130. Then, the hash value of the first n characters of the initial value 112 is calculated by the hash function 113, and the position information of the initial value 112 is registered in the same row as the hash value in the hash table 111. Also, the position information of the initial value 112 having the hash value registered in the hash table 111 is registered in the row in which the hash value is stored.

  Sample data 120 is No. 1-No. 4 sample data. No. The sample data of 1 (hereinafter referred to as sample data 1 for convenience) is a character string “COMPRESS_”. No. The sample data 2 (hereinafter referred to as sample data 2 for convenience) is a character string “DECOMPRESS_”. No. Sample data 3 (hereinafter referred to as sample data 3 for convenience) is a character string “COMPRESSION_”. No. The sample data 4 (hereinafter referred to as sample data 4 for convenience) is a character string “COMPARE”.

The initial value creation program 130 includes a frequency counter 131, a succeeding most frequent character holding unit 132, and a registration check unit 133.
The frequency counter 131 classifies all character strings appearing in the sample data 1 to 4 into character strings having the same first three characters, and detects the most frequently occurring character string from character strings having the same first three characters. Used for. The frequency counter 131 includes an area for storing the first three characters and the frequency, and a character string having the same first three characters is connected to the area by a pointer. The above configuration is provided for each of the first three characters. The elements of the list are composed of a sample data No, a character string start position, and an end position.

  The succeeding most frequent character holding unit 132 holds a succeeding character following the first three characters in the process of the initial value creating program 130 creating the initial value 112. The succeeding most frequently appearing character holding unit 132 finally stores the end character of the most frequently appearing character string.

  The registration check unit 133 holds status information indicating whether or not the most frequently appearing character string held in the list of the frequency counter 131 is registered as an initial value in the hash table. The status information is “TRUE” when registered as an initial value, and “FALSE” when not registered. The status information is initially set to “FALSE” before the start of the initial value creation process.

FIG. 7 is a block diagram showing the system configuration of the information processing apparatus on the data compression side.
The information compression apparatus 140 on the data compression side holds the compression program 150 in a memory or an external storage device. The compression program 150 is a program that compresses and restores data, and stores therein a hash table 151, an initial value (initial value character string) 152, and a hash function 153. The dictionary 151, the initial value 152, and the hash function 153 have the same configuration and function as the hash table 111, the initial value 112, and the hash function 113 that the compression program 110 of the information processing apparatus 130 in FIG. Detailed description is omitted.

FIG. 8 is a block diagram showing the system configuration of the information processing apparatus on the data restoration side.
The information restoration device 160 on the data restoration side holds the compression program 170 in a memory or an external storage device. The compression program 170 is a program for compressing and restoring data, and holds a dictionary (hash table) 171, an initial value (initial value character string) 172, and a hash function 173 therein. The dictionary 171, the initial value 172, and the hash function 173 have the same configuration and function as the hash table 111, the initial value 112, and the hash function 113 held by the compression program 110 of the information processing apparatus 130 in FIG. 6. Detailed description is omitted.

The three information processing apparatuses 100, 140, and 160 are connected via a network and can transmit and receive data to and from each other.
The outline of the processing of the first embodiment will be described with reference to FIG. {Pre-processing}
Before compression and decompression of data, generation of initial values and learning of a dictionary (hash table) are performed as pre-processing. Dictionary learning is a process of registering an initial value and its position information in the dictionary.

  Thereafter, the initial value and the dictionary are distributed to the information processing apparatus 140 on the compression side and the information processing apparatus 160 on the restoration side via the network. The information processing device 140 and the information processing device 160 store the initial value and the dictionary distributed from the information processing device 100 in an external storage device.

With reference to FIG. 9A, the procedure of the pre-processing performed by the information processing apparatus 100 will be described.
(1) An initial value is generated from sample data (step S1).
(2) The dictionary is made to learn the position information of the generated initial value, and the dictionary is completed (step S2).
{Compression processing}
The procedure of the data compression process will be described with reference to FIG. This data compression process is performed by the information processing apparatus 140 executing the compression program 150.
(1) The initial value and dictionary stored in the external storage device are loaded onto the memory (step S3).
(2) The compressed data is generated by compressing the data to be compressed based on the initial value and the dictionary (step S4).

The information processing device 140 transmits the compressed data to the information processing device 160.
{Restore processing}
With reference to FIG. 9B, the decompression process of the compressed data will be described. This data restoration process is performed by the information processing apparatus 160 executing the compression program 170.
(1) The initial value and dictionary stored in the external storage device are loaded onto the memory (step S5).
(2) The compressed data is restored from the compressed data based on the initial value and the dictionary (step S6).
{Details of initial value generation processing and dictionary generation processing}
FIG. 10 is a flowchart for explaining the details of the initial value generation processing and dictionary generation processing (dictionary learning processing) performed by the information processing apparatus 100 executing the initial value creation program 130. 11 to 14 are diagrams for facilitating understanding of the processing shown in the flowchart of FIG. 10.

  Details of the initial value generation process performed by the information processing apparatus 100 will be described with reference to FIGS. In this example, character strings having the same first three characters are grouped and processed.

The information processing apparatus 100 extracts all three character strings existing in the sample data 120 (sample data 1 to 4) (step S11).
As a result, as shown in FIG. 11, character strings of “COM”, “OMP”, “MPR”,..., “ARE” are extracted.

  Next, the character string extracted in step S11 (hereinafter referred to as “extracted character string” for the sake of convenience) is all stored in the frequency counter 131 with “character string”, “frequency”, “start position”, and “end position”. Store (step S12).

  As a result, the contents of the frequency counter 131 are as shown in FIG. 12, and “sample data No.”, “start position”, and “end position” are stored in each element of the list linked to “frequency”. The “COM” has a “frequency” of 4, and the character string “COM” appears in all of the sample data 1 to 4.

Subsequently, the processing of steps S13 to S18 is executed for all the extracted character strings while sequentially selecting the extracted character strings one by one.
First, “COM” is selected.
In step S <b> 13, the most frequently appearing character is detected from the following characters (the character next to the end position), and the most frequently occurring character is stored in the succeeding most frequently occurring character holding unit 132.

As a result, in the case of “COM”, “P” is stored in the succeeding most frequent character holding unit 132 as the most frequent character (see FIG. 13A).
Next, the number of appearances of the most frequently detected character detected in step S13 (the number of appearances in the sample data 1 to 4) is added to the “frequency” of the frequency counter 131 (step S14).

As a result, in the case of “COM”, since “P” is a subsequent character in all the sample data 1 to 4, “4” is added to “frequency”, and “frequency” is 8 (FIG. 13). (See (b)).
Next, it is determined whether or not the character (following character) at the position next to the “end position” stored in the frequency counter 131 is the most frequently appearing character (step S15). If it is the most frequently appearing character, the process proceeds to step S16, and if not, the process proceeds to step S17.

In step S16, the appearance position of the character determined as the most frequently appearing character in step S15 is stored (set) in the frequency counter 131 as a new end position.
As a result, in the case of “COM”, the “end position” of the four character strings held in the frequency counter 131 moves backward by one character (see FIG. 13C).

  In step S17, elements whose subsequent characters are not the most frequent characters are deleted from the list of the frequency counter 131. This process is a process of deleting a character string that is not a candidate for the most frequently appearing character string from the frequency counter 131.

  When a plurality of character strings having the same first three characters are stored in the frequency counter 131, either the process of step S16 or step S17 is performed for each character string depending on the determination result in step S15. .

Subsequent to the processing in step S16 or step S17, it is determined whether all subsequent characters of the character string held in the frequency counter 131 are different (step S18).
In this determination, if it is determined that all the following characters are different, the process proceeds to step S19, and if not, the process returns to step S13 to search for the subsequent most frequent character.

As described above, the processes in steps S13 to S18 are repeated until it is determined in step S18 that all subsequent characters in the character string having the same first three characters are different.
In the case of “COM”, only the subsequent character of the sample data 4 is “A” and the subsequent characters of the sample data 1 to 3 are all “R” (the most frequently appearing character) in the determination processing in the second step S15. Is determined. As a result, in step S17, the pointer corresponding to the character string “COMPA” of the sample data 4 is deleted from the frequency counter 131, and “R” is stored in the succeeding most frequent character holding unit 132 of the frequency counter 131 in step S16. (See FIG. 13D).

As described above, with regard to “COM”, the processes of steps S13 to S18 are repeated six times, and it is determined in step S19 that there is no subsequent character, and the process proceeds to step S19.
Immediately before proceeding to step S19, the contents of the frequency counter 131 and succeeding most frequent character holding unit 132 relating to “COM” are as shown in FIG. That is, “_” is stored in the succeeding most frequent character holding unit 132, and the start position and end position regarding the character string “COMPRESS_” in the sample data 1 and the sample data 2 are stored in the frequency counter 131. 22 is stored.

As described above, when the processing of steps S13 to S19 related to “COM” is completed,
The next three characters “OMP” held in the frequency counter 131 are processed in steps S13 to S19 in the same manner as “COM”.

In this way, the processes of steps S13 to S19 are repeated until the process of the last three characters “ARE” held in the frequency counter 131 is completed.
When the processing of steps S13 to S19 is completed for “ARE” and the processing of steps S13 to S19 is completed for all the first three characters extracted in step S11, the process proceeds to step S20.

  FIG. 14 shows the frequency counter 131 and the succeeding most frequent character at the time when the processing of steps S13 to S19 is completed for the processing of steps S13 to S19 for all types of character strings with different leading three characters extracted in step S11. It is a figure which shows the content of the holding | maintenance part 132. FIG.

  As shown in the figure, the frequency counter 131 contains the most frequently used character string in the sample data 120 of the character string having the first three characters “COM”, “OMP”, “MPR”,..., “APR”. The “start position” and “end position” are stored. The “frequency” of the frequency counter 131 stores the total number of appearances (frequency) of the most frequently appearing characters at each subsequent position following the first three characters in the sample data 120.

By referring to the set (“sample data No.”, “start position” and “end position”) held in the frequency counter 131, the sample is obtained as the most frequently appearing character string with “COM” as the first three characters. “COMPRESS” present in data 1 and sample data 2 In addition, “OMPRESS” existing in the sample data 1 and the sample data 2 as the most frequent character string having “OMP” as the first three characters. Similarly, the most frequent character string can be acquired for a character string having “MPR”,..., “ARE” as the first three characters.

  In step S20, the most frequent character strings of the first three characters acquired in step S19 are registered as initial values in order of frequency. That is, the “frequency” of each of the first three characters held in the frequency counter 131 is referred to and registered as an initial value in order from the most frequent character string having the highest value. The registration check unit 133 corresponding to each of the three most frequently appearing character strings registered as the initial value is set to “TRUE”.

  As a result, the contents of the frequency counter 131, the succeeding most frequent character holding unit 132, and the registration check unit 133 of the initial value creation program 130 are as shown in FIG. As shown in FIG. 15, in the case of the first embodiment, the most frequently used character strings are registered as initial values for the first three characters of all types appearing in the sample data 120.

The initial value registered as described above is output to the compression program 150. The compression program 150 holds the initial value received from the initial value creation program 130 as the initial value 112 and creates the hash table 111 based on the initial value 112.
{Modification of initial value creation program}
FIG. 16 is a block diagram showing a configuration of a modification of the initial value creation program.

  Based on the principle shown in FIG. 5, the initial value creation program 130 shown in FIG. 5 classifies the first three characters into groups of character strings having the same hash value, and sets the most frequent character strings in each group as initial values. Acquire and register as.

The initial value creation program 140 includes a frequency counter 141, a succeeding most frequent hash value holding unit 142, and a registration check unit 143.
The frequency counter 141 includes a list of elements including a set of “hash value”, “frequency”, and {“sample data No.”, “start position”, “end position”}. This list has the same configuration as the list of the initial value creation program 130.

  The subsequent most frequent hash value holding unit 142 has the same function as the subsequent most frequent character holding unit 132 of the initial value creating program 130, and stores the hash value of the subsequent characters following the first three characters. In the subsequent most frequent hash value holding unit 142, the hash value of the end character of the most frequent character string in each hash value group is finally stored.

The registration check unit 143 has the same configuration and function as the registration check unit 133 of the initial value creation program 130.
The process of registering the initial value by the initial value creating program 140 can be realized by an algorithm substantially similar to the flowchart of FIG. That is, the algorithm of the initial value registration process of the initial value creating program 140 can be realized by changing the flowchart of FIG. 10 so that “hash value” is processed instead of “character”.
[Second Embodiment]
FIG. 17 is a system configuration diagram of the second embodiment to which the data compression method of the present invention is applied.

  The computer system shown in the figure includes an AP server 200 (AP server 1), an AP server 210 (AP server 2), a front-end server 300, and a client 330. The front-end server 300 and the client computer 330 are connected to a network 321. Connected by. Although not shown in FIG. 17, the present system includes n AP servers that are server machines that provide application services such as a WEB service and a mail service. In the following description, the AP servers 200 and 210 will be mainly described when describing the AP server.

  The AP server 200, the AP server 210, and the front-end server 300 are connected by a communication line such as a LAN (Local Area Network), and constitute, for example, a data center of a service provider. The AP server 200 and the AP server 210 are server machines that perform processing related to the provided services, and the front-end server 300 is a server machine that mediates communication between the AP server and the client 330.

  The address “1.1.1.1” is assigned to the AP server 200, and the address “1.2.3.4” is assigned to the AP server 210. These addresses are used by the front-end server 300 and the client 330 as server IDs (server identifiers) of the AP servers 200 and 210. As the server ID, for example, an IP (Internet Protocol) address, a URL (Uniform Resource Locator), a domain name, or the like is used.

  The AP server 200 is an application server that transmits XML (Extensible Markup Language) format data 251 to the client 330 via the front-end server 300. The AP server 210 is an application server that transmits data 252 in HTML (Hyper Text Markup Language) format to the client 330 via the front-end server 300.

  The AP server 200 executes the initial value creation program 220 at any timing before sending the XML data 251 to the client 330, and the hash table 231 and the initial value (initial value) based on the XML sample data 201 that is sample data. Value string) 232 is created. Then, the hash table 231 and the initial value 232 are transmitted to the client 330 via the front end server 300.

  The AP server 210 executes the initial value creation program 220 at any timing before transmitting the HTML data 252 to the client 330 and the hash table 241 and the initial value (initial value characters) based on the HTML data 211 as sample data. Column) 242 is created. Then, the hash table 241 and the initial value 242 are transmitted to the client 330 via the front end server 300.

  The initial value creation program 220 is substantially the same program as the initial value creation program 130 of the first embodiment, is stored in a storage (not shown), and is shared by the AP server 200 and the AP server 210.

  The front-end server 300 is a server that mediates communication between the client 330 and the AP servers 200 and 210, compresses XML data or HTML data received from the AP servers 200 and 210, and transmits the compressed data to the client 330. The front-end server 300 is, for example, a Web server, and communicates with the client 330 using a protocol such as HTTP (Hyper Text Transfer Protocol).

  The front-end server 300 has a compression program 310 and executes the compression program 310 to compress the XML data 251 and the HTML data 252 received from the AP servers 200 and 210.

  The compression program 310 includes a hash table 311, an initial value (initial value character string) 312, and a management table 313. The hash table 311 and the initial value 312 are the hash table 231 and initial value 232 received from the AP server 200 or the hash table 241 and initial value 242 received from the AP server 210. Each time the front-end server 300 receives the hash table and initial value from the AP servers 200 and 210, the front-end server 300 rewrites the hash table 311 and the initial value 312 with the received hash table and initial value.

  The registration table 313 is a table in which a hash table and initial value management information received by the front-end server 300 from the AP server are registered. The registration table 313 is a table composed of n rows, and each row stores three items of “server ID”, “hash table name”, and “initial value name”. The server ID is the address of the AP server. The hash table name is the name of each hash table. The initial value name is the name of the initial value. With this configuration, the registration table 313 manages a hash table and initial values used to compress XML data or HTML data received from each AP server.

  The client 330 is any computer such as a portable terminal, a mobile phone, a personal computer, an RFID tag, or an in-vehicle terminal as long as it is a computer that is communicably connected to the front-end server 300 via the network 321. It doesn't matter. The client 330 has a compression program 340, and executes the compression program 340 to restore the compressed XML data 251 and HTML data 252 received from the front-end server.

  The compression program 340 has the same configuration as the compression program 310 held by the front-end server 300 described above, and includes a hash table 341, an initial value 342, and a registration table 343.

Since the configuration of the registration table 343 is the same as the registration table 313 held by the front-end server 300, detailed description thereof is omitted.
When the client 330 receives the XML data 251 or the HTML data 252 from the front end server 300, the client 330 receives the initial value identification information 320 from the front end server 300.

  The initial value identification information 320 is used to obtain an initial value and a hash table used when restoring data (XML data 251 or HTML data 252) received from the front-end server 300 with reference to the registration table 343. This is necessary information. The initial value identification information 320 includes, for example, a server ID, an initial value name or a hash table name, or a combination thereof.

Next, the operation of the second embodiment having the above configuration will be described.
FIG. 18 is a flowchart for explaining processing in which the AP server (AP servers 200 and 210) generates an initial value and a hash table.

  The AP server executes the initial value creation program 220 and generates an initial value (initial value 232 or initial value 242) based on the XML sample data 201 or the HTML sample data 211 (step S101). Subsequently, the hash value and the appearance position of the generated initial value are registered in the hash table, and a hash value is generated (step S102).

FIG. 19 is a flowchart for describing processing related to the registration table of the front-end server 300.
The front-end server 300 registers the received hash table name (hash table name) and initial value name (initial value name) in the registration table 313 (step S201).

Next, the hash table and the initial value are transmitted to the client 330 via the network 321 (step S202).
Subsequently, the hash table and information on the initial value (server ID, hash table name, initial value name) are registered in the registration table 313 (step S203).

  Through the above processing, the hash table and initial value received from the AP server are transmitted to the client 330, and information regarding the received hash table and initial value is registered in the registration table 313.

FIG. 20 is a flowchart for explaining data compression / decompression processing performed between the front-end server 300 and the client 330.
In the flowchart of FIG. 5, steps S301 to S303 are processing of the front-end server 300, and steps S304 and 395 are processing of the client 330.

  The front-end server 300 selects an initial value and a hash table to be used for compressing the data (XML data or HTML data) based on the server ID of the registration table 313 according to the AP server that is the transmission source of the data (XML data or HTML data) ( Step S301).

Next, the data received from the AP server is compressed based on the selected initial value and hash table (step S302).
Then, the initial value identification information 320 corresponding to the compressed data and the initial value used in the data compression is transmitted to the client 330 via the network 321 (step S303).

  The client 330 searches the registration table 343 based on the initial value identification information 320 received from the front end server 300, and selects an initial value and a hash table necessary for decompressing the compressed data received from the front end server 300. (Step S304).

Then, the compressed data is restored based on the selected initial value and the hash table (step S305).
As described above, in the second embodiment, on the AP server side, an initial value and a dictionary (hash table) for compressing / decompressing data to be transmitted to the client 330 by an application (service) to be provided are created, and these are stored. Transmit to the front-end server 300. The front-end server 300 manages the initial value and dictionary received from the AP server by the registration table 313 and transmits them to the client 330 via the network 321 together with the initial value identification information 320.

  The client 330 manages the initial value and dictionary received from the front-end server 300 using the registration table 343. When the compressed data and the initial value identification information 320 are received from the front-end server 300, the initial value and hash table necessary for restoring the compressed data are referred to by referring to the registration table 343 based on the initial value identification information 320. And decompress the compressed data using them.

  By the way, it is possible to restore the compressed data only from the initial value. This is because a dictionary (hash table) can be generated based on the initial values. Therefore, even in a system in which the front-end server 300 and the client 330 receive only the initial value from the AP server, data compression / decompression is possible using the data compression method of the present invention.

  In the present invention, a server or client generates and holds a plurality of dictionaries (initial values or initial values and hash tables) in advance for each service, for example, and is transmitted and received between the server and the client regarding the service. For a series of data groups, which dictionary is to be used according to the service (application) requested by the client or the service that the server intends to provide, or the side that transmits data first for the service The computer (server or client) may be determined. Information or dictionary information for the server receiving the service provision request from the client to determine a dictionary to be applied to a series of data transmission / reception of the service provision, and to identify the data and dictionary related to the service when replying to the client In the transmission / reception of data in the service, data compression and decompression may be performed using the determined dictionary.

The plurality of dictionaries held by the server or the client may be generated according to the terminal type of the client.
In this case, for example, either the server or the client actually compresses the compressed data that is actually transmitted / received data with the above-mentioned plurality of dictionaries, and the dictionary with the highest compression efficiency is adopted as the compression dictionary. The information about the adopted dictionary may be notified to the communication partner.

Further, when the client transmits data to the server, the data may be compressed using a dictionary holding the data and then transmitted to the server.
In the description of the above embodiment, an example in which text data is compressed as data to be compressed has been described. However, the present invention is not limited to this. For example, compression of a source program or an executable program (binary data) is performed. It is also applicable to.
(Appendix 1)
In a data compression method for generating an initial value from sample data, pre-registering the initial value in a dictionary, and then encoding the data using the dictionary,
Classify the strings in the sample data into groups of strings with the same first n characters,
Detecting the most frequently occurring character string from each group, and registering the most frequently occurring character string in the dictionary as the initial value;
A data compression method.
(Appendix 2)
In a data compression method for generating an initial value from sample data, pre-registering the initial value in a dictionary, and then encoding the data using the dictionary,
Classify the strings in the sample data into groups of strings with the same hash value for the first n characters,
Detecting the most frequently occurring character string from each group, and registering the most frequently occurring character string in the dictionary;
A data compression method.
(Appendix 3)
A data compression method according to appendix 1 or 2,
When detecting the most frequently occurring character string, obtain the appearance frequency of each character following the first n characters,
A data compression method characterized by determining a most frequently appearing character string on the basis of their appearance frequency.
(Appendix 4)
A data compression method according to attachment 3, wherein
A data compression method, wherein a character string having the largest sum of appearance frequencies of characters following the first n characters is determined as a most frequently appearing character string.
(Appendix 5)
In a computer-implemented method for encoding data using a dictionary in which initial values generated from sample data are registered in advance, and transmitting the encoded data between a server and a client,
Classifying the character strings in the sample data into groups of character strings in which the first n characters or the hash values of the first n characters are the same;
The server detects a most frequently occurring character string from each group, and the server holds a dictionary in which the most frequently occurring character string is registered as the initial value.
The server sends the dictionary information to the client;
When sending and receiving data between the server and the client, the data is sent and received after being compressed with the dictionary information.
And a compressed data transmission method. (1)
(Appendix 6)
A compressed data transmission method according to appendix 5,
The compressed data transmission method, wherein the server holds the dictionary information for each service to be provided. (2)
(Appendix 7)
A compressed data transmission method according to appendix 5,
The server distributes the dictionary information to the client before exchanging data with the client. (3)
(Appendix 8)
A compressed data transmission method according to appendix 5,
The server holds the dictionary information for each type of service to be provided, and when providing a service to a client, the server uses the dictionary information corresponding to the service to compress data to be transmitted to the client. A compressed data transmission method characterized by the above.
(Appendix 9)
A compressed data transmission method according to appendix 5,
The client holds the dictionary information for each type of service provided from the server, and compresses the data using the dictionary information corresponding to the service when transmitting data to the server. Compressed data transmission method.
(Appendix 10)
A compressed data transmission method according to appendix 5,
The server holds a plurality of the dictionary information generated by the data compression method according to appendix 1 or 2, and uses the dictionary information with the best compression efficiency when transmitting data to the client. A compressed data transmission method comprising compressing the data.
(Appendix 11)
A compressed data transmission method according to appendix 5,
The client holds a plurality of the dictionary information generated by the data compression method according to appendix 1 or 2, and when transmitting data to the server, the client uses the dictionary having the best compression efficiency of the data. A compressed data transmission method characterized by compressing.

(Appendix 12)
A program that generates an initial value from sample data, registers the initial value in a dictionary in advance, and then causes a computer to execute a data compression process that encodes the data using the dictionary,
Classifying the character strings in the sample data into groups of character strings having the same first n characters;
Detecting the most frequently occurring character string from each of the groups, and registering the most frequently occurring character string in the dictionary as the initial value;
A program for causing a computer to execute a process comprising:
(Appendix 13)
A program that generates an initial value from sample data, registers the initial value in a dictionary in advance, and then causes a computer to execute a data compression process that encodes the data using the dictionary,
Classifying the character strings in the sample data into groups of character strings having the same hash value of the first n characters;
Detecting the most frequently occurring character string from each group, and registering the most frequently occurring character string in the dictionary;
A program for causing a computer to execute a process comprising:
(Appendix 14)
The program according to appendix 12 or 13, wherein
When detecting the most frequently occurring character string, obtain the appearance frequency of each character following the first n characters,
A program that causes a computer to execute a process including a step of determining a most frequently appearing character string based on their appearance frequency.
(Appendix 15)
In a data compression device that generates an initial value from sample data, registers the initial value in a dictionary in advance, and encodes the data using the dictionary.
Means for classifying the character strings in the sample data into groups of character strings having the same first n characters;
Means for detecting a most frequently occurring character string from each group and registering the most frequently occurring character string in the dictionary as the initial value;
A data compression apparatus comprising:
(Appendix 16)
In a data compression device that generates an initial value from sample data, registers the initial value in a dictionary in advance, and encodes the data using the dictionary.
Means for classifying the character strings in the sample data into groups of character strings having the same hash value of the first n characters;
Means for detecting the most frequently occurring character string from each of the groups and registering the most frequently occurring character string in the dictionary;
A data compression apparatus comprising:
(Appendix 17)
The data compression device according to appendix 15 or 16,
The registration means includes means for obtaining an appearance frequency of each character subsequent to the first n characters when detecting the most frequent character string, and determining the appearance frequency as a most frequent character string based on the appearance frequency. A data compression device.
(Appendix 18)
The data compression method according to appendix 17, wherein
The data compression apparatus according to claim 1, wherein the registration means includes means for determining a character string having the largest sum of appearance frequencies of the characters following the first n characters as a most frequently appearing character string.

  The present invention relates to data transmitted / received in a system that communicates a relatively small amount of data, such as communication of a mobile terminal, communication of RFID (Radio Frequency Identification), communication between vehicles in ITS (Intelligent Transportation Systems), etc. Suitable for compression.

It is a figure explaining the encoding method of the to-be-compressed data of this invention. It is a figure which shows notionally the initial value generation method of this invention. It is a figure which shows the example which applied the initial value production | generation method of this invention to character strings with the same top three characters. It is a figure explaining the overwrite of the positional information on the initial value which generate | occur | produces when using a hash table as a dictionary. It is a figure explaining the production | generation method of the initial value which does not overwrite the said positional information in a hash table. It is a block diagram which shows the system configuration | structure of the information processing apparatus for initial value preparation in 1st Example. It is a block diagram which shows the system configuration | structure of the information processing apparatus by the side of the data compression in 1st Example. It is a block diagram which shows the system configuration | structure of the information processing apparatus by the side of data restoration in 1st Example. It is a figure explaining the flow of the whole process in 1st Example. It is a flowchart which shows the initial value creation and registration process in 1st Example. FIG. 11 is a diagram (part 1) for explaining the processing of the flowchart of FIG. 10; FIG. 11 is a diagram (part 2) for explaining the processing of the flowchart of FIG. 10; FIG. 11 is a third diagram illustrating the process of the flowchart in FIG. FIG. 11 is a diagram (No. 4) for explaining the process of the flowchart in FIG. 10; FIG. 11 is a diagram (No. 5) for explaining the processing of the flowchart in FIG. 10; It is a figure which shows the structural example of the frequency counter which manages the top 3 characters with a hash value. It is a block diagram which shows the system configuration | structure of 2nd Example of this invention. It is a flowchart explaining the initial value production | generation process and hash table production | generation process by AP server. It is a flowchart explaining the process of a front end server. It is a flowchart explaining the decompression | restoration process of the compressed data performed between a front end server and a client. It is a figure explaining the data compression method using a slide dictionary. It is a figure explaining the conventional dictionary creation system.

Explanation of symbols

14 Hash table (learned)
21-1 to 21-4 Sample data 24 Initial value 100 Information processing apparatus 110 for creating initial value Compression program 111 Hash table 112 Initial value (initial value character string)
113 Hash function 120 Sample data 130 Initial value creation program 131 Frequency counter 132 Successive most frequent hash value holding unit 133 Registration check unit 140 Information processing device 150 on the compression side Compression program 151 Hash table 152 Initial value (initial value character string)
153 Hash function 160 Restoration program 170 Compression program 171 Hash table 172 Initial value (initial value character string)
173 Hash function 200 AP server 1
201 Sample data for XML 210 AP server 2
211 Sample data 220 for HTML 220 Initial value creation program 231, 241 Hash table 251 XML data 252 HTML data 300 Front-end server 310 Compression program 311 Hash table 312 Initial value (initial value character string)
313 Registration table 321 Network 330 Client 340 Compression program 341 Hash table 342 Initial value (initial value character string)
343 Registration Table

Claims (3)

In a computer-implemented method for encoding data using a dictionary in which initial values generated from sample data are registered in advance, and transmitting the encoded data between a server and a client,
Classifying the character strings in the sample data into groups of character strings in which the first n characters or the hash values of the first n characters are the same;
The server detects a most frequently occurring character string from each group, and the server holds a dictionary in which the most frequently occurring character string is registered as the initial value.
The server sends dictionary information that is information about the dictionary to the client,
When sending and receiving data between the server and the client, the data is sent and received after being compressed with the dictionary information.
And a compressed data transmission method. The compressed data transmission method according to claim 1,
The server holds the dictionary information for each type of service to be provided, and when providing a service to a client, the server uses the dictionary information corresponding to the service to compress data to be transmitted to the client. A compressed data transmission method characterized by the above. The compressed data transmission method according to claim 1,
The server has a plurality of the dictionary information, and when the data is transmitted to the client, the server compresses the data using the dictionary information having the best compression efficiency of the data. Method.

Images