JP2007537642A - Method and apparatus for compression and decompression of structured block unit of XML data - Google Patents

Abstract

  The present invention provides a method and apparatus for XML compression and decompression. An XML document is assumed to consist of a single metadata template and similar data blocks that can be decomposed into contained content data. The template is analyzed, and LZW-based data compression (ZLIB) of the content data is performed using the reference of the position in the template. The template itself is compressed separately. The content data for the data block is compressed in block units (referred to herein as “sequentially”) so that selective decompression of the data block can be performed without decompressing the entire file.

Description

  The present invention relates to a method and apparatus for compressing / decompressing data, and more particularly to a method and apparatus for sequentially compressing / decompressing data.

  Currently, sequential compression / decompression of data is the usual method of data compression / decompression. This method utilizes data already processed during the compression process as a reference for compressing / decompressing subsequent data, thereby reducing redundancy. Typical sequential compression schemes are LZ77 and Zlib. Zlib is a sequential compression scheme based on Huffman code and LZ77.

  FIG. 1 is a schematic structural block diagram showing an apparatus for sequentially compressing / decompressing data according to the prior art. As shown in FIG. 1, the apparatus stores a serial encoder 120 that is used to sequentially encode and compress the entire N data segments as a whole, and the compressed data. And a sequential decoder 140 which is used to sequentially decode the compressed data and separate the N data segments from each other using data properties or index tables. Such a compression method can achieve maximum compression efficiency and can store redundancy between data segments. Data redundancy after compression can be greatly reduced, especially when N data segments have the same or similar data structure.

  However, with the apparatus shown in FIG. 1, it is impossible to randomly access each compressed data segment. This is because in the compression process as shown in FIG. 1, the compression of the next data segment, ie, data segment K (where K is a number from 1 to N), is all the previously compressed data segments, ie This is because it depends on the data segments 1 to K. Thus, during the decompression process, it must first decompress all data segments (data segments 1 to K) before the data segment (data segment K), and then the compressed data segment (data segment K). ) Can be decompressed. As an extreme example, to obtain the last data segment (data segment N) from the compressed data, the data segment until all of the previous compressed data segments of data segment N are sequentially decompressed and discarded. N cannot be acquired.

  FIG. 2 is a structural schematic block diagram illustrating another apparatus for sequentially compressing / decompressing data according to the prior art. The difference from the above device is that the compression / decompression of the data segments is independent of each other. As shown in FIG. 2, the apparatus is compressed with a sequential encoder 120 that is used to independently and sequentially compress data segment K, which can be any one of data segments 1-N. A memory 130 used to store the data segment K, and a sequential decoder 140 used to decompress the compressed data segment K and decompress the data segment K. In the above compression process, the N data segments are separated and compressed / decompressed, the compression / decompression process of each data segment is independent, and the sequential compression / decompression of subsequent data segments is performed on the previously compressed data segment. Do not depend. When data segment K is obtained by decompression, it is only necessary to detect compressed data segment K directly from memory 130 and perform sequential decompression without having to decompress the data segment before that. This data segment is randomly accessible by some approach because the device compresses the data segment independently. However, since each data segment is compressed independently, the redundancy between them cannot be exploited, which results in a low overall data compression ratio.

  Accordingly, there is a need for a new method and apparatus for sequentially compressing / decompressing data that is efficiently compressed and randomly accessible.

  One object of the present invention is to overcome the shortcomings of current sequential compression / decompression schemes and provide a new method and apparatus for sequentially compressing / decompressing data, thereby efficiently and sequentially In addition to being able to compress, it is possible to randomly access data.

  The present invention provides a method for sequentially compressing data segments having a data structure. First, a plurality of compression parameters are acquired, and second, data segments are sequentially compressed according to the acquired parameters, and compressed data segments are acquired. Here, the data segment can be obtained by data preprocessing. The compression parameters are obtained from the storage device, and perhaps the compression parameters can also be obtained from compression of a template having this data structure. In the compression of the template having this data structure, it is possible to obtain a compressed template in addition to the compression parameter. The compressed template is stored separately from the compressed data segment, or the compressed template is discarded.

  The present invention also provides a method for sequentially decompressing compressed data segments having a data structure. First, a plurality of decompression parameters are acquired, and secondly, data segments compressed according to the acquired decompression parameters are sequentially decompressed to obtain decompressed data segments. The decompression parameters are obtained from the storage device, and possibly the decompression parameters can also be obtained from the sequential decompression of the compressed template having the above data structure. Sequential decompression of the compressed template can obtain the decompressed template in addition to the decompression parameters. The decompressed template is discarded.

  The present invention also provides an apparatus for sequentially compressing data segments, the apparatus comprising an acquisition device and a compression device, wherein the data segment has a data structure. An apparatus for sequentially compressing data segments can optionally include a preprocessing device, a storage device, and a discarding device. The acquisition device is used to acquire a plurality of compression parameters, and the compression device is used to sequentially compress the data segments according to the acquired compression parameters, thereby acquiring the compressed data segments. The compression device is also used to sequentially compress the template having the above data structure, thereby obtaining the compression parameters and the compressed template. The preprocessing device is used to preprocess data, thereby acquiring the data segment. The storage device is used to store compression parameters, and the discard device is used to discard the compressed template.

  The present invention provides an apparatus for sequentially decompressing compressed data segments, the apparatus having an acquisition device and a decompression device. An apparatus for sequentially decompressing compressed data segments can also optionally include a storage device and a discard device. The acquisition device is used to acquire a plurality of decompression parameters, and the decompression device is used to sequentially decompress data segments compressed according to the acquired decompression parameters, thereby having a special data structure. Get the decompressed data segment. The decompressor is also used to decompress the compressed template, thereby obtaining a decompressed template having the decompression parameters and the above data structure, and the storage device is utilized to store the decompression parameters; The discard device is used to discard the decompressed template.

  The above method and apparatus for sequentially compressing / decompressing data provided by the present invention improves the data compression ratio because the data structure part (template) can be filtered after compression of the data segment having a special data structure. After individual compression of multiple data segments with special data structure, the data structure part of each data segment can be filtered, but keep the compressed template or all compressed It only discards the template, which greatly improves the data compression ratio and allows each data segment to be processed randomly.

  Other objects and advantages of the present invention will become apparent and a comprehensive understanding of the invention can be obtained by reference to the following description, which is taken in conjunction with the drawings and the claims.

  FIG. 3 is a structural schematic block diagram illustrating an apparatus for sequentially compressing data segments having a data structure according to an embodiment of the present invention. The device 300 includes an acquisition device 310 and a compression device 320. The device 300 can also include a pre-processing device 330, a storage device 340, and a discard device 350.

  The acquisition device 310 is used to acquire a plurality of compression parameters indicating encoded compression information of a template having a data structure including information such as a template data structure, a storage position, and a compression mode. The compression parameter corresponds to the internal state of the compression apparatus 320 after compressing the template having the data structure.

  The compression device 320 is used to sequentially compress data segments that are compressed according to the acquired compression parameters, and as a result, acquires compressed data segments. The compression device 320 is also used to sequentially compress a template having a data structure, thereby obtaining the compression parameters. The data to be compressed is input from the pre-processor 330, and possibly it can also be input from other data sources (not shown).

  The preprocessing device 330 is used to preprocess input data, and as a result, generates a data segment according to the data structure. This data segment is dynamic data and changes depending on the difference of input data. For example, one suitable specific example of the pre-processing device is a database record publishing processing unit by XML (extensible Markup Language). When the input data is blank data having no content, the preprocessing device 330 can generate a template having the data structure. This template describes the data structure of the data segment, but it does not contain dynamic content data.

  The preprocessor 330 can output any type of category of data segment according to different circumstances. For example, it can generate different data segments according to different types of data input from a database. In this case, for each type of data segment, it is possible to generate a template having a corresponding data structure that is used to compress the corresponding data segment. Perhaps a generic template can be configured for use by all types of data segments during compression, but this scheme can only provide a compression ratio with sub-optimal performance.

  Storage device 340 is used to store compressed data segments and compression parameters, and is also used to store templates of compressed data structures. The storage device 340 may be a hard disk, a USB (Universal Serial Bus) disk, a cache, or the like. However, it should be noted that the compressed data segment and the compressed template should be stored separately. For example, they can be stored with different file names. The compressed template stored in the storage device 340 is for use during decompression of the data segment.

  The discard device 350 is used to discard the compressed template. The compressed template is input from the compression device 320. In order to avoid occupying too much space in the storage device, the compressed template can also be discarded.

  In FIG. 4, the execution process of the apparatus 300 is described in detail below.

  FIG. 4 is a flowchart for sequentially compressing data segments according to an embodiment of the present invention. The data segment has a certain data structure. First, a template having the data structure is acquired (step S410). This template can be obtained after preprocessing data that does not have valid content. In the preprocessing process, a tag such as a file end tag can be added to the end of the template. This tag is used to indicate the end position of the template file.

  The data structure of the television program data is described below.

上記データ構造は、当該テンプレートのコンテンツである。

The data structure is the content of the template.

  Next, the templates are sequentially compressed (step S420). In particular, the encoding state is started first, and then the template is sequentially compressed using a Zlib sequential compression algorithm or the like, whereby a compressed template and a plurality of compression parameters are obtained. The compression parameter indicates the encoded compression information of the template and its storage position in the compression device, including information such as the compression mode, data structure, and Huffman list. Note that the Zlib sequential compression algorithm is not the only compression scheme. The implementer can select different compression algorithms according to the needs.

  Thereafter, the compressed template is discarded (step S430). This compressed template does not affect the overall data compression rate and can be discarded by the system to secure storage space. Since there is a tag such as a file end tag at the end of the template, the system can automatically discard the compressed template after reading the tag. Perhaps the compressed template can also be stored in a special location on a local storage device for subsequent decompression. In particular, when a plurality of data segments having the data structure are sequentially compressed using the method of the present invention, one of the compressed templates can be maintained and the other compressed templates can be discarded. .

  Steps S410 to S430 are procedures for sequentially compressing a template having the data structure and obtaining a compression parameter thereafter. The compression parameters can be stored in a local storage device or the like for use in compressing other data segments having the same or similar data structure. In this case, when the compression parameter is stored in the local storage device, the steps S410 to S430 can be omitted when compressing another data segment having the same data structure. That is, the compression parameter can be obtained directly from the storage device without compressing the template having the data structure. Before obtaining the compression parameters directly from the storage device, the coding state should first be initialized to ensure that the current process is not affected by the compression parameters at the end of the previous process.

  Next, a data segment having the data structure is acquired (step S440). This data segment is acquired from pre-processing of data, such as acquisition of a data segment according to XML (extensible Markup Language) from a publishing processing unit for database records in XML format. Perhaps the data segment can also be input from other data sources.

  Using the following specific example, the relationship between the template and the data segment can be explained in more detail. For example, after pre-format processing, two data segments of TV program data are acquired, and data segment 1 is

であり、データセグメント２は、

And data segment 2 is

である。

It is.

  Both of the above data segments have the same data structure of the above template, and the difference between them is that each data segment is a new TV talk show every Friday, such as “New Talk Show”, etc. The data segment 2 includes different effective contents such as a nightly news report.

  Next, the data segments are sequentially compressed according to the acquired compression parameters (step S450). Since compression parameters include compression information of templates having the same data structure and information such as storage addresses and lengths, sequential compression characters are characters that appear when data segments are sequentially compressed according to the compression parameters. The string can be automatically searched, and the position and length at which the character string of the template appears can be automatically output. Thus, the principle of sequential coding can automatically replace the same segment of the data segment and the compressed template with the storage address and length and the storage address point to the compressed template, after The data segment size is much smaller than that of the original data segment. Since the data segment is sequentially compressed by the compression parameter generated during the template compression process, and the compressed template may be discarded, the compression ratio of the data segment can be greatly improved.

  Finally, the compressed data segment is stored (step S460). The storage locations of the compressed data segment and the compressed template are different in order not to affect the compression ratio, and for example, they are stored with different file names.

  The above embodiment completes compression for one data segment, and in many cases, it is required to sequentially compress multiple data segments having the same template. For example, it may compress 20 television program data (ie, 20 data segments), which have the same or similar data structure after preprocessing. After compressing the data segment according to the process S460, the next data segment can be sequentially compressed according to the process. At the beginning of every process, the encoding state is, for example, when the current process ends, the parameters already have some compression parameters in the data segment corresponding to valid content, so at the end of the current process. It should be noted that the compression parameters should be initialized, such as clearing all compression parameters of the compressor, to ensure that the compression parameters do not affect the next process.

  FIG. 5 is a structural schematic block diagram illustrating an apparatus for sequentially decompressing compressed data segments according to an embodiment of the present invention. A compressed data segment is obtained from the device 300, and the data segment has this data structure. The device 500 includes an acquisition device 510 and a decompression device 520. The device 500 can also include a storage device 540 and a discard device 550.

  The acquisition device 510 is used to acquire a plurality of decompression parameters. These decompression parameters indicate the decompression information of the compressed template including information such as the template data structure, storage location, decompression style, and the like. These compression parameters may be obtained from the storage device 540. The decompressed parameters correspond to the internal state of the decompression device 520 after decompressing the compressed template.

  The decompressor 520 is used to sequentially decompress data segments that have been compressed according to the obtained decompression parameters, thereby obtaining the decompressed data segments. If the decompression parameters cannot be obtained directly from the acquisition device 510, the decompression device 520 can also be used to sequentially decompress the compressed template to obtain the decompressed template and decompression parameters. And the template has the above data structure.

  A storage device 540 is used to store the decompressed data segments and store decompression parameters.

  A discard device 550 is used to discard the decompressed template.

  In FIG. 6, the execution process of the apparatus 500 is described below.

  FIG. 6 is a flowchart for sequentially decompressing compressed data segments according to an embodiment of the present invention. The compressed data segment is generated by the process of FIG. 4, and this data segment has the data structure.

  First, a compressed template is acquired (step S610). This template has the data structure. The compressed template is obtained from a storage device and can be obtained from a network or other device. A compressed template can also be obtained by sequentially compressing a template having the data structure.

  Thereafter, the compressed templates are sequentially decompressed (step S620). More specifically, after initialization of the decoding state, the compressed template is sequentially decompressed, and the decompressed template and a plurality of answer parameters are obtained. These decompression parameters are available for decompressing subsequent data segments to be decompressed and include decompression information for a compressed template having information such as decompression style, address, data structure, and the like. Sequential decompression can utilize the Zlib sequential decompression principle. In order to secure storage space, the decompressed template can be discarded (step S630), and this destruction can be realized by the file end tag of the template.

  Steps S610 to S630 are procedures for sequentially decompressing the compressed template and thereby obtaining the decompression parameters. These decompression parameters can be stored, for example, in a local storage device and can be used to sequentially decompress other compressed data segments having the same or similar data structure. In this case, if the decompression parameters are stored in a local storage device, the above steps S410-S430 can be skipped when decompressing other compressed data segments, ie, the decompression parameters are: It is possible to obtain the compressed template directly from the storage device without decompression. Before obtaining the decompression parameters directly from the storage device, it should first be initialized to ensure that the current process is not affected by the decompression parameters at the end of the previous process.

  Next, a compressed data segment is acquired (step S640). This compressed data segment can be obtained from a storage device or transferred from a network or other device.

  Thereafter, the compressed data segments are sequentially decompressed (step S650). This sequential decompression can use the Zlib sequential decompression principle. According to the storage address of the compressed data segment and the obtained decompression parameter, corresponding replacement information is detected, the decompressed data segment is replaced with the replacement information, and then the completed decompressed data segment is obtained. The decompressed data segment has the data structure and includes specific valid content.

  Finally, the decompressed data segment is stored (step S660).

  The decompression process completes decompression of the compressed data segment. In many cases, it is necessary to sequentially decompress a number of compressed data segments having the same / similar data structure. After decompressing the compressed data segment using the process S660, the next compressed data segment can be sequentially decompressed using the process. At the beginning of each process, the encoding state is uncompressed at the end of the current process, for example because the parameters at the end of the current process already have some decompression parameters in the data segment corresponding to valid content. Note that it should be initialized, such as clearing all decompression parameters of the compressor, to ensure that the parameters do not affect the next process.

  The method of sequentially compressing / decompressing data in the above embodiment mainly transfers data by simultaneously preprocessing the data into data segments having the same data structure, and a data segment for compressing / decompressing the data is used. Get the template to configure. The method obtains a compression parameter required for compression of a data segment having the same data structure by compressing a template having a certain data structure, and sequentially compresses the data segment according to the parameter. Discard the generated template, thereby obtaining a data segment having a high compression ratio. In the meantime, since each data segment is separately and separately compressed, it may randomly access each data segment. Correspondingly, the method obtains the compression parameters required for decompressing the compressed data segment having the same data structure as the template by decompressing the compressed template and compresses according to the parameters. The decompressed data segment is decompressed, thereby obtaining the decompressed data segment. During this time, since each data segment is separately and sequentially compressed, it may decompress each compressed data segment separately.

  The following experimental results can further show the effect of the compression ratio of the present invention compared to the prior art. This experiment is based on the compression ratio of data segments using the technique described in FIG. 2. When the same data segment is sequentially compressed, the corresponding ratio is 38. 4% improvement. All these methods utilize Zlib as a sequential compression / decompression principle. These two results are listed in the following table.

本発明が好適な実施例に関して説明されたが、ここに記載された処理に代替、改良及び変更が適用可能であるということは当業者に明らかであろう。従って、以下の請求項に与えられるような本発明の範囲及びコンセプトの範囲内に、このようなすべての代替、改良及び変更は含まれるとみなされる。

Although the present invention has been described in terms of a preferred embodiment, it will be apparent to those skilled in the art that alternatives, improvements and modifications may be applied to the processes described herein. Accordingly, all such alternatives, modifications and variations are considered to be included within the scope and concept of the invention as provided in the following claims.

FIG. 1 is a structural schematic block diagram illustrating an apparatus for sequentially compressing / decompressing data according to the prior art. FIG. 2 is a structural schematic block diagram illustrating another apparatus for sequentially compressing / decompressing data according to the prior art. FIG. 3 is a structural schematic block diagram illustrating an apparatus for sequentially compressing data segments according to an embodiment of the present invention. FIG. 4 is a flowchart for sequentially compressing data segments according to an embodiment of the present invention. FIG. 5 is a structural schematic block diagram illustrating an apparatus for sequentially decompressing compressed data segments according to an embodiment of the present invention. FIG. 6 is a flowchart for sequentially decompressing compressed data segments according to an embodiment of the present invention.

Claims (25)

A method for sequentially compressing a data segment having a data structure,
a. Obtaining a plurality of compression parameters;
b. Sequentially compressing the data segments according to the acquired compression parameters to obtain compressed data segments;
A method characterized by comprising: The method of claim 1, comprising:
The step a includes sequentially compressing a template having the data structure, thereby obtaining the compression parameter. The method of claim 2, comprising:
The method, wherein step a further comprises obtaining a compressed template. The method of claim 3, further comprising:
A method comprising discarding the compressed template. The method of claim 3, further comprising:
Storing each of the compressed template and the compressed data segment. The method of claim 1, comprising:
The compression parameter in step a is obtained from a storage device. The method of claim 1, further comprising:
A method comprising pre-processing data to thereby obtain the data segment. The method of claim 1, further comprising:
A method comprising storing the compressed data segment. A method for sequentially decompressing a compressed data segment having a data structure comprising:
a. Obtaining a plurality of decompression parameters;
b. Sequentially decompressing the compressed data segments according to the obtained decompression parameters, thereby obtaining decompressed data segments;
A method characterized by comprising: The method of claim 9, comprising:
The step a includes sequentially decompressing a compressed template having the data structure, thereby obtaining the decompression parameters. The method of claim 10, further comprising:
A method comprising: sequentially compressing a template having the data structure, thereby obtaining the compressed template. The method of claim 11, comprising:
The method wherein the compressed template is obtained from a storage device. The method of claim 10, comprising:
The method, wherein step a further comprises obtaining a decompressed template. 14. The method of claim 13, further comprising:
A method comprising discarding the decompressed template. The method of claim 10, further comprising:
A method comprising storing the decompressed data segment. The method of claim 9, comprising:
The method wherein the decompression parameter is obtained from a storage device. An apparatus for sequentially compressing a data segment having a data structure,
Obtaining means for obtaining a plurality of compression parameters;
Compression means for sequentially compressing the data segments according to the acquired compression parameters, thereby obtaining compressed data segments;
A device characterized by comprising: The apparatus of claim 17, further comprising:
An apparatus comprising preprocessing means for preprocessing data and thereby obtaining said data segment. The apparatus of claim 17, further comprising:
An apparatus comprising storage means for storing the compression parameter. The apparatus of claim 17, comprising:
The apparatus is also characterized in that the compression means can be used to sequentially compress the template having the data structure, thereby obtaining the compression parameter and the compressed template. The apparatus of claim 20, further comprising:
An apparatus comprising: discarding means for discarding the compressed template. An apparatus for sequentially decompressing a compressed data segment having a data structure comprising:
Obtaining means for obtaining a plurality of thawing parameters;
Decompression means for sequentially decompressing the compressed data segments according to the obtained decompression parameters, thereby obtaining decompressed data segments;
A device characterized by comprising: The apparatus of claim 22, further comprising:
An apparatus comprising storage means for storing the decompression parameter. 23. The apparatus of claim 22, wherein
The decompression means can be used to sequentially decompress the compressed template, thereby obtaining the decompression parameter and the decompressed template having the data structure. apparatus. 25. The apparatus of claim 24, further comprising:
An apparatus comprising: discarding means for discarding the decompressed template.

Images