JP3344755B2 - Ascending integer sequence data compression and decoding system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a system for compressing and decoding monotonically increasing ascending integer sequence data.
In particular, the present invention relates to a system for compressing and decoding data obtained when data searched in a database search system for extracting necessary information from a database is integer string data arranged monotonically.

[0002]

2. Description of the Related Art Conventionally, Huffman method, Shannon Fano method, Gilbert Moore method, run-length encoding method and the like are known as typical methods for compressing and decoding data. For example, Japanese Patent Application Laid-Open No. 2-78323 is an example using the Huffman method.

[0003]

These methods mainly measure the appearance frequency of each character of data, and compress the data size preferentially from those having higher frequency. Although these methods have an advantage that they can be applied to data of any form, they require several stages of processing for compression and decoding, and are therefore unsuitable especially when high speed is required.

SUMMARY OF THE INVENTION In view of the above problems, the present invention provides a method for compressing integer sequence data arranged monotonically (in ascending order) at a high speed, and reducing the capacity of a storage means for storing the compressed data. It is an object of the present invention to provide a compression and decoding system capable of performing the following.

[0005]

According to the compression and decoding system of the present invention, in the compression and decoding of integer sequence data arranged in ascending order, the n-th integer sequence data arranged in ascending order is used.
From the data of the eye, the (n-1) th stored in the first storage means
Of the nth data,
Subtraction means for sending to the first storage means, division by using the difference value obtained by the subtraction means as a dividend, quotient and remainder to be output, and quotient obtained by the first division means to be 0. Quotient comparison means for comparing with the quotient non-zero result of the comparison by the quotient comparison means, as a dividend, outputting the remainder together with the quotient and the carry mark, and the second division means. that stores the carry marks and the remainder from the first and second storage means for storing the remainder output from the division means, second storage means to the stored carry marks and two much data Decoding means for decoding the original integer sequence data.

[0006]

According to the present invention, at the time of compression, the ascending data is subtracted from the already stored old integer data, the difference obtained by the subtraction is divided as the dividend, the remainder is output, and the quotient is set to 0. The quotient other than 0 is divided as the dividend, and the remainder is output together with the carry mark. This division is repeated until the quotient becomes 0. The carry mark obtained by the latter division, the remainder, the former The remainder obtained by dividing by is saved. Therefore, since the difference value is divided and the obtained data is stored, the amount of calculation can be greatly reduced as compared with the conventional general compression encoding method, and compression and decoding can be performed at high speed. it can. In addition, since parameters for the entire data such as statistics are not required, addition or deletion of data can be easily performed.

[0007]

FIG. 1 shows an embodiment of the system according to the present invention. As shown in the drawing, ascending integer sequence data D1 includes 320, 333, 401. . . And are arranged in a monotonically increasing manner (ascending order). These data are represented by, for example, 32 bits. The integer sequence data D1 is sent to the subtraction unit 22 in the compression device. The subtraction unit 22 performs subtraction using the integer value of the data sent before the data, and sends the currently sent integer value to the storage unit 11.
That is, since the data transmitted immediately before the data transmitted this time is stored in the storage unit 11, the data is read out by the subtraction unit 22, and the subtraction unit 22 reads the data transmitted immediately before the data transmitted this time. Is subtracted, and the result of the subtraction is sent to the division unit 12.

The subtraction unit 22 performs this subtraction and sends the data sent this time to the storage unit 11. Storage unit 11
Stores the latest data sent from the subtraction unit 22. Note that the initial value of the storage unit 11 is 0.

The division unit 12 divides the subtraction data sent from the subtraction unit 22 by a predetermined value. In this embodiment, the input subtraction data is divided by 255. The obtained quotient is the quotient comparison unit 13
The remainder is sent to the compressed sequence D2 processing unit 16.

The quotient comparing section 13 compares the input quotient with 0, and sends the quotient to the dividing section 14 if the quotient is not 0. If the quotient is 0, no data is sent to the division unit 14,
An instruction is given to input the next data of the ascending integer sequence data D1 to the subtraction unit 22. The division unit 14 divides the quotient input from the quotient comparison unit 13 by a predetermined value, in this embodiment, 256, and sends the obtained quotient to the quotient comparison unit 13 again. The remainder is a mark character indicating a carry. It is sent to the compression sequence D2 processing unit 16 together with C. The division by the division unit 14 is repeated until the quotient sent to the quotient comparison unit 13 is determined to be 0.

The compression processing of the ascending integer sequence data is performed as described above, and the compressed data sequence is processed by the compression sequence processing unit 1.
6 is stored.

Next, a specific example will be described. When the first data 320 is sent to the subtraction unit 22, the initial value stored in the storage unit 11 is 0, so 0 is subtracted from 320, the difference value becomes 320 and sent to the division unit 12. The division unit 12 divides the difference value 320 by 255 to obtain a quotient 1 and a remainder 65. The quotient comparison unit 13 compares the input quotient with 0, and since the quotient is not 0, the quotient 1 is sent to the division unit 14. The division unit 14 divides the quotient 1 by 256 to obtain the quotient 0 and the remainder 1, so that the division unit 14
At the same time, the remainder 1 is sent to the compressed sequence processing unit 16 to notify the next integer value to be read.

The compression number sequence processing section 16 carries the carry mark character C sent from the division section 14 with the remainder 1 and the division section 12
The remaining 65 sent from is stored.

Next, when the data 333 is sent from the integer sequence data D 1, the subtraction unit 22 subtracts 333 by 320 stored in the storage unit 11 to obtain a difference 13. The difference 13 is sent to the division unit 12, which divides by 255 to obtain a quotient 0 and a remainder 13. The quotient 0 is sent to the quotient comparing unit 13, and the quotient comparing unit 13 determines the quotient 0 and issues an instruction to read the next integer value from the integer sequence D1. In this case, since the quotient 0 is determined by the quotient comparing unit 13, only the remainder 13 sent from the dividing unit 12 is sent to the compressed sequence processing unit 16 and stored.

By repeating the same operation, the compressed data is sequentially sent to the compressed sequence D2 processing unit 16. These compressed data are stored in the storage unit 15.

As described above, in the compression processing, a difference D of a sequence is calculated in order from the head of an ascending integer sequence, a quotient P (0) obtained by dividing the difference D by a constant L (0), and a remainder Q (0) are obtained. If P (0) is 0, only the remainder Q (0) is stored in the storage means. If P (0) is not 0, the quotient P (0) is further divided by a constant L (1).
(1) The remainder Q (1) is calculated. Then, this quotient P (i) (i = 1,
The division is repeated with the quotient P (i-1) calculated immediately before until (2,...) Becomes 0.

When expressed by a recurrence formula, D = L (0) × P (0) + Q (0) P (i−1) = L (i) × P (i) + Q (i) (i = 1,2 , ...) At this time, a numerical value is stored for one difference value D as follows. When P (0) = 0, only Q (0) When P (n) = 0 (n> 1), C, Q (1), C, Q (2),. . . C, Q (n), Q (0) where C is a mark character distinguishable from Q (0). As for the leading value of the integer sequence, the value itself is used as the difference value. The divisor L (i) is defined in advance.

If L (i) is set large, the result of division is
The case where the quotient becomes 0 increases, and the calculation cost can be reduced. Conversely, if L (i) is set to be small, the number of divisions increases and the calculation cost increases, but the remainder can be kept small, and the storage area can be reduced.

Next, in decoding, the numerical value decoded immediately before is stored and used. For the first integer value, 0 is used as a value held in advance.

First, the compressed data stored in the storage unit 15 is taken out by the compressed sequence D2 processing unit 16 and read by the reading unit 17.
Is read by When the mark character C indicating a carry appears in the compressed data, the reading unit 17 sends the data immediately after that to the bias processing unit 18. Further, regardless of the presence or absence of the mark character C, the remaining data is sent to the adding unit 19 and notified to the bias processing unit 18.

For example, since the first compressed data in the present embodiment is the mark character C indicating a carry, the reading unit 17 reads the immediately following data 1 and sends it to the bias processing unit 18. Next, the reading unit 17 reads the remaining data 65 and sends it to the adding unit 19.

The bias processing section 18 counts the number of mark characters after the data sent from the reading section 17. The bias is calculated according to the count number and sent to the adding unit. The count number is initialized to 0 when the reading unit 17 receives a notice of the remaining reading.

In the present embodiment, the first data sent to the bias processing unit 18 is multiplied by 255,
The nth (n = 2, 3,...) -Th data is multiplied by 255 × 256 ⁽ⁿ⁻¹⁾ and sent to the adder 19. Therefore, in the present embodiment, 255 obtained by multiplying the first sent data 1 by 255 is sent to the adder 19.

The adding section 19 sequentially adds the bias values sent from the bias processing section 18, adds the remainder sent from the reading section 17, and decodes and adds the remainder to the storage section 23. The integers are read and added. The initial value of the storage unit 23 is set to 0.

In this embodiment, 255 sent from the bias processing unit 18 and 65 sent from the reading unit 17 are used.
Is added, and the initial value 0 of the storage unit 23 is further added.
The decrypted data 320 is obtained. The obtained decoded data is sent to the restored integer sequence D3 holding unit 21 and output as needed.

Thus,

(Equation 1)

When the n mark characters and the remainder are read, the difference D becomes D = Q (0) or

[0028]

(Equation 2)

The difference D can be decoded as
The original integer value can be decoded in addition to the held immediately previous decoded integer value.

According to the present embodiment, the ascending data is subtracted from the already stored old integer data at the time of compression as described above, the difference obtained by the subtraction is used as a dividend, the remainder is output, and the remainder is output. , The quotient is compared with 0, the non-zero quotient is divided as a dividend, the remainder is output together with the carry mark, and this division is repeated until the quotient becomes 0, and the carry obtained by the latter division is obtained. The mark, the remainder, and the remainder obtained by the former division are preserved. Therefore, since the difference value is divided and the obtained data is stored, the amount of calculation can be significantly reduced as compared with the conventional general compression encoding method, so that high-speed compression and decoding are performed. Can be. In addition, since parameters for the entire data such as statistics are not required, addition or deletion of data can be easily performed.

A compression and decoding system according to the present invention comprises:
The present invention can be applied to compression and decoding of various types of integer sequence data arranged in ascending order. For example, the present invention can be applied to data processing in the following data search system.

FIG. 2 is a data flow diagram of a pattern search system by extracting a nearby feature quantity showing an embodiment to which the present invention is applied. In this search system, neighboring feature data is created from all target properties in advance by omitting all phase information of events (information), and a search for all properties is performed for the data group. The search algorithm includes a learning step and a search step. In the learning step, a neighborhood feature amount matrix is created as phase information for each property. In FIG. 2, this corresponds to a step of creating a neighborhood feature amount matrix 30 from the search target 10 and storing it in the structure file 40.
Also, in the search step, the same processing as in the learning step is performed on the search key to obtain the neighboring feature amount of the search key, and a matching operation is performed with the neighboring feature amount matrix of the property, and the matching degree ( (Similarity) is obtained. In FIG. 2, a matching operation with the property data of the structure file 40 is performed in search S4 based on the search key 50, and the evaluation result list 70 or the sorted list 80 is displayed.
Corresponds to the steps until the result is output. Hereinafter, each step will be described.

(1) Learning step In FIG. 2, the search target 10 is, for example, Japanese, English,
Document data in German, French, Hebrew, Russian, etc., or quantized waveform numerical data, chemical structural formulas, genetic information, etc. For such a search target, first, normalization processing is performed by the normalization means S1. Generally, a search target is represented by a sequence of the minimum unit of information (a character such as an alphabet in a document, a real number at a certain time in a numerical chart, and the like). It is converted into an integer sequence of n gradations by some method. This is called data normalization.

For example, in the case of English document data, ASCII
By using the code table as it is, the following 25
It is realized as a numerical representation of six gradations. …… This is a pen. …… 84 ｜ 104 ｜ 105 ｜ 115 ｜ 32 ｜ 105 ｜ 115 ｜ 32 ｜ 97 ｜ 32 ｜ 112 ｜ 101 ｜ 110 ｜ 46 ｜

In the above code, T is 84 and h is 10
Four. . It corresponds to.

Next, from the normalized data 20, a neighboring feature value is calculated by the learning means S2, and is convolved into a format of a nearby feature value matrix 30 by the procedure described below. Here, various arithmetic expressions for calculating the neighborhood feature amount can be considered. This arithmetic expression also affects the sharpness of the search (less overdetection).

As an example of the learning means S2, a procedure for obtaining a quantization amount from the normalized data 20 and obtaining a neighboring feature amount matrix 30 using the quantization amount will be described. For example, FIG.
Suppose that there are a plurality of target properties (documents) to be searched, and the quantization of the i-th property is considered. Here, j-th data (characters) to C _i of the i-th property _(document), and _j, C _i, the data for the k-neighborhood of _{j C i, j + 1,} C i, j + 2,. ..., Ci _{, j + k} . In the i-th property, a normalized numeric string 135, 64, 3 as shown in FIG.
7,71,101 and ... is that alongside, C _i, the quantization amount regarding _j x and C _i, the quantization weight for Upcoming k near the _j y
X = f (C _{i, j} ) y = g (C _{i, j} , C _{i, j + 1} , C _{i, j + 2} , ..., C _{i, j + k} ) ... ).

Here, f (C _{i, j} ) is an n-stage quantization function for C _{i, j} . That is, it is a value obtained by performing a predetermined operation on the j-th data C _{i, j} of the i-th property, and is represented by any integer from 1 to n. Therefore, the position in the x-axis direction in the matrix (coordinates) shown in FIG. 4 is determined in the range of 1 to n by the value of the quantization amount x obtained by the calculation of the n-stage quantization function f.

G (C _{i, j} , C _{i, j + 1} , C _{i, j + 2} ,..., C
_{i, j + k} ) is an m-step quantization function for the neighborhood of k in front of C _{i, j} . That is, the j-th data C of the i-th property
_{i, j} and a predetermined number of data C near the data C _{i, j
i, j + 1} , C _{i, j + 2} ,..., C _{i, j + k} are values obtained by performing a predetermined operation, and are represented by any integer from ₁ to _m . For example, as shown in FIG. 3, the j-th data C _{i, j}
Is 135 and k is 3, C _{i, j + 1} , C _{i, j + 2} ,
Data 64, 37 following data 135 as C _{i, j + 3} ,
71, and a predetermined calculation is performed on the correlation between these data and the data 135. If the j-th data Ci _{, j} is the next 64, the data 37, 71, 101 following the data 64 are extracted as Ci _{, j + 1} , Ci _{, j + 2} , Ci _{, j + 3.} Then, a predetermined operation is performed on the correlation between the data and the data 64. The position of the matrix (coordinates) shown in FIG. 4 in the y-axis direction in the range of 1 to m is determined by the value of the quantization amount y obtained by the calculation of the m-stage quantization function g.

Therefore, by obtaining the quantization amounts x and y from the data 20 normalized as described above, FIG.
Is determined in the matrix (coordinates) shown in FIG. There are various arithmetic expressions f () and g () for obtaining the quantization amount. For example, f: x → xg: (x, y) → xy (or | xy |) The operation expression f () may use the input value as the quantization amount as it is, and the operation expression g () may use the difference between the two input values or the absolute value of the difference as the quantization amount. In this case, the normalized data 20 corresponds to the previous example 84 | 104 | 105 | 11.
5 ....., assuming that data C _{i, j} is 84, C _{i, j} and C _{i, j}
The coordinate positions of the quantization amounts x and y with respect to the neighborhood of k in front of
4,20), (84,21), (84,31), ..... In addition to the equation (2), the neighborhood feature may be extracted by performing four arithmetic operations on individual character integer values of some character strings. The coordinate positions (51, 51) of the quantization amounts x and y shown in FIG.
71), (32, 103),... Are obtained by a method different from the above equation (2).

In the present system, each piece of property information is stored as a set of a property serial number i and a weight w (x, y, i) with respect to the quantization amounts x and y obtained as described above. Weight w (x, y,
i) is obtained by a predetermined calculation from the data x, y, and i, but usually the value of the weight w (x, y, i) may be fixed to 1.

As described above, data C for each property
4 based on the values of the quantization amounts x and y obtained for each of _{i and j} .
Store the data as indicated by the bar at That is, data C _i, the quantization amount x of _j, the position of coordinates defined by the value of y, the weight w and the serial number i of that property
Data containing (x, y, i) is stored. In the figure, the length of the bar is shown to be extended each time such data is stored. If the weight w (x, y, i) is 1, only the data of the serial number i of the property is stored at the position of the coordinates determined by the values of x and y. If the data of the serial number i of the property is integer data arranged in ascending order like the integer sequence data D1 shown in FIG. 1, it is suitable for compression and decoding by the above-described method. Therefore, by performing the above-described compression, data can be compressed at a high speed, and the storage capacity of the data can be reduced.

The property identification number is added to the neighborhood feature amount matrix created in this way, and is stored as the structure file 40.

(2) Search Step First, a search key 50 is input. For example, "This is a pe
n. "is used as a search key. The key information is normalized to the following integer sequence with respect to this search key 50 by the normalization means S3 based on the same normalization method as the normalization means S1 in the learning step. ｜ 104 ｜ 105 ｜ 115 ｜ 32 ｜ 105 ｜ 115 ｜ 32 ｜ 97 ｜ 32 ｜ 112 ｜ 101 ｜ 110 ｜ 46 ｜

Next, in the search means S4, quantization is performed from the head of the numerical sequence of the search key 50 normalized using the same autocorrelation calculation formulas f () and g () as in the learning means S2 in the learning step. Create a series of pairs of quantities x, y. Next, based on the series of pairs of quantization amounts x and y of the search key 50, the search key
0 as containing the frequency omega _{k of, V (x j, y j} , k) and j =. 1 to
It is calculated by summing m.

However, V (x _j , y _j , k) is equal to the weight of the property i stored in the structure file 40, and is set to 0 when it has no weight.

Therefore, when there is data (when there is a bar) at the positions of x and y of the quantization amount in FIG. 4 corresponding to the combination of the quantization amounts x and y obtained from the numerical sequence of the key 50 to be searched.
In the storage section provided separately, the value of the weight is stored in the storage location of the serial number i of the property indicated by the data, the structure evaluation value sc
It is stored as ore (degree of match).

Next, in the evaluation result output means S5, the structure evaluation value scor obtained for each property in the structure file 40 is obtained.
e (matching degree) is the evaluation value for an exact match (in this case,
By dividing by the number of characters -k,), the content probability of the search key 50 is obtained, and a list 70 of evaluation results is obtained. Further, sorting means S6
, The list 70 is sorted in descending order of the content probability to obtain a sorted list 80.

The sorted list 80 is a search result, and by referring to a higher order property, it is possible to know a property name having a high probability that the search key is included in the property. Since the content probabilities are obtained for all of the perfect match and the incomplete match, a fuzzy match search can be performed.

Further, since all the properties are searched for all the information of the search key, the probability of occurrence of a search omission is essentially zero.

The evaluation time of the search key for one property depends only on the number of characters of the key and does not depend on the size of the property. Therefore, the search can be performed at a very high speed.

By performing a logical operation between the search result lists, search operation processing such as AND, OR, and the like for the search condition can be executed at high speed.

The neighboring feature amounts need not be extracted for all data of each property. For example, one or more specific integer values in property data, a specific range of integer values,
Alternatively, one or more specific bits in each byte constituting the data string may be excluded and the neighboring feature amounts may be omitted. Further, in the case of a two-byte character as in a Japanese document, for example, the upper-order byte may be excluded and the neighboring feature amount may be extracted from the lower-order byte.

In the above example, the matrix generated by the neighborhood feature is a 256-order bit matrix, which is 8K
Equivalent to bytes. Therefore, a database in which the data of one property is about 1 Kbyte cannot be said to be an efficient system. Therefore, it is preferable to provide the data compression means S7 as described above and perform data compression to reduce the capacity of the structure file 40.

In the above embodiment, the normalizing means S1,
Learning means S2, normalization means S3, search means S4, evaluation result output means S5, sort means S6, data compression means S7
Can be configured by a computer program, but dedicated hardware may be configured using logic circuit elements.

[0056]

As described above, according to the present invention, the maximum number to be stored is suppressed by calculating the difference and compressing it based on the difference, so that the compression ratio can be improved and the conventional general method can be achieved. The amount of calculation can be greatly reduced as compared with a typical compression encoding method. Therefore, compression and decoding can be performed at high speed. In addition, since parameters for the entire data such as statistics are not required, addition or deletion of data can be easily performed.

[Brief description of the drawings]

FIG. 1 is a data flow diagram of one embodiment of a compression / decoding system according to the present invention.

FIG. 2 is a data flow diagram of a database search system to which the compression / decoding system according to the present invention is applied.

FIG. 3 is a diagram illustrating quantization of neighborhood information.

FIG. 4 is a diagram showing an information structure to be stored.

[Explanation of symbols]

 Reference Signs List 10 search target 11 storage unit 12 division unit 13 quotient comparison unit 14 division unit 15 storage unit 16 compression sequence D2 processing unit 17 reading unit 18 bias processing unit 19 addition unit 20 normalized data 21 restoration sequence D3 holding unit 22 subtraction unit 23 storage Part 30 Neighborhood feature matrix 40 Structure file 50 Search key 60 Normalization key 70 Evaluation result list 80 Sorted list S1 Normalization means S2 Learning means S3 Normalization means S4 Search means S5 Evaluation result output means S6 Sorting means S7 Data compression means

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-64-23623 (JP, A) JP-A-4-326164 (JP, A) JP-A-5-174067 (JP, A) JP-A-5- 181719 (JP, A) JP-A-5-225238 (JP, A) JP-A-5-225248 (JP, A) JP-A-6-274193 (JP, A) (58) Fields investigated (Int. ⁷ , DB name) G06F 5/00 G06F 17/30 H03M 7/18 G06F 12/00 511

Claims (4)

(57) [Claims] 1. A system for compressing and decoding integer sequence data arranged in ascending order, comprising :
Subtraction of the (n-1) th data stored in the first storage means
And stores the n-th data in the first storage
Subtraction means for sending to the means; first division means for performing division with the difference value obtained by the subtraction means as a dividend, and outputting a quotient and a remainder; and quotient obtained by the first division means to be 0. Quotient comparing means for comparison; second dividing means for performing division as a dividend with a quotient other than 0 as a result of the comparison by the quotient comparing means, and outputting a remainder together with the quotient and the carry mark; output from the second dividing means Second storage means for storing the carry mark and the remainder to be output, and the remainder output from the first division means; and the carry mark and the two carry means stored in the second storage means. Decoding means for decoding the original integer sequence data from the remaining data. A system for compressing and decoding ascending integer sequence data. 2. The ascending integer sequence data according to claim 1, wherein said second dividing means outputs said carry mark when dividing a non-zero quotient sent from said quotient comparing means. Compression and decoding system. 3. A third storage means for storing, for each property to be searched, its neighboring feature quantity, and a search means for performing an ambiguous search based on the degree of matching between the neighborhood feature quantity of the search key and the neighborhood feature quantity of the search subject. comprising a, on the data stored in said third memory means, said subtracting means, first and second dividing means, the quotient comparison means, and that the data compressed using the second storage means 2. The system for compressing and decoding ascending integer sequence data according to claim 1, wherein: 4. A quantization amount x for a j-th data string C _{i, j} of an i-th property to be searched and k data strings C _{i, j + 1} , C _{i, j + 2} , ...., Quantization amount y for C _{i, j + k}
And x = f (C _{i, j} ) y = g (C _{i, j} , C _{i, j + 1} , C _{i, j + 2} , ..., C _{i, j + k} ) 4. The compression and compression of ascending integer sequence data according to claim 3, wherein the data is used for searching a database that stores the serial number i of the property at the position of the third storage means determined based on the values of x and y obtained. Decryption system.