JPH05181913A - Compression and decoding system for ascending-order integer string data - Google Patents

Abstract

PURPOSE:To shorten the processing time by decreasing the calculation quantity in the compression and decoding of the ascending-order integer string data. CONSTITUTION:The ascending-order integer string data D1 are divided by a divisor part 12 and the obtained quotient is compared by a quotient storage and comparison part 14 with old quotients which are obtained so far; only when the quotient is varied, the difference and remainder of the quotient are preserved as a compressed string D2 and when not, only the remainder is preserved. The quantity of data is decreased by the division, so the processing time for the compression and decoding is shortened. Further, since parameters required for the whole data are not necessary, the data can be added and deleted.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a compression and decoding system for ascending integer sequence data arranged in a monotonically increasing manner.
In particular, the present invention relates to a compression and decoding system for data retrieved in a database retrieval system for retrieving necessary information from a database, which is integer sequence data arranged in a monotonically increasing manner.

[0002]

2. Description of the Related Art Conventionally, as typical methods for compressing and decoding data, the Huffman method, Shannon-Fano method, Gilbert-Moore method, run-length coding method and the like are known. For example, as a method using the Huffman method, there is JP-A-2-78323.

[0003]

These methods mainly measure the appearance frequency of each character of the data and preferentially compress the size of the data in descending order of frequency. These methods have the advantage that they can be applied to any form of data, but require several stages of processing for compression and decoding, and are therefore unsuitable especially when speed is required.

In view of the above problems, the present invention compresses integer sequence data arranged in a monotonically increasing order (ascending order) at high speed and reduces the capacity of the storage means for storing the compressed data. It is an object of the present invention to provide a compression and decoding system capable of performing.

[0005]

In the compression and decoding system of the present invention, in the compression and decoding of the integer sequence data arranged in ascending order, the dividing means and the dividing means for dividing the integer sequence data arranged in ascending order. The quotient obtained by the above is compared with the already stored old quotient, and when the obtained quotient is larger than the old quotient, the quotient memory comparison means for outputting the difference between these quotients and the quotient difference comparison means A storage means for storing the remainder obtained by the dividing means together with the quotient difference when the is output, and for storing only the remainder obtained by the dividing means if the quotient difference is not output from the quotient storage comparing means. , And decoding means for decoding the original integer sequence data from the quotient difference and the remainder data stored in the storage means.

[0006]

According to the present invention, the data arranged in ascending order is divided at the time of compression, the obtained quotient is compared with the old quotient, and the quotient difference is saved only when there is a quotient difference. The remainder is saved, and if there is no quotient difference, only the remaining data is saved. Therefore, the amount of calculation can be greatly saved as compared with the conventional general compression encoding method, so that compression and decoding can be performed at high speed. Also,
Since data-wide parameters such as statistics are not required, data can be easily added or deleted.

[0007]

1 shows an embodiment of the system according to the invention. As shown in the figure, integer string data D1
320, 333, 401. ． ． And are arranged in a monotonically increasing order (ascending order). These data are for example 32
Expressed in bits. The integer string data D1 is sent to the division unit 12 in the compression device. The division unit 12 divides the input data by a predetermined value. In this embodiment, the input data is divided by 255. The obtained quotient is sent to the quotient storage comparison section 14, and the remainder is the compressed sequence D2 processing section 16
Sent to.

The quotient memory comparing unit 14 receives the new quotient P
Compare new with the stored old quotient Pold. The old quotient Pold is given 0 as an initial value. Quotient memory comparison unit 1
When Pnew> Pold, 4 sends the mark character C indicating a carry and the difference Pnew-Pold of the quotient to the compressed sequence D2 processing unit 16 and stores the new quotient Pnew instead of the stored old quotient Pold. To do. If this condition is not satisfied, the quotient memory comparison unit 14 determines the compressed sequence D2 processing unit 1
No data is sent to 6.

In this embodiment, the first data 320
Dividing 1 by 255 gives the quotient 1 and the remainder 65, but since the initial value of the old quotient Pold is 0, Pne
When w> Pold is satisfied, the quotient memory comparing unit 14 compresses the mark character C indicating a carry and the difference 1 of the quotient into the compressed sequence D2 processing unit 1
A new quotient 1 is stored instead of the stored old quotient 0.

The compressed sequence D2 processing unit 16 stores the mark character C indicating the carry and the quotient difference 1 sent from the quotient storage comparing unit 14, and the remainder 65 sent from the dividing unit 12.

Next, when 333 is sent as the integer string data D1, the division unit 12 similarly divides this by 255. In this case, the quotient is 1, and the remainder is 78. The quotient memory comparison unit 14 compares the new quotient Pnew with the stored old quotient Pold. In this case, since both Pnew and Pold are 1, the above condition Pnew> Pold is not satisfied. Therefore, only the residual data from the division unit 12 is sent to the compressed sequence D2 processing unit 16.

By repeating the same operation, the compressed data is sequentially sent to the compressed sequence D2 processing section 16. These compressed data are stored in the storage unit 18.

In decoding, the compressed data stored in the storage unit 18 is taken out by the compressed sequence D2 processing unit 16 and read by the reading unit 22. The reading unit 22 is
When the mark character C indicating a carry appears in the compressed data, the data immediately after that appears in the bias storage unit 24. Further, the remainder data is sent to the addition unit 26 regardless of the appearance of the mark character C.

For example, since the first compressed data in this embodiment has the mark character C indicating a carry, the data 1 immediately after that is sent to the bias storage section 24. Further, the remainder data 65 is sent to the addition unit 26.

As shown in the figure, the bias storage unit 24 stores the value I based on the quotient, and when the reading unit 22 sends the data ΔP immediately after the mark character C, that is, the difference between the quotients, the divisor. The product L × ΔP of L and the quotient difference ΔP is added to the value I that has been saved up to that point, and the obtained value is saved as a new value I and is output to the adder 26.
The initial value of I is 0.

In this embodiment, 1 is sent as the data ΔP immediately after the mark character C as described above, and the divisor L is 255, so the bias storage unit 24 has 255.
A value 255 obtained by adding x1 to the initial value 0 of I is stored and output to the addition unit 26.

The addition unit 26 adds the I sent from the bias storage unit 24 and the remainder sent from the reading unit 22. In this example, 25 sent from the bias storage unit 24.
5 and the remainder 65 sent from the reading unit 22 are added to obtain the decoded data 320. The obtained decoded data is sent to the restored sequence D3 holding unit 28 and output as necessary.

According to the present embodiment, as described above, the ascending order data is divided by the divisor L at the time of compression, and the obtained quotient is compared with the old quotient so far. And the remainder are saved, and when there is no difference in quotient, only the remaining data is saved. Therefore,
Since the amount of calculation can be significantly saved as compared with the conventional general compression encoding method, compression and decoding can be performed at high speed. In addition, since a parameter for the entire data, such as a statistic, is not required, it is possible to easily add or delete the data.

The compression and decoding system according to the present invention comprises:
It can be applied to compression and decoding of various integer sequence data arranged in ascending order. For example, it can be applied to data processing in the following data search system.

FIG. 2 is a data flow diagram of the pattern search system by extracting the neighborhood feature amount showing an embodiment to which the present invention is applied. In this search system, neighborhood feature amount data in which all phase information of events (information) is removed from all target properties is created in advance, and all property searches are performed on the data group. The search algorithm includes a learning step and a search step. In the learning step, a neighborhood feature amount matrix is created as phase information for each property. In the search step, a matching operation between the search key and the neighborhood feature amount matrix is performed to obtain an evaluation result indicating the matching degree (similarity) for each property. Each step will be described below.

(1) Learning Step In FIG. 2, the search target 10 is, for example, Japanese, English,
Document data in German, French, Hebrew, Russian, etc., or quantized waveform numerical data, chemical structural formulas, genetic information, and the like. For such a search target, the normalization means S1 first performs a normalization process. In general, a search target is represented by a column of minimum units of information (characters such as alphabets in the case of documents, real numerical values at a certain time in the case of numerical charts). It is converted into an integer sequence of n gradations by some method. This is called data normalization.

For example, in the case of English document data, ASCII
By using the code table as it is, the following 25
It is realized as a numerical expression with 6 gradations. …… This is a pen. …… 84 ｜ 104 ｜ 105 ｜ 115 ｜ 32 ｜ 105 ｜ 115 ｜ 32 ｜ 97 ｜ 32 ｜ 112 ｜ 101 ｜ 110 ｜ 46 ｜

In the above code, T is 84 and h is 10
Four . ． It corresponds to.

The normalized data 20 is then convoluted into the form of the neighborhood feature quantity matrix 30 by the learning means S2. Here, various arithmetic expressions for obtaining the neighborhood feature amount are possible. This arithmetic expression also affects the sharpness of search (the degree of overdetection is small).

[0025] Now, j-th data (characters) to C _i of the i-th property _(document), and _j, C _i, the quantization amount x and C _i relates _j, quantization amount for Upcoming k near the _j y Is calculated as follows. Here, the target property (document) to be searched is n
Given that there are individual pieces, the quantization of the i-th property will be described. Assuming that the normalized numerical value sequence 135,64,37,71,101, ... is arranged in the i-th property as shown in FIG. 3, the quantization amount x for C _{i, j} is x = f (C _{i, j} ) Quantization amount y for the front k neighborhood of C _{i, j} is y = g (C _{i, j} , C _{i, j + 1,} C _{i, j + 2, ...,} C _{i , j + k} ).

Here, f (C _{i, j} ) is an n-step quantization function for C _{i, j} . That is, it is a value obtained by performing a predetermined operation on the j-th data C _{i, j} of the i-th property, and is represented by any integer of 1 to n. Therefore, the position in the x-axis direction in the matrix (coordinates) shown in FIG. 4 is determined within the range of 1 to n by the obtained value of x.

Further, g (C _{i, j} , C _{i, j + 1,} C _{i, j + 2, ...,}
C _{i, j + k} ) is an m-step quantization function with respect to the front k neighborhood of C _{i, j} . That is, it is a value obtained by performing a predetermined operation on the j-th data C _{i, j of} the i-th property and a predetermined number of data in the vicinity of that data, and is represented by an integer of 1 to _m. It For example, as shown in FIG. 3, when the j-th data C _{i, j} is 135 and k is 3,
The data 64, 37, 71 following the data 135 are extracted as C _{i, j + 1,} C _{i, j + 2,} C _{i, j + 3} , and a predetermined calculation is performed on the correlation between these data and the data 135. . When the j-th data C _{i, j} is the next 64, C _{i, j + 1,} C _{i, j + 2,}
Data 37, 71, 1 following data 64 as C _{i, j + 3}
01 is extracted, and a predetermined calculation is performed on the correlation between these data and the data 64.

According to the value of y thus obtained,
The position in the y-axis direction in the matrix (coordinates) shown in FIG.
Determined in the range of m. Therefore, by determining x and y as described above, the position in the matrix (coordinates) shown in FIG. 4 is determined.

In this system, each piece of property information has a serial number i and a weight w of the property for x and y obtained as described above.
It is stored as a set of (x, y, i). The weight w (x, y, i) is obtained from the data x, y, i by a predetermined calculation,
Normally, the value of the weight w (x, y, i) is fixed to 1.

The data C _{i, j} obtained as described above
The data is stored for each one based on the x, y values, as indicated by the bars in FIG. That is, the data in which the serial number i of the property and its weight w (x, y, i) are combined is stored at the position of the coordinates determined by the values of x and y of the data C _{i, j} . In the figure, the length of the bar is shown to be extended each time such data is stored. Usually weight w
Since (x, y, i) is set to 1, only the data of the serial number i of the property is stored at the position of the coordinates determined by the values of x and y. Since the data of the serial number i of this property is integer data arranged in ascending order, it is suitable for compression and decoding by the above-mentioned method. Therefore, by performing the above-described compression, the data can be compressed at high speed and the data storage capacity can be reduced.

The identification number of the property is added to the neighborhood feature amount matrix created in this way, and the structure file 40 is saved.

(2) Search Step First, the search key 50 is input. For example, "This is a pe
n. "is used as the search key. The key information is normalized to an integer sequence by the normalization means S3 based on the same normalization method as the learning step for this search key 50. 84 | 104 | 105 | 115 | 32 ｜ 105 ｜ 115 ｜ 32 ｜ 97 ｜ 32 ｜ 112 ｜ 101 ｜ 110 ｜ 46 ｜

Next, in the search means S4, a set of x and y from the head of the normalized numerical value sequence corresponding to each property is calculated using the same autocorrelation calculation formulas f () and g () as in the learning step. Create a series. Next, based on the series of the set of x and y, the search key content frequency ω _k for the property k is V
It is calculated by summing (x _j, y _j, k) for j = 1 to m.

However, V (x _j, y _j, k) is set to be equal to the weight when the property information list has the weight for the property i, and is set to 0 when the property information list does not have the weight.

Therefore, when there is data (when there is a bar) at the position of x, y in FIG. 4 corresponding to the set of x, y of the numerical sequence to be searched (when there is a bar), that data of the storage means provided separately. The value of the weight is stored in the storage location of the serial number i of the property shown in FIG.

Next, in the evaluation result output means S5, the structure evaluation value score (degree of coincidence) obtained for each property is divided by the evaluation value in the case of perfect match (in this case, the number of characters-k,) and a search is performed. A list 70 of evaluation results for which the key content probability is calculated
To get Further, in the sorting means S6, this list 70
Are sorted in descending order of content probability to obtain a sorted list 80.

This sorted list 80 is a search result, and by referring to the higher-ranked property, it is possible to know the property name with a high probability that the search key is included in the property. Since the content probability is obtained for all of the perfect match and the incomplete match, the fuzzy match search can be performed.

Further, since it is a search for all properties for all information of the search key, the probability that a missed search will occur is essentially zero.

Further, the evaluation time of the search key for one property depends only on the number of characters of the key and does not depend on the size of the property. Therefore, the search can be performed very quickly.

By performing a logical operation between the search result lists, search operation processing such as AND and OR for the search condition can be executed at high speed. The autocorrelation equation of the equation (1) can be variously considered in addition to the above example. For example, if f: x → xg: (x, y) → xy (or | xy |), the difference between adjacent characters and every other character (or the absolute value of the difference) is used as correlation information in the neighborhood feature matrix. Can be made. Alternatively, the neighborhood feature amount may be extracted by performing four arithmetic operations on individual character integer values of some character strings.

The neighborhood feature amount does not have to be extracted for all data of each property. For example, a specific one or more integer values in property data, an integer value in a specific range,
Alternatively, one or more specific bits in each byte forming the data string may be excluded to eliminate the neighborhood feature amount. In the case of a double-byte character like a Japanese document, for example, the upper byte may be excluded and the lower-order byte may be taken as the target to extract the neighborhood feature amount.

In the above example, the matrix generated by the neighborhood feature amount is a 256th-order bit matrix, which is 8K.
Equivalent to bytes. Therefore, it cannot be said that a database in which the data for one property is about 1 Kbyte is an efficient system. Therefore, it is preferable to reduce the capacity of the structure file 40 by providing the data compression means S7 as described above to perform data compression.

FIG. 5 shows an example of the data compression method. In this example, the property name 40a (identification code) whose element value is 1 is stored as a 1-byte / case data string for each element of the 256th-order neighborhood feature amount matrix. Therefore, the property name whose element value is 0 is excluded as unnecessary data.

When the number of properties is 255 or more, the property name 40a cannot be represented by 1 byte, so only the lower 1 byte is stored. For example, when the number of properties is 10,000, the property name is represented by 2 bytes, but the lower 1 byte is used. Then, every time the property name code exceeds 255, the marker 40b is inserted into the data string.

At the time of search, the data string of the structure file corresponding to each of the neighborhood feature amounts of the search key is taken out, and the appearance frequency table for each property name is created. At this time, the marker 40
Add 255 to the property name code every time it exceeds b. The evaluation result list 70 of FIG. 2 is obtained based on the appearance frequency table created in this way.

When the data string of the property name code is, for example, more than half of all properties, the neighboring feature amount matrix element may be regarded as common to each property and the element may be deleted.

In the above embodiment, the normalizing means S1,
Learning means S2, normalization means S3, search means S4, evaluation result output means S5, sorting means S6, data compression means S7.
Can be configured by a computer program, but dedicated hardware may be configured by using a logic circuit element.

[0048]

As compared with the conventional general compression encoding method of the present invention, the amount of calculation can be greatly saved, so that compression and decoding can be performed at high speed. In addition, since a parameter for the entire data, such as a statistic, is not required, it is possible to easily add or delete the data.

[Brief description of drawings]

FIG. 1 is a data flow diagram of an embodiment of a compression decoding system according to the present invention.

FIG. 2 is a data flow diagram of a database search system to which the compression decoding system according to the present invention is applied.

FIG. 3 is a diagram showing quantization of neighborhood information.

FIG. 4 is a diagram showing a stored information structure.

FIG. 5 is a data configuration diagram of a compressed neighborhood feature amount.

[Explanation of symbols]

 10 search target 12 division unit 14 quotient memory comparison unit 16 compressed sequence D2 processing unit 18 storage unit 20 normalized data 22 reading unit 24 bias storage unit 26 addition unit 28 restored sequence D3 holding unit 30 neighborhood feature matrix 40 structure file 50 search Key 60 Normalized key 70 Evaluation result list 80 Sorted list S1 Normalization means S2 Learning means S3 Normalization means S4 Search means S5 Evaluation result output means S6 Sorting means S7 Data compression means

Claims (4)

[Claims] 1. In a compression and decoding system for integer sequence data arranged in ascending order, division means for dividing integer sequence data arranged in ascending order, and an old quotient obtained by the division means are already stored. A quotient and a quotient, and outputs the difference between these quotients when the obtained quotient is greater than the old quotient; and when the quotient memory comparison means outputs the quotient difference Storage means for storing the remainder obtained by the dividing means together with the quotient difference, and for storing only the remainder obtained by the dividing means when the quotient storage comparing means does not output the quotient difference, A decoding and decoding system for decoding original integer sequence data from the quotient difference and the remainder data stored in the storage means. 2. The ascending integer of claim 1, wherein the quotient memory comparing means outputs the difference between the quotients together with a mark indicating a carry when the obtained quotient is larger than the old quotient. Column data compression and decoding system. 3. A storage unit that stores the neighborhood feature amount for each search target property, and the degree of matching between the search key neighborhood feature amount and the search target neighborhood feature amount is determined for each property, and the property number is matched. The ascending integer sequence data compression and decoding system according to claim 1, wherein the compression and decoding system is used for a database search comprising a search means for outputting in descending order of degree. 4. The quantization amount x for the j-th data string C _{i, j} of the i-th property to be searched and k data strings C _{i, j + 1,} C _{i, j + 2, in the} vicinity thereof _{. ..,} C _{i, j + k} quantized amount y and x = f (C _{i, j} ) y = g (C _{i, j} , C _{i, j + 1,} C _{i, j + 2, ..,} C _{i, j + k} ), and is used for a database search for storing the serial number i of the property at the position of the storage means determined based on the obtained x and y values. 4. A compression and decoding system for ascending integer sequence data according to claim 3.