CN110399372B

CN110399372B - Method for compressing and decompressing ROWID corresponding relation data

Info

Publication number: CN110399372B
Application number: CN201910485933.7A
Authority: CN
Inventors: 高志会; 李天伟; 林广生
Original assignee: Shanghai Yingfang Software Co Ltd
Current assignee: Shanghai Yingfang Software Co Ltd
Priority date: 2019-06-05
Filing date: 2019-06-05
Publication date: 2020-03-27
Anticipated expiration: 2039-06-05
Also published as: CN110399372A

Abstract

The invention provides a method for compressing and decompressing corresponding relation data of ROWID, which belongs to the field of synchronization of an ORACLE database of a computer, fully utilizes the characteristics of data in a mapping block, sets a flag byte FLG, judges whether SLOT is recorded, DBA is repeated, OBJD is repeated and ROWID is continuous one by one, realizes compression, and realizes decompression by judging whether the bit of the FLG is set; the algorithm is simple, the execution speed is high, and the ROWID mapping data with high continuity has high compression rate.

Description

Method for compressing and decompressing ROWID corresponding relation data

Technical Field

The invention belongs to the field of computer ORACLE database synchronization, and particularly relates to a method for compressing and decompressing ROWID corresponding relation data.

Background

The ORACLE database is a very widely used database, and each line of record stored in the ORACLE database has a unique identifier called a rodid, and the record can be queried by the rodid.

The ROWID is divided into three parts, OBJD, DBA and SLOT, for 10 bytes. OBJD occupies 4 bytes and is a unique number distributed to each table, a table partition or a sub-partition by an ORACLE database; DBA is the number of the data block, and occupies 4 bytes, and ORACLE stores the record on different data blocks; multiple records may be stored in a block, and the ROWIDs of all records in the same block, whose OBJD and DBA are the same, are numbered as SLOT, and take up to 2 bytes.

For convenience of description, the recorded ROWIDs of the source and target repositories are hereinafter referred to as a source and target ROWID, respectively.

When ORACLE database synchronization is performed, the record change of the source library can be correctly updated to the target library only by corresponding the record of the source library to the record of the target library one by one. One of the ways is to use ROWID mapping, that is, store the corresponding relationship between the source ROWID and the target ROWID, and when the update and deletion actions for the record are synchronized, firstly, query the target ROWID according to the ROWID of the source record, so as to correspond to the target record.

When storing the ROWID correspondence, the specific storage block is usually determined by OBJD, DBA of the source library, and then the source SLOT and the target ROWID are stored in each storage block, each record needs 12 bytes of 10+2, and if 1000 records exist in one block, the data amount reaches 12000 bytes. For convenience of description, the ROWID mapping data of each source library data chunk is hereinafter referred to as a mapping chunk.

The mapping block needs to occupy a certain space, but it has a large amount of same or similar data, the target SLOT is continuous many times, the target OBJD is usually the same value, and the target DBA is also easily repeated, so there is a large compression space on the storage, and the compression and decompression speed is fast in view of the frequent inquiry. The existing compression algorithms such as zip and minilzo are not fast enough, and the compression effect is not ideal.

Disclosure of Invention

The invention aims to provide a compression and decompression method of a mapping block of ROWID corresponding relation data, which can achieve a certain compression rate and can quickly compress and decompress.

In order to achieve the purpose, the technical scheme adopted by the invention is as follows:

a method for compressing ROWID corresponding relation data comprises the following steps:

for ROWID corresponding relation data, starting from a source SLOT 0, judging whether a record exists in the current source SLOT, counting the number of the subsequent SLOT which are continuously recorded, and outputting a mark byte FLG if the record does not exist; if yes, generating a new flag byte FLG, outputting a target SLOT, and updating the FLG according to whether the SLOT is smaller than 256;

judging whether the current target DBA is repeated with the previous DBA or not, if not, outputting a new DBA and recording; if yes, outputting the index of the DBA, and updating the FLG according to whether the index number of the DBA is less than 256;

judging whether the current target OBJD is repeated with the previous OBJD or not, and if not, outputting and recording a new OBJD; if yes, outputting the index of the OBJD, and updating the FLG according to whether the index number of the OBJD is smaller than 256;

judging whether a target ROWID exists subsequently and the current continuity exists, if so, outputting the continuous number, and updating the FLG according to whether the continuous number is smaller than 256;

and executing the steps until no ROWID record exists, and finishing the compression.

Further, each piece of compressed data starts with an FLG that contains 8 bits as follows:

bit 7: the source SLOT has no recording mark bit, and if the bit is set, the other 7 bits represent how many SLOTs are not recorded continuously;

bit 6: the target SLOT occupies 1 byte or 2 bytes;

bit 5: a new target DBA flag, set to indicate that a new DBA is encountered, the compressed data contains 4 bytes of DBA; otherwise, the compressed data contains the index of the DBA, which represents that the used DBA is encountered;

bit 4: whether the DBA index occupies 1 byte or 2 bytes;

bit 3: a new object OBJD flag, set to indicate that a new OBJD is encountered and that the compressed data contains 4 bytes of OBJD; otherwise, the compressed data contains the index of the OBJD, which represents that the used OBJD is encountered;

bit 2: whether the OBJD index occupies 1 byte or 2 bytes;

bit 1: indicating that the target ROWID is continuous, and the compressed data contains the number of continuous ROWIDs;

bit 0: the number of consecutive ROWIDs is 1 byte or 2 bytes.

Further, the number of the subsequent SLOTs which are not recorded continuously is counted to be not more than 127.

Further, the updating of the FLG according to whether SLOT is less than 256 means that if SLOT is less than 256, 1 byte is output, otherwise, 2 bytes are output, and FLG bit 6 is updated.

Further, the updating the FLG according to whether the index number of the DBA is smaller than 256 means that if the index number of the DBA is smaller than 256, 1 byte is output, otherwise, 2 bytes are output, and

FLG bits

5 and 4 are updated.

Further, the updating of the FLG according to whether the index number of the OBJD is less than 256 means that if the index number of the OBJD is less than 256, 1 byte is output, otherwise, 2 bytes are output, and

FLG bits

3 and 2 are updated.

Further, it is determined whether there is a target road and whether the criterion of the current continuity is that OBJD and DBA are the same as the current target road and the target SLOT is continuously incremented.

Further, the updating of the FLG according to whether the consecutive number is less than 256 means that if the consecutive number is less than 256, 1 byte is output, otherwise, 2 bytes are output, and

FLG bits

1 and 0 are updated.

Further, if there are also ROWID records, it is further determined whether there are records in the current source SLOT, and the subsequent operations are performed.

A method for decompressing ROWID corresponding relation data comprises the following steps:

for a data packet of ROWID corresponding relation data, taking a current mark byte FLG, if a bit 7 is set, taking a numerical value N of another 7 bits of the current mark byte FLG, marking continuous N source SLOT as no record, and repeating the step;

if bit 7 is not set, then 1 or 2 bytes of target SLOT is taken out according to the indication of bit 6;

if bit 5 of FLG is set, then take out target DBA of 4 bytes and record, otherwise take out DBA index number of 1 or 2 bytes according to the instruction of bit 4, then take out target DBA in DBA record with the index;

if bit 3 of FLG is set, taking out 4 bytes of object OBJD and recording, otherwise taking out 1 or 2 bytes of OBJD index number according to the indication of bit 4, and then taking out object OBJD in OBJD record by using the index;

if bit 1 of the FLG is set, 1 or 2 bytes are taken out according to the indication of bit 0 to serve as the number of continuous ROWIDs, and the corresponding number of continuous ROWIDs are generated;

and executing the steps until the uncompressing data does not exist, and completing the uncompressing.

The compression and decompression method of the invention fully utilizes the characteristics of the data in the mapping block, has simple algorithm and high execution speed, and has very high compression ratio for ROWID mapping data with high continuity.

Drawings

FIG. 1 is a block diagram of a ROWID.

Fig. 2 is an example of a ROWID within a block.

FIG. 3 is a flowchart of a method for compressing ROWID mapping data.

Fig. 4 is a flowchart of a method for decompressing data of the roild correspondence.

Fig. 5 is an example of data for a mapping block.

Fig. 6 is a compression result of the mapping block of fig. 5.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below.

The embodiment discloses a method for compressing data of a ROWID correspondence relationship, which specifically includes the following steps as shown in a flowchart shown in FIG. 3:

1) defining a flag byte, abbreviated as FLG, each segment of compressed data starts with one FLG, and 8 bits of the FLG have different purposes:

bit 7: the source SLOT has no recording mark bit, and if this bit is set, the other 7 bits represent how many SLOTs are not recorded consecutively.

Bit 6: the target SLOT takes 1 byte or 2 bytes.

Bit 5: a new target DBA flag, set to indicate that a new DBA is encountered, the compressed data contains 4 bytes of DBA; conversely, the compressed data contains an index of the DBA, representing that a used DBA was encountered.

Bit 4: the DBA index takes 1 byte or 2 bytes.

Bit 3: a new object OBJD flag, set to indicate that a new OBJD is encountered and that the compressed data contains 4 bytes of OBJD; instead, the compressed data contains an index to the used OBJD, representing that the used OBJD was encountered.

Bit 2: the OBJD index takes 1 byte or 2 bytes.

Bit 1: indicating that the target ROWID is consecutive, the compressed data contains the number of consecutive ROWIDs.

Bit 0: the number of consecutive ROWIDs is 1 byte or 2 bytes.

2) The process starts from the source SLOT 0.

3) If the current source SLOT has a record, entering the next step; otherwise, checking how many continuous SLOT are not recorded later, counting the number of SLOT not more than 127, and then outputting FLG (flash memory) to re-enter the step 3).

4) Generating a new FLG and outputting a target SLOT, if the SLOT is less than 256, outputting 1 byte, otherwise, outputting 2 bytes; FLG bit 6 is updated.

5) Judging whether the current target DBA is repeated with the previous DBA, if not, outputting a new DBA, and recording; if yes, outputting the index of the DBA, and outputting 1 byte when the index number is less than 256, otherwise, outputting 2 bytes;

FLG bits

5 and 4 are updated.

6) Judging whether the current target OBJD is repeated with the previous OBJD or not, if not, outputting a new OBJD and recording; if yes, outputting the index of the OBJD, and outputting 1 byte when the index number is less than 256, otherwise, outputting 2 bytes;

FLG bits

3 and 2 are updated.

7) And judging whether a target ROWID and the current continuity exist subsequently, wherein the standard is that the OBJD and the DBA are the same as the current target ROWID, and the target SLOT is continuously increased. If yes, outputting continuous number, wherein the continuous number is less than 256 to output 1 byte, is more than or equal to 256 to output 2 bytes, and updating

FLG bits

1 and 0. If not, then

FLG bits

1 and 0 are updated and the next step is performed

8) If the ROWID records still exist, repeating the step 3, otherwise, completing the compression.

The embodiment also discloses a method for decompressing the ROWID corresponding relation data, which comprises the following steps:

1) adopting the compression format defined by the compression method;

2) taking the current FLG, if the bit 7 is set, taking the numerical value N of the other 7 bits, marking the continuous N source SLOT as no record, and then repeating the step 2; if bit 7 is not set, then 1 or 2 bytes of target SLOT is taken out according to the indication of bit 6;

3) if bit 5 of FLG is set, taking out target DBA of 4 bytes and recording, otherwise, taking out DBA index number of 1 or 2 bytes according to the indication of bit 4, and then taking out target DBA in DBA record by using the index;

4) if bit 3 of FLG is set, taking out 4 bytes of object OBJD and recording, otherwise, taking out 1 or 2 bytes of OBJD index number according to the indication of bit 4, and then taking out object OBJD in OBJD record by using the index;

5) if bit 1 of the FLG is set, 1 or 2 bytes are taken out according to the indication of bit 0 to serve as the number of continuous ROWIDs, and the corresponding number of continuous ROWIDs are generated; if not, the next step is executed

6) If the data which is not decompressed exist, repeating the step 2), otherwise, decompressing and finishing.

The compression and decompression method is described below by taking a specific example of compression and decompression as an example, and fig. 3 is a typical mapping block used in this example, which is described in detail below.

The compression method comprises the following steps:

s1: source SLOT 0 has a record, target SLOT is 0,

output

0, 1 byte, FLG value 00000000.

S2: the target DBA has not appeared, FLG is updated to 00100000, 0x11BBC8B is output, 4 bytes, and the DBA is saved.

S3: target OBJD is not present, FLG is updated to 00101000, 26803 and 4 bytes are output, and the OBJD is saved

S4: the source SLOT 1 has a record but is not continuous with the source SLOT 0, and therefore, the source SLOT 0 process is completed.

S5: the target SLOT of source SLOT 1 is 500, the output is 500, 2 bytes, and the FLG value is 01000000.

S6: target DBA appeared,

output index number

0, 1 byte, FLG value 01000000.

S7: target OBJD appears, with

output index number

0, 1 byte, FLG value 01000000.

S8: the source SLOT 1 has no subsequent continuous ROWID and the process is complete.

S9:

sources SLOT

2 and 3 are not recorded. Two unrecorded SLOT output FLG values 10000010.

S10: the source SLOT 4 has a record, the target SLOT is 7, the output is 7, 1 bytes, and the FLG value is 00000000.

S11: the target DBA is not present, FLG is updated to 00100000, 0x10825DB is output, 4 bytes, and the DBA is saved.

S12: target OBJD is present,

output index

0, 1 byte, FLG value 00100000.

S13: the next 2 target ROWIDs are consecutive, the FLG value is updated to 00100010, and 2, 1 bytes are output.

S14: after the compression process is completed, the compression result is shown in fig. 4.

The decompression method comprises the following steps:

s1: take the 1 st FLG, according to the mark bit, take the target SLOT (0) of 1 byte, the target DBA (0x11BBC8B) of 4 bytes and the target OBJD (26803) of 4 bytes, and save DBA and OBJD.

S2: and taking the 2 nd FLG, and taking out a target SLOT (500) with 2 bytes according to the mark bits, wherein the target DBA and the OBJD are taken out according to the index, and the result is the same as that of the previous record.

S3: take the 3 rd FLG, set bit 7, and the remaining 7 bit value is 2, representing two unrecorded source SLOTs.

S4: the 4 th FLG is taken, and the 1-byte target SLOT (7) and the 4-byte target DBA (0x10825DB) are taken out according to the mark bit, and the target OBJD (26803) is taken out in an index mode. Setting FLG bit 1, representing that there are continuous target ROWIDs, taking out the continuous ROWID number (2) of 1 byte, and generating the mapping data of two source SLOTs, wherein the target SLOTs are respectively 8 and 9.

S5: and (5) finishing decompression processing.

A group of 100 rows of ROWID mapping data is used as a test sample, 6 rows of ROWID mapping in the sample are not continuous with other records, and the condition that the ROWID mapping is discontinuous due to a small quantity of deleting operations is simulated. The following table compares the inventive method to ZIP data after 100 million compression and decompression tests.

TABLE 1 methods of the invention and ZIP experimental data

Method of producing a composite material	Before compression (byte)	After compression (byte)	Compression time (millisecond)	Decompression time (millisecond)
					ZIP	1200	360	31944	18310
The method of the invention	1200	60	431	405

As can be seen from the above table, the compression and decompression method of the present invention makes full use of the characteristics of the data in the mapping block, has simple algorithm, fast execution speed and high compression rate for ROWID mapping data with high continuity.

The above embodiments are only intended to illustrate the technical solution of the present invention and not to limit the same, and a person skilled in the art can modify the technical solution of the present invention or substitute the same without departing from the spirit and scope of the present invention, and the scope of the present invention should be determined by the claims.

Claims

1. A method of compressing data corresponding to a rodid including a unique number OBJD assigned by an ORACLE database to each table, partition or sub-partition of a table, a data block number DBA storing records of the ORACLE database, and a number SLOT recorded in a data block, the method comprising the steps of:

2. The compression method as set forth in claim 1, wherein the number of subsequent consecutive unrecorded SLOTs is counted as not more than 127 SLOTs.

3. The compression method as set forth in claim 1, wherein the criterion for determining whether or not there is a target ROWID and the current continuation is whether OBJD and DBA are the same as the current target ROWID, and the target SLOT is continuously incremented.

4. The compression method as claimed in claim 1, wherein if there is also a ROWID record, it is further determined whether there is a record in the current source SLOT, and the subsequent operation is performed.

5. The compression method of claim 1, wherein each segment of compressed data begins with an FLG having 8 bits as follows:

bit 6: the target SLOT occupies 1 byte or 2 bytes;

bit 4: whether the DBA index occupies 1 byte or 2 bytes;

bit 2: whether the OBJD index occupies 1 byte or 2 bytes;

bit 0: the number of consecutive ROWIDs is 1 byte or 2 bytes.

6. The compression method as claimed in claim 5, wherein the updating FLG according to whether SLOT is less than 256 means that if SLOT is less than 256, 1 byte is output, otherwise 2 bytes are output, FLG bit 6 is updated.

7. The compression method as claimed in claim 5, wherein the updating the FLG according to whether the index number of the DBA is less than 256 means that if the index number of the DBA is less than 256, 1 byte is outputted, otherwise 2 bytes are outputted, and FLG bits 5 and 4 are updated.

8. The compression method of claim 5, wherein updating the FLG according to whether the index number of the OBJD is less than 256 means outputting 1 byte if the index number of the OBJD is less than 256, and otherwise outputting 2 bytes, updating FLG bits 3 and 2.

9. The compression method as claimed in claim 5, wherein said updating the FLG according to whether the consecutive number is less than 256 means that if the consecutive number is less than 256, 1 byte is outputted, otherwise 2 bytes are outputted, and FLG bits 1 and 0 are updated.

10. A method for decompressing data corresponding to a rodid including a unique number OBJD assigned by an ORACLE database to each table, partition or sub-partition of a table, a data block number DBA storing records of the ORACLE database, and a number SLOT recorded in a data block, the method comprising the steps of: