CN113890541A - Data compression method and system, computer readable storage medium - Google Patents
Data compression method and system, computer readable storage medium Download PDFInfo
- Publication number
- CN113890541A CN113890541A CN202111123996.1A CN202111123996A CN113890541A CN 113890541 A CN113890541 A CN 113890541A CN 202111123996 A CN202111123996 A CN 202111123996A CN 113890541 A CN113890541 A CN 113890541A
- Authority
- CN
- China
- Prior art keywords
- data
- frame
- comparison
- sampling time
- compression
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000013144 data compression Methods 0.000 title claims abstract description 36
- 238000005070 sampling Methods 0.000 claims abstract description 77
- 238000007906 compression Methods 0.000 claims abstract description 67
- 230000006835 compression Effects 0.000 claims abstract description 65
- 238000012545 processing Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 5
- 230000008859 change Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 4
- 101100114365 Caenorhabditis elegans col-8 gene Proteins 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 101100328518 Caenorhabditis elegans cnt-1 gene Proteins 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
- H03M7/3084—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction using adaptive string matching, e.g. the Lempel-Ziv method
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The invention discloses a data compression method and system and a computer readable storage medium. The data compression method comprises the following steps: dividing a frame of data corresponding to each sampling time into a plurality of segments of data which can be arranged into a plurality of rows and a plurality of columns, wherein each segment of data can be uniquely identified by the sampling time, the row number and the column number; setting comparison frame data; reading a frame of data as a current frame according to the sequence of the sampling time, and comparing each section of data of the current frame with each section of data of a row and a column corresponding to the comparison frame; if the corresponding segment data of the current frame and the comparison frame are different, outputting the corresponding segment data and the data distance of the corresponding segment data relative to the comparison frame, and taking the current frame as comparison frame data to continue the comparison of the next frame until all the frame data are processed; if the data of the current frame and the comparison frame are the same, the comparison frame keeps unchanged and continues the comparison of the next frame until all the frame data are processed. The invention has small compression ratio and simple and easy-to-implement coding mode.
Description
Technical Field
The present invention relates to the field of data compression technologies, and in particular, to a data compression method and a data compression system for reducing data changes.
Background
In order to reduce the storage space of the data or the transmission size of the data, many methods for compressing the data have appeared in the prior art.
Taking simulation verification of a design on an FPGA platform as an example, a large number of logic signal values need to be sampled for storage during simulation verification, and waveform display software reads the data to generate waveforms of corresponding signals. Since the data size of these logic signal values is large, a large amount of storage space is required to directly store the original sample data, and therefore, it is necessary to compress and store these data.
How to adopt a compression method with a small compression rate for the data with large data volume is a technical problem to be solved urgently.
Disclosure of Invention
The invention provides a data compression method and system and a computer readable storage medium, aiming at solving the technical problem of how to compress large-data-volume data in the prior art.
The data compression method provided by the invention comprises the following steps:
step 1, dividing a frame of data corresponding to each sampling time into a plurality of data segments which can be arranged into a plurality of rows and a plurality of columns, wherein each data segment can be uniquely identified by the sampling time, the row number and the column number;
step 2, setting comparison frame data;
step 3, reading a frame of data as a current frame according to the sequence of the sampling time, and comparing each section of data of the current frame with each section of data of a row and a column corresponding to the comparison frame;
step 4, if the segment data corresponding to the current frame is different from the segment data corresponding to the comparison frame, outputting all the segment data different from the segment data corresponding to the comparison frame and the data distance relative to the comparison frame of the current frame, and returning the current frame as the comparison frame data to the step 3 for continuously comparing until all the frame data are processed;
and 5, if the data of the current frame and the data of the comparison frame are the same, the comparison frame is kept unchanged, and the step 3 is returned to continue to carry out comparison until all the frame data are processed.
Further, outputting all the segment data of the current frame different from the segment data corresponding to the comparison frame and the data distance relative to the comparison frame comprises the following elements: the sampling time difference of the current frame relative to the comparison frame, the numerical encoding of all the segment data different from the comparison frame, the row encoding, and the column encoding.
Further, the numerical code, the row code and the column code of all the section data different from the section data corresponding to the comparison frame are represented by hexadecimal, wherein the high four bits are used for identifying the type of the element, and the low four bits are the value of the element.
Further, when the sampling time difference is smaller than 32, the sampling time difference is represented by hexadecimal, the upper 4bits of the sampling time difference are used for identifying the sampling time difference smaller than 32, and the lower four bits are values of the sampling time difference.
Further, when the sampling time difference is greater than or equal to 32, the method includes:
converting the sampling time difference into binary data;
dividing the binary data into at least one section from low order to high order according to a first preset length, wherein the first preset length is less than or equal to 8;
converting each section of binary data into hexadecimal;
the high 4bits of the first hexadecimal data of the sampling time difference are used for identifying the sampling time difference larger than 32, the low 4bits are used for identifying the number of the segments of the binary data which are divided, and the second hexadecimal and above hexadecimal are the corresponding hexadecimal numerical values of each segment of the binary data.
Further, the sampling time is represented by N, wherein N is a natural number which is more than or equal to 1;
further, each frame data is divided into a plurality of lines by 256 bits.
Furthermore, each row of data is divided into a plurality of columns according to 4 bits.
Further, the comparison frame data is the initial value defined by the user, or the comparison frame data is the first frame data arranged according to the sequence of the sampling time.
Further, the data is waveform data.
Further, the data compression method of the present invention further includes:
step 6, distributing the data output in the step 4 to a plurality of LZ compression units according to a second preset length for parallel processing;
step 7, performing two-stage compression on the data output in the step 4 by a plurality of LZ compression units;
and 8, splicing the data output by the multiple LZ compression units to form complete compressed data.
The data compression system provided by the invention comprises:
the data input and unpacking module is used for receiving the signal data obtained by sampling and unpacking the data according to the format of the data; dividing a frame of data corresponding to each sampling time into a plurality of segments of data which can be arranged into a plurality of rows and a plurality of columns, wherein each segment of data can be uniquely identified by the sampling time, the row number and the column number;
the first-stage compression module is used for setting comparison frame data, reading one frame data as a current frame according to the sequence of sampling time, comparing each section of data of the current frame with each section of data of a row corresponding to the comparison frame, if the section of data corresponding to the current frame is different from the section of data corresponding to the comparison frame, outputting all section data of the current frame different from the section of data corresponding to the comparison frame and the data distance of the section data relative to the comparison frame, and continuously comparing the current frame as the comparison frame data with the next frame data until all the frame data are processed; if the data of the current frame and the comparison frame are the same, the comparison frame is kept unchanged and is continuously compared with the next frame data until all the frame data are processed.
The data compression system of the present invention further comprises:
the data distribution module distributes the data output by the primary compression module;
the two-stage compression module comprises a plurality of LZ compression units, and the plurality of LZ compression units perform parallel processing on the data distributed by the data distribution module and perform secondary compression by adopting an LZ algorithm;
and the data merging and outputting module is used for splicing the data output by the multiple LZ modules to form complete compressed data.
The computer-readable storage medium provided by the present invention is used for storing a computer program, and when the computer program runs, the data compression method according to the above technical solution is executed.
Compared with the prior art, the invention has the advantages of simple coding mode, easy realization by using a logic circuit and realization of very high data processing bandwidth. The data compression method and the data compression system are particularly suitable for the situation that the data change of each sampling time sample is less than one fourth of the total data amount, and the waveform data is taken as an example and has the characteristic.
Drawings
The invention is described in detail below with reference to examples and figures, in which:
fig. 1 is a schematic structural diagram of an embodiment of the present invention.
FIG. 2 is a flow chart of an embodiment of the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantageous effects to be solved by the present invention more clearly apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Thus, a feature indicated in this specification will serve to explain one of the features of one embodiment of the invention, and does not imply that every embodiment of the invention must have the stated feature. Further, it should be noted that this specification describes many features. Although some features may be combined to show a possible system design, these features may also be used in other combinations not explicitly described. Thus, the combinations illustrated are not intended to be limiting unless otherwise specified.
The principles of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.
As shown in fig. 1, the data compression system of the present invention may include only data input and unpacking modules and a first-stage compression module in one embodiment. When the sampled data is waveform data, in a preferred embodiment, the data compression system of the present invention includes a data distribution module, a secondary compression module, and a data merging output module in addition to the data input and unpacking module and the primary compression module.
And a data input and unpacking module, which samples signal data by a data sampling module (not shown in the figure) and sends the signal data to the data input and unpacking module, and the data input and unpacking module unpacks the data according to a data format to form frame data capable of being parallel and sends the frame data to the first-stage compression module. The data input and unpacking module divides one frame of data corresponding to each sampling time into a plurality of data segments which can be arranged into a plurality of rows and a plurality of columns, and each data segment can be uniquely identified by the sampling time, the row number and the column number. The data-using module may not be a module of the data compression system, but may also be a module of the data compression system. Here, the data input and unpacking module unpacks the data according to the format of the data, and the format of the data may be various, for example, in one embodiment, the format of the data includes a frame header, a data content, and a CRC frame portion, and when compressing, the frame header and the CRC frame portion are removed, and only the data content portion is analyzed and compressed. In a further embodiment, the frame header can also contain some information needed for compression, for example, the lower 48bits of the frame header is timestamp information (timestamp information), indicating the starting timestamp value ts of 32 frames of data in the packet. The 48-55 bits are mode information, which represents different meanings when the mode information is different values. If the data frame is 0, 1 timestamp in each 1 frame of the data frame in the packet, and 32 timestamps of data are shared, which respectively correspond to ts, ts +1, ts +2, …, and ts +31, and the data bit width of each timestamp is 256 bits; when the data frame number is 1, 1 timestamp in each 2 frames of the data frame in the packet, and 16 timestamps of data are shared, wherein the data correspond to ts, ts +1, ts +2, …, ts +15, and the data bit width of each timestamp is 512 bits; when the data frame number is 2, 1 timestamp in each 4 frames of the data frame in the packet, and the total data of 8 timestamps corresponds to ts, ts +1, ts +2, … and ts +7 respectively, and the data bit width of each timestamp is 1024 bits. When the number is 3, the data frame in the packet has 1 timestamp per 8 frames, and the total number of data of 4 timestamps is respectively corresponding to ts, ts +1, ts +2 and ts +3, and the data bit width of each timestamp is 2048 bits; when the number is 4, the data frame in the packet has 1 timestamp per 16 frames and has 2 timestamps, which respectively correspond to ts and ts +1, and the data bit width of each timestamp is 4096 bits; when the data frame number is 5, 1 timestamp in each 32 frames of the data frame in the packet, the total number of data of 1 timestamp, corresponding to ts, and the data bit width of each timestamp is 8192 bits.
And the primary compression module compares the received changes of the two frames of data before and after, and only encodes and packages the changed parts of the data according to a specific format, thereby achieving the purpose of compressing the data.
The data after the first-level compression module is compressed are output to the data distribution module, the data distribution module distributes the data output by the first-level compression module to each LZ compression unit of the second-level compression module according to a specific length, a plurality of LZ compression units with lower input bandwidth can process the input data with larger bandwidth in parallel, the data compressed by the LZ compression units are output to the data combination output module, the data combination output module integrates and splices the data output by the second-level compression module again to form complete compressed data, and therefore the whole compression process is completed.
The first-stage compression module further comprises a frame data bit change detection unit and a coding and packaging processing unit, the former unit divides frame data into sections according to sampling time sequence and comparison frames, each section of data is compared according to bit one-to-one correspondence, the section of data with the change of the number value is output, the section of data without the change is not output, and the latter unit codes and packages the changed section of data to finally form compressed data. When the compressed data is waveform data, the first-stage compression module mainly compresses signal data with less or no change in the waveform data.
The second-stage compression module is provided with a plurality of LZ compression units for processing data to be compressed in parallel. The LZ compression algorithm adopted by the LZ compression unit realizes compression by using a dictionary mode, so that the LZ compression unit mainly compresses data with periodic change in waveform data, and can achieve the effects of small compression ratio and high compression efficiency.
As shown in fig. 2, the main part of the data compression method of the present invention is a difference compression method. A large amount of data with less change is compressed by a difference compression method, so that a smaller compression ratio is achieved. The compression process of the difference compression method is mainly completed by the first-stage compression module.
In one embodiment, the data input and unpacking module divides one frame of data corresponding to each sampling time into a plurality of segments of data which can be arranged in a plurality of rows and a plurality of columns, and each segment of data can be uniquely identified by the sampling time, the row number and the column number.
The first-stage compression module needs to set comparison frame data first. The initial value of the comparison frame data may be customized by the user, for example, the user sets a dummy frame with all bits being 0. The comparison frame data may also be the first frame data of the sample data.
After the comparison frame data is set, reading a frame data as a current frame according to the sequence of the sampling time, for example, when the comparison frame data is a virtual frame, the current frame is the first frame data of the sampling data, and if the comparison frame is the first frame data of the sampling data, the current frame is the second needle data of the sampling data.
And the first-stage compression module compares each section of data of the current frame with each section of data of the row and column corresponding to the comparison frame, outputs all section data of the current frame different from the section data corresponding to the comparison frame and the data distance relative to the comparison frame if the section data corresponding to the current frame is different from the section data corresponding to the comparison frame, and continuously compares the current frame serving as comparison frame data with the next frame data until all the frame data are processed. If the data of the current frame and the comparison frame are the same, the comparison frame is kept unchanged and is continuously compared with the next frame data until all the frame data are processed. For example, if the first frame data and the tenth frame data are identical data, the frame data is compared with the first frame data when the eleventh frame data is compared with the first frame data, and if the eleventh frame data is different from the first frame data, the data distance between the data of the different part and the first frame data is output, and then the comparison frame becomes the eleventh frame data.
Outputting all the section data of the current frame different from the section data corresponding to the comparison frame and the data distance relative to the comparison frame, wherein the section data comprises the following elements: the sampling time difference of the current frame relative to the comparison frame, the numerical encoding of all the segment data different from the comparison frame, the row encoding, and the column encoding. The following describes the numerical encoding, navigation encoding, column encoding, and sampling time difference of the segment data in detail by taking the waveform data as an example.
Each clock beat of the original waveform sampling data has one frame of data, and the bit width of the data is assumed to be M bits, that is, the corresponding M signals are M sampling data in the clock beat. Each clock beat may be referred to as a time stamp (timestamp) of one sample time for one frame of sample data. Therefore, the original data of the waveform is composed of a plurality of data frames with the width of M bits of timestamp.
Assuming that sampling data with the width of M bits can be divided into N rows according to 256bits, N = M/256, the decimal part is only ignored, the data of each row can be called the data of a certain row under a timestamp, and the row number of the Nth row of data is the row N. Each row of data with a width of 256bits may be further divided into 64 columns by 4bits, and the number of each column may be referred to as col, for example, column 0 may be referred to as col 0. Thus, the sample data with the width of Mbits is divided into the segment data which can be arranged into rows and columns, each segment data is 4bits, and thus, certain 4bits data of each time map can be expressed as time map t row n col y.
The data of 4bits section divided into rows and columns needs to be converted into 8bits, the conversion method is that the high 4bits are 0, the value of the numerical code is represented, namely the high 4bits mark the element type, the low 4bits are effective data, namely the specific value of the numerical code, and the numerical code, the row code and the column code of the section data are represented by hexadecimal system. For example, 0xa may be changed to 0x0a, where 0x is a hexadecimal notation.
A 4-bits segment data corresponds to a row code (row n code) of 0x80+ n, n ranges from 0 to 63, so this method supports a maximum M value of 64 × 256bits =16384 bits. The upper four bits 8 identify the element type row code, and 0+ n is the valid data, i.e. the specific value of the row code.
The column code (col y code) corresponding to 4bits data is 0x40+ y, y ranges from 0 to 63, the high four bits 4 identify element type column codes, and 0+ y is effective data, namely the specific value of the column code.
There are two cases of coding of a sampling time difference m (delta time estimate m) corresponding to 4bits of segment data, taking the sampling time as N as an example, where N is a natural number greater than or equal to N, that is, the sampling time is represented as 1, 2, 3, … ….
The first case is when m < 32: the code is 0xc0+ m. When the sampling time difference is smaller than 32, the sampling time difference is expressed by hexadecimal, the high 4bits c of the sampling time difference are used for identifying the sampling time difference smaller than 32, and the low four bits are the values of the sampling time difference.
The second case is when m ≧ 32: the code is 0xe0+ length, m 6:0, m 7:13 …, m …. The length ranges from 1 to 15, the length indicates a few 16 bytes later, namely the length of the delta timestamp, the numerical value of the byte after the length, the upper two bits of each byte are 0, and the lower 6bits represent the numerical value of the delta timestamp. The specific treatment process comprises the following steps:
the sampling time difference is first converted into binary data, for example, when the sampling time difference m is 100000, the corresponding binary is 0b0000_0001_1000_0110_1010_ 0000.
According to a first preset length, binary data are divided into at least one section from low order to high order, and the first preset length is less than or equal to 8. In one embodiment, the first preset length is 6bits, and then the first 6bits from back to front are 10_0000, which is converted into 8bits with hexadecimal representation, which is 0x20, the second 6bits are 0110_10, which is converted into 8bits with hexadecimal representation, specifically 0x15, the third 6bits are 01_1000, which is converted into 8bits with hexadecimal representation, specifically 0x18, so that the length can be determined as 3, and the sampling time difference in the sampling distance is output as 0xe3,0x20,0x15, and 0x 18.
In the above process, the high 4bits of the first hexadecimal data of the sampling time difference are used for identifying the sampling time difference larger than 32, the low 4bits are used for identifying the number of the divided segments of the binary data, and the second and above hexadecimal are the corresponding hexadecimal values of each segment of the binary data.
The following describes a process of comparing each frame data with the comparison frame data by the first-stage compression module by using a specific example.
When one frame data and the comparison frame data are compared with each other by segment data of which the number of 4bits is a group, if all the 4bits are the same, the current frame data does not need to be coded and output, and if the 4bits are different, the different 4bits of segment data are output according to the coding mode described in the foregoing, wherein the same row code is used for the change data in the same row, and each 4bits of segment data has a col code.
Each pair compares one frame data, the variable ts _ cnt representing the current frame is added with 1, if the difference between the current frame data and the comparison frame data is detected, the variable pre _ ts _ cnt representing the comparison frame stores the value of ts _ cnt-1, and the sampling time difference delta is timestamp = ts _ cnt-pre _ ts _ cnt; if they are the same, the value of pre _ ts _ cnt remains unchanged.
All the changed 4bits data under the current time frame share one delt time frame code, that is, a plurality of segments of data of one sampling time share the code of the same sampling time difference.
For example, the values of 5 pieces of data, i.e., col 0, col 5, col 8 of row 0, col 7 of row1, col 63, are changed compared with the data at the corresponding positions of the comparison frame, and if the sampling time difference delta is 1, the output is 0xc1, 0x80 value 00 x40 value 50 x45 value 80 x48, and 0x81 value 70 x47 value 630 x 7F. Where 0xc1 is the coding of the sampling time difference, 0x80 is the row coding of row 0, value 00 x40 value 50 x45 value 80 x48 is the column coding of col 0, col 5, col 8 of row 0,0x 81 is the row coding of row1, value 70 x47 value 630 x7F is the column coding of col 7, col 63 of row 1. If the sampling time difference delta timestamp is 100000, the output is 0xe 30 x 200 x 150 x18, 0x80 value 00 x40 value 50 x45 value 80 x48, 0x81 value 70 x47 value 630 x 7F.
When the data is waveform data, the data compression method of the present invention may further include a two-stage compression step.
And distributing the data output by the first-stage compression module to a plurality of LZ compression units according to a second preset length for parallel processing, performing second-stage compression on the data output by the first-stage compression module by the plurality of LZ compression units, and finally splicing the data output by the plurality of LZ compression units to form complete compressed data.
The invention carries out two-stage lossless compression processing on the data in parallel, and has the advantages of small compression rate and high compression rate compared with the prior compression method.
The invention also protects a computer-readable storage medium for storing a computer program which, when run by a computer, performs the above-described data compression method of the invention.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (14)
1. A method of data compression, comprising:
step 1, dividing a frame of data corresponding to each sampling time into a plurality of data segments which can be arranged into a plurality of rows and a plurality of columns, wherein each data segment can be uniquely identified by the sampling time, the row number and the column number;
step 2, setting comparison frame data;
step 3, reading a frame of data as a current frame according to the sequence of the sampling time, and comparing each section of data of the current frame with each section of data of a row and a column corresponding to the comparison frame;
step 4, if the segment data corresponding to the current frame is different from the segment data corresponding to the comparison frame, outputting all the segment data different from the segment data corresponding to the comparison frame and the data distance relative to the comparison frame of the current frame, and returning the current frame as the comparison frame data to the step 3 for continuously comparing until all the frame data are processed;
and 5, if the data of the current frame and the data of the comparison frame are the same, the comparison frame is kept unchanged, and the step 3 is returned to continue to carry out comparison until all the frame data are processed.
2. The data compression method of claim 1, wherein outputting all the segment data of the current frame different from the segment data corresponding to the comparison frame and the data distance thereof with respect to the comparison frame comprises the following elements: the sampling time difference of the current frame relative to the comparison frame, the numerical encoding of all the segment data different from the comparison frame, the row encoding, and the column encoding.
3. The data compression method of claim 2, wherein the numerical codes, the row codes and the column codes of all the segment data different from the segment data corresponding to the comparison frame are represented in hexadecimal, wherein the upper four bits are used for identifying the kind of the element, and the lower four bits are the value of the element.
4. The data compression method as claimed in claim 2, wherein when the sampling time difference is less than 32, the sampling time difference is represented in hexadecimal, the upper 4bits of the sampling time difference are used to identify the sampling time difference less than 32, and the lower four bits are the values of the sampling time difference.
5. The data compression method as claimed in claim 2, wherein when the sampling time difference is greater than or equal to 32, the method comprises:
converting the sampling time difference into binary data;
dividing the binary data into at least one section from low order to high order according to a first preset length, wherein the first preset length is less than or equal to 8;
converting each section of binary data into hexadecimal;
the high 4bits of the first hexadecimal data of the sampling time difference are used for identifying the sampling time difference larger than 32, the low 4bits are used for identifying the number of the segments of the binary data which are divided, and the second hexadecimal and above hexadecimal are the corresponding hexadecimal numerical values of each segment of the binary data.
6. The data compression method as claimed in claim 1, wherein the sampling time is represented by N, N being a natural number equal to or greater than 1.
7. The data compression method of claim 1, wherein each frame data is divided into a plurality of lines by 256bits for one line.
8. A method of data compression as claimed in claim 1 in which each row of data is divided into columns by 4bits for one column.
9. The data compression method of claim 1, wherein the comparison frame data is an initial value defined by a user, or the comparison frame data is a first frame data arranged according to a sequence of sampling times.
10. The data compression method of claim 1, wherein the data is waveform data.
11. The data compression method of any one of claims 1 to 10, further comprising:
step 6, distributing the data output in the step 4 to a plurality of LZ compression units according to a second preset length for parallel processing;
step 7, performing two-stage compression on the data output in the step 4 by a plurality of LZ compression units;
and 8, splicing the data output by the multiple LZ compression units to form complete compressed data.
12. A data compression system, comprising:
the data input and unpacking module is used for receiving the signal data obtained by sampling and unpacking the data according to the format of the data; dividing a frame of data corresponding to each sampling time into a plurality of segments of data which can be arranged into a plurality of rows and a plurality of columns, wherein each segment of data can be uniquely identified by the sampling time, the row number and the column number;
the first-stage compression module is used for setting comparison frame data, reading one frame data as a current frame according to the sequence of sampling time, comparing each section of data of the current frame with each section of data of a row corresponding to the comparison frame, if the section of data corresponding to the current frame is different from the section of data corresponding to the comparison frame, outputting all section data of the current frame different from the section of data corresponding to the comparison frame and the data distance of the section data relative to the comparison frame, and continuously comparing the current frame as the comparison frame data with the next frame data until all the frame data are processed; if the data of the current frame and the comparison frame are the same, the comparison frame is kept unchanged and is continuously compared with the next frame data until all the frame data are processed.
13. The data compression system of claim 12, further comprising:
the data distribution module distributes the data output by the primary compression module;
the two-stage compression module comprises a plurality of LZ compression units, and the plurality of LZ compression units perform parallel processing on the data distributed by the data distribution module and perform secondary compression by adopting an LZ algorithm;
and the data merging and outputting module is used for splicing the data output by the multiple LZ modules to form complete compressed data.
14. A computer-readable storage medium for storing a computer program, characterized in that it executes a data compression method according to any one of claims 1 to 11 when the computer program is run.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111123996.1A CN113890541A (en) | 2021-09-24 | 2021-09-24 | Data compression method and system, computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111123996.1A CN113890541A (en) | 2021-09-24 | 2021-09-24 | Data compression method and system, computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113890541A true CN113890541A (en) | 2022-01-04 |
Family
ID=79006468
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111123996.1A Pending CN113890541A (en) | 2021-09-24 | 2021-09-24 | Data compression method and system, computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113890541A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0123456A2 (en) * | 1983-03-28 | 1984-10-31 | Compression Labs, Inc. | A combined intraframe and interframe transform coding method |
| JP2000293200A (en) * | 1999-04-12 | 2000-10-20 | Matsushita Electric Ind Co Ltd | Audio compression coding method |
| JP2002354270A (en) * | 2001-05-29 | 2002-12-06 | Nec Viewtechnology Ltd | Still image data compression device, its method and recording medium |
| JP2005122034A (en) * | 2003-10-20 | 2005-05-12 | Fujitsu Ten Ltd | Audio data compression method |
| CN101217655A (en) * | 2008-01-04 | 2008-07-09 | 广东威创视讯科技股份有限公司 | Data compressing method |
| CN102270262A (en) * | 2011-08-23 | 2011-12-07 | 复旦大学 | Method and device for compressing and decompressing analog waveform of integrated circuit |
| CN105049055A (en) * | 2015-06-30 | 2015-11-11 | 郑州宇通客车股份有限公司 | Data compression method and data decompression method |
| US20170017407A1 (en) * | 2013-08-19 | 2017-01-19 | Huawei Technologies Co., Ltd. | Data object processing method and apparatus |
-
2021
- 2021-09-24 CN CN202111123996.1A patent/CN113890541A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0123456A2 (en) * | 1983-03-28 | 1984-10-31 | Compression Labs, Inc. | A combined intraframe and interframe transform coding method |
| JP2000293200A (en) * | 1999-04-12 | 2000-10-20 | Matsushita Electric Ind Co Ltd | Audio compression coding method |
| JP2002354270A (en) * | 2001-05-29 | 2002-12-06 | Nec Viewtechnology Ltd | Still image data compression device, its method and recording medium |
| JP2005122034A (en) * | 2003-10-20 | 2005-05-12 | Fujitsu Ten Ltd | Audio data compression method |
| CN101217655A (en) * | 2008-01-04 | 2008-07-09 | 广东威创视讯科技股份有限公司 | Data compressing method |
| CN102270262A (en) * | 2011-08-23 | 2011-12-07 | 复旦大学 | Method and device for compressing and decompressing analog waveform of integrated circuit |
| US20170017407A1 (en) * | 2013-08-19 | 2017-01-19 | Huawei Technologies Co., Ltd. | Data object processing method and apparatus |
| CN105049055A (en) * | 2015-06-30 | 2015-11-11 | 郑州宇通客车股份有限公司 | Data compression method and data decompression method |
Non-Patent Citations (2)
| Title |
|---|
| VAN LONG DO: "Compression of wideband IQ samples using linear predictive coding and frame of reference", 《2017 IEEE 2ND INTERNATIONAL CONFERENCE ON SIGNAL AND IMAGE PROCESSING (ICSIP)》, 31 December 2017 (2017-12-31), pages 1 - 4 * |
| 马欣: "采用帧间联合编码的2kbps MELP语音压缩编码的研究与实现", 《图像 仿真 信息技术》, 31 December 2002 (2002-12-31), pages 291 - 294 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8164490B2 (en) | Data compression/decompression method and computer readable storage medium storing compression/decompression program | |
| EP1744461A2 (en) | Data compression | |
| CN106849956B (en) | Compression method, decompression method, device and data processing system | |
| CN115276666B (en) | Efficient data transmission method for equipment training simulator | |
| CN104040899A (en) | Generating a code alphabet of symbols to generate codewords for words used with a program | |
| KR20120137235A (en) | Method and apparatus for compressing genetic data | |
| CN101751897A (en) | Lookup table compression and decompression method and relevant device thereof | |
| CN113312325B (en) | Track data transmission method, device, equipment and storage medium | |
| EP1266455A1 (en) | Method and apparatus for optimized lossless compression using a plurality of coders | |
| JPH03503707A (en) | A system for decoding statistically encoded digital data | |
| US20140139858A1 (en) | Data processing device and data processing method | |
| CN112290953B (en) | Array encoding device and method, array decoding device and method for multi-channel data stream | |
| CN113890541A (en) | Data compression method and system, computer readable storage medium | |
| US10931303B1 (en) | Data processing system | |
| US20130110846A1 (en) | Data processing method and apparatus | |
| JP2006259937A (en) | Data collection device and data recovery device | |
| JP2536422B2 (en) | Data compression device and data decompression device | |
| CN112995560B (en) | Transmission method, device and equipment for parallel video signals and storage medium | |
| CN109525795B (en) | Method and device for converting logic picture into bmp picture | |
| US10938411B1 (en) | Compression and/or decompression of activation data | |
| CN114070470A (en) | Encoding and decoding method and device | |
| CN115250351A (en) | Compression method, decompression method and related products for image data | |
| TW202002527A (en) | Data staging compression transmission method and device for implementing method and electronic device | |
| CN112290952B (en) | Parallel data coding device and method | |
| CN110797082A (en) | Method and system for storing and reading gene sequencing data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |