CN112104658B - Message compression method and system - Google Patents
Message compression method and system Download PDFInfo
- Publication number
- CN112104658B CN112104658B CN202010979885.XA CN202010979885A CN112104658B CN 112104658 B CN112104658 B CN 112104658B CN 202010979885 A CN202010979885 A CN 202010979885A CN 112104658 B CN112104658 B CN 112104658B
- Authority
- CN
- China
- Prior art keywords
- compression
- engine
- message
- cache
- learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000006835 compression Effects 0.000 title claims abstract description 114
- 238000007906 compression Methods 0.000 title claims abstract description 114
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000011156 evaluation Methods 0.000 claims abstract description 34
- 238000004590 computer program Methods 0.000 claims description 15
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000010586 diagram Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 4
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The invention belongs to the field of message compression, and provides a message compression method and a message compression system. The message compression method comprises the steps of initializing symbol units of a message to be compressed and respectively transmitting the symbol units to a learning engine and an accelerated compression engine; compressing the message to be compressed by using an accelerated compression engine, and feeding the compressed message data into a learning engine for learning so as to determine whether the symbol unit is optimal; feeding the results of the learning engine learning to an evaluation engine to determine whether the total length is the minimum; the symbol unit evaluation information is fed into an arbitrator to carry out arbitration selection, and the arbitrator determines the optimal compression symbol unit of the compression engine. The method can balance the cost of circuit resources and the cost of time, and saves the circuit resources while improving the compression speed.
Description
Technical Field
The invention belongs to the field of message compression, and particularly relates to a message compression method and a message compression system.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
LZ77 compression is a classic message compression processing method, which searches for matching substring from the look window in the look-ahead window and matches it to the maximum length matching substring, as shown in fig. 1. The inventor finds that the traditional message compression processing method is realized by computer software, the digital logic hardware operation time overhead (cycles/single matching) related to the compression processing is not fully considered, a specific hardware logic realization method is not given, and the balance between the circuit resource cost and the time cost cannot be achieved.
Disclosure of Invention
In order to solve the above problems, the present invention provides a method and a system for message compression, which can balance the cost of circuit resources and the cost of time, and save the circuit resources while increasing the compression speed.
In order to achieve the purpose, the invention adopts the following technical scheme:
a first aspect of the present invention provides a method for compressing a packet.
In one or more embodiments, a method for message compression includes:
initializing symbol units of a message to be compressed and respectively transmitting the symbol units to a learning engine and an accelerated compression engine;
compressing the message to be compressed by using an accelerated compression engine, and feeding the compressed message data into a learning engine for learning so as to determine whether the symbol unit is optimal;
feeding the results of the learning engine learning to an evaluation engine to determine whether the total length is the minimum;
the symbol unit evaluation information is fed into an arbitrator to carry out arbitration selection, and the arbitrator determines the optimal compression symbol unit of the compression engine.
A second aspect of the invention provides a message compression system.
In one or more embodiments, a message compression system includes a learning engine, an accelerated compression engine, an evaluation engine, and an arbiter;
the learning engine and the accelerated compression engine are used for receiving the initialized symbol unit of the message to be compressed;
the accelerating compression engine is used for compressing the message to be compressed and feeding the compressed message data into the learning engine;
the learning engine is used for learning the received message data to determine whether the symbol unit is optimal or not; the learned results are also fed to the evaluation engine;
an evaluation engine for evaluating the learned result to determine whether the overall length is the smallest; symbol unit evaluation information is also fed to the arbiter;
and the arbiter is used for arbitrating selection and determining the optimal compression symbol unit of the compression engine.
A third aspect of the invention provides a computer-readable storage medium.
A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the message compression method as set forth above.
A fourth aspect of the invention provides a computer apparatus.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the message compression method as described above when executing the program.
Compared with the prior art, the invention has the beneficial effects that:
the embodiment utilizes the accelerated compression engine to compress the message to be compressed, and feeds the compressed message data into the learning engine for learning so as to determine whether the symbol unit is optimal, thereby realizing the self-adaptive adjustment of the optimal matching level based on the pre-learning; feeding the results of the learning engine learning to an evaluation engine to determine whether the total length is the minimum; the symbol unit evaluation information is fed into the arbiter for arbitration selection, and the arbiter determines the optimal compression symbol unit of the compression engine, thereby improving the flexibility of message compression matching and the compression processing speed, and balancing the circuit resource cost and the time cost.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the invention and together with the description serve to explain the invention and not to limit the invention.
FIG. 1 is a flow of a conventional LZ77 compression algorithm;
FIG. 2 is a packet compression flow according to an embodiment of the present invention;
FIG. 3 is a diagram of multiple S-base pointer locations to be traversed for multiple ways in parallel, according to an embodiment of the present invention;
FIG. 4 shows an embodiment of the present invention that performs forward search and matches a maximum length substring based on a fixed S-base pointer and a fixed L-pointer and a forward temporary S-pointer and a temporary L-pointer;
FIG. 5 is a cache-based maximum match length prediction according to an embodiment of the present invention;
FIG. 6 is a multi-way S-base pointer position estimation candidate method according to an embodiment of the present invention.
Detailed Description
The invention is further described with reference to the following figures and examples.
It is to be understood that the following detailed description is exemplary and is intended to provide further explanation of the invention as claimed. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
Example one
Referring to fig. 2, the packet compression method of this embodiment includes:
step 1: and initializing the symbol unit of the message to be compressed and respectively transmitting the symbol unit to the learning engine and the accelerated compression engine.
Wherein, the symbol unit can be a single byte, a double byte or three bytes.
Step 2: and compressing the message to be compressed by using the accelerated compression engine, and feeding the compressed message data into a learning engine for learning so as to determine whether the symbol unit is optimal.
In the concrete implementation, in an accelerated compression engine, a method of parallel forward matching pins of multiple paths of S base fingers is adopted to accelerate the compression of a message to be compressed; wherein the S-base pointer is the starting position of the sub-string.
And the moving multipath of the sub-string matching window is parallel, so that the balance of hardware resource consumption and clock cycle consumption is realized.
Specifically, as shown in fig. 3, when the position of the L pointer is determined (i.e. the left boundary of the look-ahead window), the S pointer needs to be moved forward and backward, where the backward movement of the S pointer can be performed in parallel, there is no data dependency, and the existing structural dependency can be fully solved by using the multi-path parallel matching unit;
as shown in fig. 3, the L pointer is temporarily fixed to fully move the S pointer, the S-base pointer is a starting position of sub-string, and needs to traverse from a position of a symbol unit behind the L pointer to a left boundary, where the left boundary may be a left boundary of a search window or a left boundary of a message;
in fig. 3, the L pointer is fixed, and only after all the S pointers are searched, the L pointer is moved forward according to the maximum matching length of the search, and the offset of the forward movement is the most matching length + 1;
the left boundary of the S pointer may be the left boundary of the search window or the left boundary of the message, and the left boundary determination is needed;
the S pointer can be used for parallel searching and matching calculation along the descending pointer.
As shown in fig. 4, the largest matching sub-string is searched forward from each S-base pointer, and the temporary S pointer and the temporary L pointer need to be moved forward to obtain the largest-length matching sub-string;
the forward search of the multiple S-base pointers may be performed in parallel, that is, the multiple S-base pointers are operated simultaneously or divided into groups, and a group of temporary S pointers are advanced by using hardware matching logic processing resources, and of course, the temporary L pointers are moved synchronously to obtain all or a group of matching lengths, and the matching length of each S-base pointer is recorded;
the matching substring lengths obtained for all the S-base pointers are compared, leaving only the longest matching length, and if there is the same maximum matching length, that maximum matching length corresponds to the farthest S-base pointer position from the fixed L-pointer offset position.
It should be noted here that it is ensured that the temporary L pointer does not exceed the right boundary, and the right boundary may be the right boundary of the search window or the right boundary of the message.
In FIG. 4, sub-string (max) maximum match substring refers to: based on a fixed S-base pointer, an S temporary pointer is moved forward, an L pointer is also fixed, and a temporary L pointer is moved forward to search for the maximum matching length;
similar operations are performed for the rest of S-base pointers, and the parallel operations are processed, that is, different S-base pointers can advance in parallel and search at the same time, so as to increase the search speed and the compression rate.
The present embodiment adaptively adjusts the best matching level based on the pre-learning, i.e., the symbol unit is adjustable.
In order to increase the forward matching speed, in the embodiment, in the accelerated compression engine, the forward matching speed is increased based on the maximum matching length prediction method of the cache.
As shown in fig. 5, the process of accelerating the forward matching speed by the maximum matching length prediction method based on the cache is as follows:
firstly, when the cache comprises 2 sub-caches, one is a time cache, the other is a space cache, and when the LZ77 accelerates the processing of a compression engine, the prefix of the substring which is the maximum matching can be one or 2 symbol units and is used as a key value K, hash calculation is carried out to obtain a hash (K) which is used as the index of the cache, the data content after addressing corresponding to the cache index is the maximum matching length, and the cache is continuously filled until the cache is full;
when the cache stores a certain content, performing hit operation of the cache, if hit is detected, obtaining the predicted maximum matching length of the substring currently performed, if the matching length is larger than the existing length in the cache for the same hash (K) index, performing cache content, namely updating the maximum length, if the matched position offset exceeds a preset threshold value, updating in the time cache, otherwise, updating in the space cache.
If miss occurs to the cache, replacing the cache index without hit for a long time and the corresponding maximum matching length, and similarly, selecting the long-time-related cache or the near-time space-related cache according to the position offset at the time of maximum matching.
And step 3: the results of the learning engine learning are fed to an evaluation engine to determine if it has the smallest overall length.
The learning result of the learning engine includes output parameters, such as a position offset for maximum matching of different symbol units, a maximum matching length for different symbol units, an uncompressed symbol unit, and the like.
And 4, step 4: the symbol unit evaluation information is fed into an arbitrator to carry out arbitration selection, and the arbitrator determines the optimal compression symbol unit of the compression engine.
Wherein the symbol unit evaluation information comprises a symbol length evaluation and possibly a compression bounce warning.
In order to increase the compression speed, in the embodiment, in the accelerated compression engine, the candidates are eliminated by the multiple S-base pointer position measures, the exclusive or matching calculation is performed on the candidates, and the multiple S-base pointer parallel search results are output.
As shown in fig. 6, when the hit operation of the cache is successful, the maximum matching length L is obtained. Compressing the message data by an accelerated compression engine, performing parallel matching on a plurality of paths of S-base pointers to obtain a plurality of paths of positions, eliminating the plurality of paths of positions to obtain candidate positions, changing the matching length to be 2 times of the maximum matching length, judging whether substring suffixes are matched, if so, the matching length is 2L, and if not, the matching length is 2/3 xL; and finally, carrying out XOR exclusive or matching calculation on the candidates, ending the multi-path S-finger parallel search and outputting the search result.
The embodiment utilizes the accelerated compression engine to compress the message to be compressed, and feeds the compressed message data into the learning engine for learning so as to determine whether the symbol unit is optimal, thereby realizing the self-adaptive adjustment of the optimal matching level based on the pre-learning; feeding the results of the learning engine learning to an evaluation engine to determine whether the total length is the minimum; the symbol unit evaluation information is fed into the arbiter for arbitration selection, and the arbiter determines the optimal compression symbol unit of the compression engine, thereby improving the flexibility of message compression matching and the compression processing speed, and balancing the circuit resource cost and the time cost.
Example two
The embodiment provides a message compression system which comprises a learning engine, an accelerated compression engine, an evaluation engine and an arbitrator.
Specifically, the learning engine and the accelerated compression engine are both configured to receive the initialized symbol unit of the message to be compressed.
Wherein, the symbol unit can be a single byte, a double byte or three bytes.
And the accelerated compression engine is used for compressing the message to be compressed and feeding the compressed message data into the learning engine.
In the concrete implementation, in an accelerated compression engine, a method of parallel forward matching pins of multiple paths of S base fingers is adopted to accelerate the compression of a message to be compressed; wherein the S-base pointer is the starting position of the sub-string.
And the moving multipath of the sub-string matching window is parallel, so that the balance of hardware resource consumption and clock cycle consumption is realized.
Specifically, as shown in fig. 3, when the position of the L pointer is determined (i.e. the left boundary of the look-ahead window), the S pointer needs to be moved forward and backward, where the backward movement of the S pointer can be performed in parallel, there is no data dependency, and the existing structural dependency can be fully solved by using the multi-path parallel matching unit;
as shown in fig. 3, the L pointer is temporarily fixed to move the S pointer sufficiently, the S base pointer is a starting position of sub-string, and needs to traverse from a position of a symbol unit behind the L pointer to a left boundary, where the left boundary may be a left boundary of a search window or a left boundary of a packet;
in fig. 3, the L pointer is fixed, and only after all the S pointers are searched, the L pointer is moved forward according to the maximum matching length of the search, and the offset of the forward movement is the most matching length + 1;
the left boundary of the S pointer may be the left boundary of the search window or the left boundary of the message, and the left boundary determination is required;
the S pointer can be used for parallel searching and matching calculation along the descending pointer.
As shown in fig. 4, the largest matching sub-string is searched forward from each S-base pointer, and the temporary S pointer and the temporary L pointer need to be moved forward to obtain the largest-length matching sub-string;
the forward search of the multiple S-base pointers may be performed in parallel, that is, the multiple S-base pointers are operated simultaneously or divided into groups, and a group of temporary S pointers are advanced by using hardware matching logic processing resources, and of course, the temporary L pointers are moved synchronously to obtain all or a group of matching lengths, and the matching length of each S-base pointer is recorded;
the matching substring lengths obtained for all the S-base pointers are compared, leaving only the longest matching length, and if there is the same maximum matching length, that maximum matching length corresponds to the farthest S-base pointer position from the fixed L-pointer offset position.
It should be noted here that it is ensured that the temporary L pointer does not exceed the right boundary, and the right boundary may be the right boundary of the search window or the right boundary of the message.
In FIG. 4, sub-string (max) maximum match substring refers to: based on a fixed S-base pointer, an S temporary pointer is moved forward, an L pointer is also fixed, and a temporary L pointer is moved forward to search for the maximum matching length;
similar operations are performed for the rest of S-base pointers, and the parallel operations are processed, that is, different S-base pointers can advance in parallel and search at the same time, so as to increase the search speed and the compression rate.
The present embodiment adaptively adjusts the optimum matching level based on the pre-learning, i.e., the symbol unit is adjustable.
In order to increase the forward matching speed, in the embodiment, in the accelerated compression engine, the forward matching speed is increased based on the maximum matching length prediction method of the cache.
As shown in fig. 5, the process of accelerating the forward matching speed by the maximum matching length prediction method based on the cache is as follows:
firstly, when the cache comprises 2 sub-caches, one is a time cache, the other is a space cache, and when the LZ77 accelerates the processing of a compression engine, the prefix of the substring which is the maximum matching can be one or 2 symbol units and is used as a key value K, hash calculation is carried out to obtain a hash (K) which is used as the index of the cache, the data content after addressing corresponding to the cache index is the maximum matching length, and the cache is continuously filled until the cache is full;
when the cache stores a certain content, performing hit operation of the cache, if hit is detected, obtaining the predicted maximum matching length of the substring currently performed, if the matching length is larger than the existing length in the cache for the same hash (K) index, performing cache content, namely updating the maximum length, if the matched position offset exceeds a preset threshold value, updating in the time cache, otherwise, updating in the space cache.
If miss occurs to the cache, replacing the cache index without hit for a long time and the corresponding maximum matching length, and similarly, selecting the long-time-related cache or the near-time space-related cache according to the position offset at the time of maximum matching.
The learning engine is used for learning the received message data to determine whether the symbol unit is optimal or not; the learned results are also fed to the evaluation engine.
The learning result of the learning engine includes output parameters, such as a position offset for maximum matching of different symbol units, a maximum matching length for different symbol units, an uncompressed symbol unit, and the like.
An evaluation engine for evaluating the learned result to determine whether the overall length is the smallest; symbol unit evaluation information is also fed to the arbiter.
And the arbiter is used for arbitrating selection and determining the optimal compression symbol unit of the compression engine.
In order to increase the compression speed, in this embodiment, in the arbiter, the multiple paths of S-base pointer positions are used to eliminate the candidate method, and the candidates are subjected to xor matching calculation to output the multiple paths of S-base pointer parallel search results.
As shown in fig. 6, when the hit operation of the cache is successful, the maximum matching length L is obtained. Compressing the message data by an accelerated compression engine, performing parallel matching on a plurality of paths of S-base pointers to obtain a plurality of paths of positions, eliminating the plurality of paths of positions to obtain candidate positions, changing the matching length to be 2 times of the maximum matching length, judging whether substring suffixes are matched, if so, the matching length is 2L, and if not, the matching length is 2/3 xL; and finally, carrying out XOR exclusive or matching calculation on the candidates, ending the multi-path S-finger parallel search and outputting the search result.
Wherein the symbol unit evaluation information comprises a symbol length evaluation and possibly a compression bounce warning.
The embodiment utilizes the accelerated compression engine to compress the message to be compressed, and feeds the compressed message data into the learning engine for learning so as to determine whether the symbol unit is optimal, thereby realizing the self-adaptive adjustment of the optimal matching level based on the pre-learning; feeding the results of the learning engine learning to an evaluation engine to determine whether the total length is the minimum; the symbol unit evaluation information is fed into the arbiter for arbitration selection, and the arbiter determines the optimal compression symbol unit of the compression engine, thereby improving the flexibility of message compression matching and the compression processing speed, and balancing the circuit resource cost and the time cost.
EXAMPLE III
The present embodiment provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps in the message compression method as described in the first embodiment above.
The embodiment utilizes the accelerated compression engine to compress the message to be compressed, and feeds the compressed message data into the learning engine for learning so as to determine whether the symbol unit is optimal, thereby realizing the self-adaptive adjustment of the optimal matching level based on the pre-learning; feeding the results of the learning engine learning to an evaluation engine to determine whether the total length is the minimum; the symbol unit evaluation information is fed into the arbiter for arbitration selection, and the arbiter determines the optimal compression symbol unit of the compression engine, thereby improving the flexibility of message compression matching and the compression processing speed, and balancing the circuit resource cost and the time cost.
Example four
The embodiment provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the computer program to implement the steps in the message compression method according to the first embodiment.
The embodiment utilizes the accelerated compression engine to compress the message to be compressed, and feeds the compressed message data into the learning engine for learning so as to determine whether the symbol unit is optimal, thereby realizing the self-adaptive adjustment of the optimal matching level based on the pre-learning; feeding the results of the learning engine learning to an evaluation engine to determine whether the total length is the minimum; the symbol unit evaluation information is fed into the arbiter for arbitration selection, and the arbiter determines the optimal compression symbol unit of the compression engine, thereby improving the flexibility of message compression matching and the compression processing speed, and balancing the circuit resource cost and the time cost.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (9)
1. A message compression method is characterized by comprising the following steps:
initializing symbol units of a message to be compressed and respectively transmitting the symbol units to a learning engine and an accelerated compression engine;
compressing the message to be compressed by using an accelerated compression engine, and feeding the compressed message data into a learning engine for learning so as to determine whether the symbol unit is optimal; in the accelerated compression engine, accelerating the compression of the message to be compressed by adopting a method of parallel forward matching of multiple paths of S-base pointers; wherein, the S-base pointer is the starting position of sub-string;
feeding the results of the learning engine learning to an evaluation engine to determine whether the total length is the minimum;
the symbol unit evaluation information is fed into an arbitrator to carry out arbitration selection, and the arbitrator determines the optimal compression symbol unit of the compression engine.
2. The message compression method according to claim 1, characterized in that in the process of parallel forward matching of multiple S-base pointers, the matching substring lengths obtained by all S-base pointers are compared, leaving only the longest matching length.
3. The message compression method of claim 2, wherein if there is the same maximum matching length, the maximum matching length is a farthest S-base pointer position corresponding to a left boundary offset position of a fixed bitstream window to be compressed.
4. The message compression method of claim 1, wherein in the accelerated compression engine, the forward matching speed is accelerated based on a maximum matching length prediction method of cache.
5. The packet compression method according to claim 4, wherein the process of accelerating the forward matching speed based on the maximum matching length prediction method of the cache is as follows:
taking the prefix of the substring with the maximum matching as a key value K, carrying out hash calculation to obtain a hash (K) which is used as an index of the cache, continuously filling the data content which is corresponding to the cache index and is addressed to the maximum matching length until the cache is full; wherein the cache comprises a time cache and a space cache;
when the cache stores certain content, performing hit operation on the cache, and if hit occurs, obtaining the predicted maximum matching length of the substring currently in progress;
if the matching length is larger than the existing length in the cache for the same hash (K) index, updating the cache content, namely the maximum length; and if the matched position offset exceeds a preset value, updating in the time cache, otherwise, updating in the space cache.
6. The message compression method of claim 1, wherein in the arbiter, the positions of the multiple S-base pointers eliminate candidate methods, and the candidates are subjected to exclusive-or matching calculation to output the multiple S-base pointer parallel search results.
7. A message compression system is characterized by comprising a learning engine, an accelerated compression engine, an evaluation engine and an arbitrator;
the learning engine and the accelerated compression engine are used for receiving the initialized symbol unit of the message to be compressed;
the accelerating compression engine is used for compressing the message to be compressed and feeding the compressed message data into the learning engine; in the accelerated compression engine, accelerating the compression of the message to be compressed by adopting a method of parallel forward matching of multiple paths of S-base pointers; wherein, the S-base pointer is the starting position of sub-string;
the learning engine is used for learning the received message data to determine whether the symbol unit is optimal or not; the learned results are also fed to the evaluation engine;
an evaluation engine for evaluating the learned result to determine whether the overall length is the smallest; symbol unit evaluation information is also fed to the arbiter;
and the arbiter is used for arbitrating selection and determining the optimal compression symbol unit of the compression engine.
8. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of a message compression method as claimed in any one of claims 1 to 6.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps in the message compression method according to any one of claims 1-6 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010979885.XA CN112104658B (en) | 2020-09-17 | 2020-09-17 | Message compression method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010979885.XA CN112104658B (en) | 2020-09-17 | 2020-09-17 | Message compression method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112104658A CN112104658A (en) | 2020-12-18 |
| CN112104658B true CN112104658B (en) | 2022-08-12 |
Family
ID=73759902
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010979885.XA Active CN112104658B (en) | 2020-09-17 | 2020-09-17 | Message compression method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112104658B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114710565B (en) * | 2022-03-31 | 2023-11-03 | 绿盟科技集团股份有限公司 | Compression and decompression method, device and equipment of network message and readable storage medium |
| CN116436987B (en) * | 2023-06-12 | 2023-08-22 | 深圳舜昌自动化控制技术有限公司 | IO-Link master station data message transmission processing method and system |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20010054131A1 (en) * | 1999-01-29 | 2001-12-20 | Alvarez Manuel J. | System and method for perfoming scalable embedded parallel data compression |
| AU2010253964B2 (en) * | 2009-05-29 | 2015-09-17 | Bitspray Corporation | Secure storage and accelerated transmission of information over communication networks |
| US9384204B2 (en) * | 2013-05-22 | 2016-07-05 | Amazon Technologies, Inc. | Efficient data compression and analysis as a service |
| US9241044B2 (en) * | 2013-08-28 | 2016-01-19 | Hola Networks, Ltd. | System and method for improving internet communication by using intermediate nodes |
| US10503510B2 (en) * | 2013-12-27 | 2019-12-10 | Intel Corporation | SM3 hash function message expansion processors, methods, systems, and instructions |
| CN103888452B (en) * | 2014-03-10 | 2017-04-05 | 百度在线网络技术（北京）有限公司 | For the order-preserving method and device of message compression |
| US9515678B1 (en) * | 2015-05-11 | 2016-12-06 | Via Alliance Semiconductor Co., Ltd. | Hardware data compressor that directly huffman encodes output tokens from LZ77 engine |
| US9509335B1 (en) * | 2015-05-11 | 2016-11-29 | Via Alliance Semiconductor Co., Ltd. | Hardware data compressor that constructs and uses dynamic-prime huffman code tables |
| US11636327B2 (en) * | 2017-12-29 | 2023-04-25 | Intel Corporation | Machine learning sparse computation mechanism for arbitrary neural networks, arithmetic compute microarchitecture, and sparsity for training mechanism |
| CN110784225A (en) * | 2018-07-31 | 2020-02-11 | 华为技术有限公司 | Data compression, decompression method and related device, electronic equipment, system |
| CN109802685B (en) * | 2019-01-30 | 2022-12-27 | 上海兆芯集成电路有限公司 | Accelerated compression method and accelerated compression device |
-
2020
- 2020-09-17 CN CN202010979885.XA patent/CN112104658B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112104658A (en) | 2020-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20140019486A1 (en) | Logic Content Processing for Hardware Acceleration of Multi-Pattern Search | |
| CN104536911B (en) | The cache memory and its processing method that a kind of multichannel group is connected | |
| TWI594238B (en) | Hardware data compressor that directly huffman encodes output tokens from lz77 engine | |
| EP3094004B1 (en) | Hardware data compressor using dynamic hash algorithm based on input block type | |
| CN105959013B (en) | The hardware data compression device that huffman coding program is executed to matched character string or backward pointer is determined using preparatory huffman coding | |
| US7085920B2 (en) | Branch prediction method, arithmetic and logic unit, and information processing apparatus for performing brach prediction at the time of occurrence of a branch instruction | |
| CN112104658A (en) | Message compression method and system | |
| WO2017052864A1 (en) | Look-ahead hash chain matching for data compression | |
| KR101503829B1 (en) | Device and method for compressing data | |
| CN107836083A (en) | Method, apparatus and system for semantically valued data compression and decompression | |
| EP3093998B1 (en) | Hardware data compressor that sorts hash chains based on node string match probabilities | |
| US20110063143A1 (en) | Data processing apparatus and method | |
| US9912351B1 (en) | Methods for accelerating hash-based compression and apparatuses using the same | |
| CN103270699B (en) | Apparatus and method for determining search starting point | |
| US10423596B2 (en) | Efficient caching of Huffman dictionaries | |
| KR102890191B1 (en) | Pattern-based cache block compression | |
| CN109587483A (en) | Code stream extraction module | |
| Gowanlock et al. | Accelerating the similarity self-join using the GPU | |
| CN116015311A (en) | Lz4 Text Compression Method Based on Sliding Dictionary | |
| CN103544208A (en) | Method and system for matching massive feature cluster set | |
| CN108243500B (en) | Physical resource block index searching method and device and combination number arithmetic unit | |
| CN115421668A (en) | IO operation prediction method, device and related equipment | |
| CN114741118B (en) | An arbitration method for hybrid branch prediction algorithm | |
| CN118260215A (en) | Data prefetching device based on multi-access mode sensing learning | |
| CN106649836A (en) | Hardware lookup table pattern character searching method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |