CN115643310A

CN115643310A - Method, device and system for compressing data

Info

Publication number: CN115643310A
Application number: CN202211173956.2A
Authority: CN
Inventors: 李俊杰; 邢磊; 王静逸; 范秋渟; 曹岩; 徐宁; 张俊; 蔡娟; 张国超
Original assignee: CCB Finetech Co Ltd
Current assignee: CCB Finetech Co Ltd
Priority date: 2022-09-26
Filing date: 2022-09-26
Publication date: 2023-01-24

Abstract

The invention discloses a method, a device and a system for compressing data, and relates to the technical field of data security. One embodiment of the method comprises: the method, the device and the system for compressing the data can determine a target compression algorithm used for compressing the data to be compressed from a plurality of preset compression algorithms based on a set compression strategy; the method comprises the steps that each data block obtained by cutting data to be compressed is sent to a programmable controller through a preset network port bound by a target compression algorithm, the programmable controller receives a plurality of data blocks with set sizes of the data to be compressed transmitted through the preset network port, and the data blocks are compressed through the compression algorithm bound by the preset network port. The embodiment of the invention improves the flexibility and the universality of the compressed data, reduces the resource consumption of the compressed data and improves the efficiency of the compressed data.

Description

Method, device and system for compressing data

Technical Field

The present invention relates to the field of data security technologies, and in particular, to a method, an apparatus, and a system for compressing data.

Background

The compression and transmission of the data are one of the ways to improve the security and efficiency of network transmission data of internet application.

At present, a method for compressing transmission data in internet application generally uses compression software to perform data compression, or uses hardware (such as a compression card) to perform data compression, and the existing method using software compression has the problems of low compression performance and large resource occupation; the method using hardware compression has the problems of poor flexibility caused by using a customized data format and poor portability caused by high coupling with other hardware.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method, an apparatus, and a system for compressing data, which can determine, based on a set compression policy, a target compression algorithm used for compressing data to be compressed from a plurality of preset compression algorithms; the method comprises the steps that each data block obtained by cutting data to be compressed is sent to a programmable controller through a preset network port bound by a target compression algorithm, the programmable controller receives a plurality of data blocks with set sizes of the data to be compressed transmitted through the preset network port, and the data blocks are compressed through the compression algorithm bound by the preset network port. The embodiment of the invention improves the flexibility and the universality of the compressed data, reduces the resource consumption of the compressed data and improves the efficiency of the compressed data.

To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a method of compressing data, including: dividing data to be compressed into a plurality of data blocks with set sizes; determining a target compression algorithm used for compressing the data to be compressed from a plurality of preset compression algorithms based on a set compression strategy; each compression algorithm is respectively bound to a preset network port, and different compression algorithms are bound to different network ports; and sending each divided data block to a programmable controller through a preset network port bound by the target compression algorithm, so that the programmable controller compresses each data block by adopting the target compression algorithm.

Optionally, the setting of the compression policy includes: pre-compressing any one data block by utilizing the plurality of preset compression algorithms respectively; evaluating the pre-compression result of each preset compression algorithm by using a set index, and determining a target compression algorithm corresponding to the data block according to the evaluation result; the set index comprises any one or more of compression ratio, time consumption of compression, fault tolerance of compressed data, resource consumption, data type matching degree, parallelization degree and custom rule.

Optionally, the setting a compression policy further includes: in a preset compression period, taking a target compression algorithm of other compressed data as a target compression algorithm of the data to be compressed; wherein the predetermined compression period is determined by a time period between establishment and revocation of a protocol stream transmitting one or more data blocks.

Optionally, the setting a compression policy further includes: and under the condition that the data to be compressed is smaller than the set size, directly selecting a set compression algorithm from the multiple preset compression algorithms as the target compression algorithm.

Optionally, the sending, to the programmable controller, each split data block through the preset network port bound to the target compression algorithm includes: for each of the data blocks, performing: packaging the data block into a set protocol format to generate a set protocol message; adding the port number of the preset network port bound by the target compression algorithm into the set protocol message; and sending the set protocol message to the programmable controller through a preset network port bound by the target compression algorithm.

To achieve the above object, according to a second aspect of the embodiments of the present invention, there is provided a method for compressing data, which is applied to a programmable controller, including: receiving a plurality of data blocks with set sizes of data to be compressed transmitted through preset network ports connected with the programmable controller, wherein the programmable controller is connected with a plurality of preset network ports, each preset network port is bound with one compression algorithm, and different network ports are bound with different compression algorithms; compressing the data blocks by a compression algorithm bound by the preset network port; and encapsulating the compressed data into a set protocol message, and sending the set protocol message to a data receiver.

Optionally, the receiving a plurality of data blocks of a set size of data to be compressed, which are transmitted through a preset network port connected to the programmable controller, includes: and receiving a plurality of messages which are transmitted by a preset network port connected with the programmable controller and are packaged with data blocks with set sizes and port numbers of the network ports for transmitting the data blocks.

Optionally, after the receiving a plurality of messages, each of which is transmitted through a preset network port connected to the programmable controller and in which a data block of a set size is encapsulated and a port number of the network port through which the data block is transmitted, the method further includes: analyzing a preset network port from the message; and searching a compression algorithm bound with the preset network port by using the preset network port, and acquiring a compression engine corresponding to the compression algorithm so as to perform the step of compressing the plurality of data blocks by using the compression engine.

Optionally, encapsulating the compressed data into a set protocol message, and sending the set protocol message to a data receiving side, includes: when the compressed data is packaged into a set protocol message, a target compression algorithm identifier used for compressing the compressed data is added into the set protocol message, so that the data receiving party determines a compression algorithm used for decompressing the set protocol message based on the target compression algorithm identifier.

To achieve the above object, according to a third aspect of the embodiments of the present invention, there is provided an apparatus for compressing data, comprising: the device comprises a data segmentation module, an algorithm determining module and a data sending module; wherein the content of the first and second substances,

the data segmentation module is used for segmenting data to be compressed into a plurality of data blocks with set sizes;

the algorithm determining module is used for determining a target compression algorithm used for compressing the data to be compressed from a plurality of preset compression algorithms based on a set compression strategy; each compression algorithm is respectively bound to a preset network port, and different compression algorithms are bound to different network ports;

and the data sending module is used for sending each divided data block to a programmable controller through a preset network port bound by the target compression algorithm so that the programmable controller compresses each data block by adopting the target compression algorithm.

To achieve the above object, according to a fourth aspect of embodiments of the present invention, there is provided a programmable controller for compressing data, including: the device comprises a data receiving module, a data compressing module and a message sending module; wherein the content of the first and second substances,

the data receiving module is used for receiving a plurality of data blocks with set sizes of data to be compressed, which are transmitted through a preset network port connected with the programmable controller, wherein the programmable controller is connected with a plurality of preset network ports, each preset network port is bound with one compression algorithm, and different preset network ports are bound with different compression algorithms;

the compressed data module is used for compressing the data blocks through a compression algorithm bound to the preset network port;

and the message sending module is used for packaging the compressed data into a set protocol message and sending the set protocol message to a data receiving party.

To achieve the above object, according to a fifth aspect of an embodiment of the present invention, there is provided a system for compressing data, including: an apparatus for compressing data as described in the third aspect, and a programmable controller for compressing data as described in the fourth aspect.

To achieve the above object, according to a sixth aspect of an embodiment of the present invention, there is provided an electronic device for compressing data, comprising: one or more processors; storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to carry out a method as claimed in any one of the above methods of compressing data.

To achieve the above object, according to a seventh aspect of embodiments of the present invention, there is provided a computer-readable medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method as in any one of the above-described methods of compressing data.

To achieve the above object, according to an eighth aspect of the embodiments of the present invention, there is provided a computer program product including a computer program, characterized in that the program, when executed by a processor, implements the method as in any one of the methods of compressing data described above.

One embodiment of the above invention has the following advantages or benefits: the target compression algorithm used for compressing the data to be compressed can be determined from a plurality of preset compression algorithms based on a set compression strategy; the method comprises the steps that each data block obtained by cutting data to be compressed is sent to a programmable controller through a preset network port bound by a target compression algorithm, the programmable controller receives a plurality of data blocks with set sizes of the data to be compressed transmitted through the preset network port, and the data blocks are compressed through the compression algorithm bound by the preset network port. The embodiment of the invention improves the flexibility and the universality of the compressed data, reduces the resource consumption of the compressed data and improves the efficiency of the compressed data. The problems of low performance and large resource occupation caused by only using software to compress data are solved; and the problems of poor flexibility caused by using a self-defined data format, high coupling with other hardware and poor portability existing in the process of only utilizing hardware compression are solved.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

fig. 1 is a flowchart illustrating a method applied to a sending end to compress data according to an embodiment of the present invention;

FIG. 2 is a flow chart illustrating a method for compressing data applied to a programmable controller according to an embodiment of the present invention;

fig. 3 is a flowchart illustrating a method applied to transmit-end data compression according to another embodiment of the present invention;

FIG. 4 is a schematic diagram of an apparatus for compressing data according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of a programmable controller for compressing data according to an embodiment of the present invention;

FIG. 6 is a block diagram of a system for compressing data according to an embodiment of the present invention;

FIG. 7 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 8 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness. According to the technical scheme, the data acquisition, storage, use, processing and the like meet relevant regulations of national laws and regulations.

With the advance of the internet +, various traditional industries are transforming to informatization and internetworking, bringing many new opportunities and bringing new challenges. More new data can be brought in the process of industry transformation, data storage becomes a huge challenge, lossless compression of data is one of solutions for reducing storage space, and in data transmission and compression applications, high-speed and real-time processing of mass data needs a more efficient bus technology. Currently, there are techniques in the field of data compression in the following areas: 1) The method is realized by a pure software programming method, and the central processing unit compresses and decompresses; or 2) realizing compression and decompression by using a hardware acceleration method. However, neither of these two techniques can solve the current network compression problem well. 1) The method has the problems of low compression and decompression performance and overlarge CPU (Central processing Unit) load; 2) The conventional compression card communicates with the host by connecting with a new generation bus PCIe (Peripheral Component Interconnect Express). The PCIe interface has the advantages of double channels, high speed, exclusive bandwidth and the like, but is not suitable for realizing encryption and decryption of host computer transparency and network card communication; moreover, the data format of the conventional compression hardware is a custom data format. If the data processed by the traditional compression hardware is used by the network, the conversion of the data format is needed. The realization complexity is higher, and the operation efficiency is reduced; in addition, the conventional compression card sets compression related parameters by calling a driver or an application program interface. The method depends on a specific host end operating system or a chip interface, and has complex programming and poor portability.

In view of this, as shown in fig. 1, an embodiment of the present invention provides a method for compressing data, which is applied to a sending end, and the method may include the following steps:

step S101: dividing data to be compressed into a plurality of data blocks with set sizes;

step S102: determining a target compression algorithm used for compressing the data to be compressed from a plurality of preset compression algorithms based on a set compression strategy; each compression algorithm is respectively bound to a preset network port, and different compression algorithms are bound to different network ports.

The numbers of the step S101 and the step S102 are only examples, there is no strict sequential execution order between the two steps, and a target compression algorithm used by the compressed data of the compressed bag is determined through the step S101 and the step S102.

The data to be compressed may be stored in a storage medium such as a hard disk and a memory of a host or a server, and the data to be compressed may be data that needs to be transmitted through a network; the sending end may be a host client including a plurality of compression function modules according to the embodiment of the present invention, or may be a component of the host client, and the specific operations of step S101 to step S103 in the embodiment of the present invention may be implemented by the plurality of function modules of the sending end.

Further, a target compression algorithm used for compressing the data to be compressed is determined from a plurality of preset compression algorithms, and the plurality of preset compression algorithms may include: the method comprises the steps of a dictionary algorithm, a fixed bit length algorithm, a stroke length compression algorithm, an LZ77 algorithm, a Hoffman compression algorithm, an LZW compression method, a Rice algorithm, a DEFLATE and the like, wherein specific types and specific numbers of preset compression algorithms can be selected according to application scenes, and the specific types and the specific numbers of the compression algorithms are not limited.

Specifically, based on a set compression policy, a target compression algorithm used for compressing the data to be compressed is determined from a plurality of preset compression algorithms, and the method includes:

the first method comprises the following steps: dividing data to be compressed into a plurality of data blocks with set sizes, and pre-compressing any one data block by utilizing the plurality of preset compression algorithms respectively; evaluating the pre-compression result of each preset compression algorithm by using a set index, and determining a target compression algorithm corresponding to the data block according to the evaluation result; the set index comprises any one or more of compression ratio, time consumption of compression, fault tolerance of compressed data, resource consumption, data type matching degree, parallelization degree and self-defined rule.

Illustratively, the data to be compressed is a file with a size of 100KB in the memory, the file is divided into 100 data blocks with a size of 1KB (for example, 1 KB), wherein when the message is transmitted through the network, the size of the data included in the message needs to be divided when the size exceeds 1500B, therefore, preferably, the size is set to 1KB (1 KB = 1024b), the present application does not limit specific values of the size, wherein the method for dividing the file may be implemented by using compression software or custom software, further, any data block (for example, data block 1) is selected from 100 data blocks, the data block 1 is pre-compressed by using a plurality of preset compression algorithms (for example, a dictionary algorithm, a fixed bit length algorithm, a run length compression algorithm, an LZ77 algorithm, a huffman compression algorithm, a w compression method, a Rice algorithm, a late algorithm, etc.), and then the result of the pre-compression is determined by using one or more set indexes (the set indexes include compression ratio, time consumption, resource consumption, parallelism, degree of data, degree of fault tolerance of the compression, degree of the LZ, degree of data compression, degree of precompression, and degree of self-compression, or any one of the result of pre-compression algorithm, or the result of the pre-compression algorithm is evaluated according to each set index; it can be understood that, according to different data types and different application scenarios, the determined target compression algorithm is different, for example: in a data transmission scene, a target compression algorithm with the shortest consumed time can be obtained by using the consumed time of compression, and in a data backup scene, a target compression algorithm with the highest compression ratio can be obtained by using the compression ratio; aiming at a more complex application scene, a target compression algorithm can be obtained by adopting comprehensive evaluation of various set indexes; when evaluating the compression algorithm, the sum evaluation can be carried out by using any one or more of set indexes including compression ratio, compression time consumption, fault tolerance of compressed data, resource consumption, data matching degree, parallelization degree and custom rules; the compression ratio is a ratio of a size of compressed data to a size of uncompressed data, compression time consumption is a time length used for compressing a data block, fault tolerance of compressed data is executed, resource consumption is calculation resources, storage resources and the like consumed for executing compression operation, and data type matching degrees are different for different types of adopted compression algorithms, for example: in order to obtain the optimized compression effect, the compression algorithms used for compressing the text and the picture are different, the parallelization degree is the capability of whether the compression operation is executed in parallel or not, other self-defined rules for evaluating the compression effect, and the like. It can be seen that pre-compression is performed on any one of the data blocks by a plurality of preset compression algorithms; and the pre-compression result of each preset compression algorithm is evaluated by using the set index, and the target compression algorithm corresponding to the data block is determined according to the evaluation result, so that the flexibility and the universality of compressed data are improved, and the efficiency of the compressed data is improved.

The second method comprises the following steps: in a preset compression period, taking a target compression algorithm of other compressed data as a target compression algorithm of the data to be compressed; wherein the predetermined compression period is determined by a time duration between establishment and revocation of a protocol stream transmitting the data block.

In one embodiment of the invention, the predetermined compression period is determined by the length of time between the establishment and the revocation of a protocol stream transmitting one or more data blocks; taking the TCP protocol stream as an example, the time duration between the TCP protocol stream establishment and the revocation is, for example, 600s, it can be understood that the time duration between the TCP protocol stream establishment and the revocation is variable, that is, the predetermined compression period is variable, so that the predetermined compression period can be recorded by recording the time stamp of the TCP protocol stream establishment and the revocation; preferably, the same target compression algorithm is adopted for the TCP protocol stream, wherein the target compression algorithm can be determined according to setting indexes, pre-compression effect and the like from a plurality of preset compression algorithms. That is, for a certain data block, in a predetermined compression period, the target compression algorithm of other compressed data is used as the target compression algorithm of the data to be compressed, that is, in the predetermined compression period, the same target compression algorithm is used to compress the data transmitted through the protocol stream in the period. Therefore, the same algorithm is adopted in the preset compression period, the pre-compression calculation times can be reduced, the flexibility of data compression is further improved, and the consumption of calculation resources is reduced. Pre-compression operation is carried out on data carried by different protocol streams, so that different target compression algorithms are adopted according to different data; that is, different compression algorithms can be used for data in different protocol streams, and the flexibility of data compression is further improved through an adaptive compression algorithm selection scheme.

The third method comprises the following steps: and under the condition that the data to be compressed is smaller than the set size, directly selecting a set compression algorithm from the multiple preset compression algorithms as the target compression algorithm. Specifically, when it is determined that the data to be compressed is smaller than a set size (e.g., 1 KB), a set compression algorithm is directly selected from the multiple preset compression algorithms as the target compression algorithm, for example, the directly selected set compression algorithm is a DEFLATE compression algorithm with low resource consumption, and the like, it can be understood that the set compression algorithm may be different in different application scenarios; the flexibility of the data is further compressed by the method, and the consumption of computing resources is reduced.

Further, each compression algorithm is respectively bound to a preset network port, and different compression algorithms are bound to different network ports. For example: the adopted compression algorithms are 9, if the port numbers of different network ports are 50000-50008, the compression algorithm 1 is bound to the port 50000, the compression algorithm 2 is bound to the port 50001 \8230, and the like. The method for binding different compression algorithms to different network ports can be completed through a driver of the programmable controller and compression software of a sending end, the binding relationship between the compression algorithms and the interfaces can be preset in the compression software, and can also be customized and modified by a user, for example, when the port number 50000 is occupied by other services, the bound port number can be modified to 55000, and the like, and meanwhile, the updated network ports can be updated into the programmable controller through the driver of the programmable controller so as to be realized.

Therefore, the method for binding different compression algorithms to different network ports of the embodiment of the invention overcomes the problems of poor flexibility and strong coupling caused by setting compression related parameters by calling a driver or an application program interface in the prior art, and is particularly suitable for compression and transmission of network data. The preset network port number is a simple and easy scheme for classifying messages and distinguishing services, and the process of analyzing and selecting is realized by binding a compression algorithm through the specified port number without self-defining certain data structures, for example, a data packet can be received through a standard TCP protocol stack, and the TCP protocol stack can automatically classify the messages according to the port number and directly send the messages with different port numbers to different compression and decompression algorithm modules, so that different compression algorithms selected by different ports are directly realized.

Step S103: and sending each divided data block to a programmable controller through a preset network port bound by the target compression algorithm, so that the programmable controller compresses each data block by adopting the target compression algorithm.

Specifically, in one embodiment of the present invention, the Programmable controller is an FPGA (Field Programmable Gate Array). Sending each split data block to a programmable controller through a preset network port bound by the target compression algorithm, including: for each of the data blocks, performing: packaging the data block into a set protocol format to generate a set protocol message; adding the port number of the preset network port bound by the target compression algorithm into the set protocol message; and sending the set protocol message to the programmable controller through a preset network port bound by the target compression algorithm. Specifically, the data block may be encapsulated into a set protocol format of a TCP/IP protocol or a set protocol format of a UDP protocol, a set protocol packet is generated, a port number of the preset network port bound by the target compression algorithm is added to the set protocol packet, and further, the set protocol packet is sent to the programmable controller through the preset network port bound by the target compression algorithm; the method includes the steps that a driver of a network card can be called by using a bottom function of an operating system, a plurality of set protocol messages are sent to the network card through a PCIe interface in sequence, furthermore, the network card is used for sending each separated data block to a programmable controller through a preset network port bound by a target compression algorithm, illustratively, the network card and an FPGA are transmitted through a network, so that the FPGA can receive the set protocol messages containing the data blocks sent by the network card, a message receiving module can be contained in the FPGA to analyze the set protocol messages, and meanwhile, due to a protocol processing module (for example: TCP/IP module) has already performed filtering processing on the set protocol packets containing data to be compressed, for example, 1000 types of network data packets are sent to the network card, 500 of the network data packets are sent to the network ports 50000-50008, and the set protocol packets containing data to be compressed, the network card can send the set protocol packets for the ports 50000-50008 to the FPGA, and the others are correspondingly processed, for example, the set protocol packets are directly sent to the host, sent to the outside, or need to be discarded, it can be understood that the port number range and the use of the preset network ports can be preset in the network card, so that the network card obtains information that the ports 50000-50008 are reserved for the FPGA, and the set protocol packets corresponding to the ports all need to be sent to the FPGA through the network ports (for example, 10G network ports, etc.), wherein the method for setting the port numbers 50000-50008 is completed by directly writing the TCP/IP protocol processing module of the network card into the network card through a driver.

Further, after receiving the message sent by the preset network port, the programmable controller may compress each data block by using a target compression algorithm bound to the preset network port.

As shown in fig. 2, an embodiment of the present invention provides a flowchart of a method for compressing data applied to a programmable controller, where the flowchart may include the following steps:

step S201: the method comprises the steps of receiving a plurality of data blocks with set sizes of data to be compressed transmitted through a preset network port connected with a programmable controller, wherein the programmable controller is connected with a plurality of preset network ports, each preset network port is bound with one compression algorithm, and different network ports are bound with different compression algorithms.

Specifically, the receiving a plurality of data blocks with set sizes of data to be compressed, which are transmitted through a preset network port connected with the programmable controller, includes: and receiving a plurality of messages which are transmitted by a preset network port connected with the programmable controller and are packaged with data blocks with set sizes and port numbers of the network ports for transmitting the data blocks. For example: the method comprises the steps that a programmable logic controller (FPGA) receives a plurality of messages containing data blocks to be compressed sent through a preset network port 50005, and the messages of the data blocks to be compressed are packaged with the data blocks with set sizes and port numbers of network ports for transmitting the data blocks;

step S202: and compressing the data blocks by a compression algorithm bound by the preset network port.

Specifically, the programmable gate array FPGA analyzes a preset network port from the packet; and searching a compression algorithm bound with the preset network port by using the preset network port, and acquiring a compression engine corresponding to the compression algorithm so as to perform the step of compressing the plurality of data blocks by using the compression engine. For example, the preset network port parsed from the TCP header of the packet is 50005, the compression algorithm 6 bound to 50005 is found, and a compression engine (e.g., a compression module) corresponding to the compression algorithm 6 is obtained, so that the compression engine is used to perform the step of compressing the data blocks.

Step S203: and encapsulating the compressed data into a set protocol message, and sending the set protocol message to a data receiving party.

Specifically, the data receiver may be a host client (receiving a message containing compressed data forwarded by the network card); the compressed data is encapsulated into a set protocol message, and the set protocol message is sent to a data receiving party. The data receiver may process the received set protocol message, for example, the data receiver sequentially stores the data blocks in the message, and generates compressed data according to each data block; or the data receiver forwards the received message to other clients or servers; or the data receiver decompresses the data blocks contained in the received message to generate decompressed data.

Further, when the compressed data is encapsulated into a set protocol message, a target compression algorithm identifier used for compressing the compressed data is added into the set protocol message, so that the data receiving party determines a compression algorithm used for decompressing the set protocol message based on the target compression algorithm identifier. The target compression algorithm identification is packaged to enable a data receiver to perform a decompression operation on data based on the same compression algorithm used for compression.

By the method for binding different compression algorithms to different network ports and acquiring corresponding compression algorithm engines according to different network ports to process the data to be compressed, the problems of poor flexibility and strong coupling caused by setting compression related parameters by calling a driver or an application program interface in the prior art are solved, and the method is particularly suitable for compression and transmission of network data.

As shown in fig. 3, an embodiment of the present invention provides a flowchart of a method applied to a sending end to compress data; the process may include the following steps;

step S301: data to be compressed is received.

Step S302: and judging whether the data to be compressed is smaller than the set size, if so, executing step S303, otherwise, executing step S304.

Step S303: and directly selecting a set compression algorithm from the plurality of preset compression algorithms as the target compression algorithm.

Step S304: is it determined whether the processing time of the data to be compressed is within a predetermined compression period? If so, step S305 is performed, otherwise step S306 is performed.

Step S305: and taking the target compression algorithm of other compressed data as the target compression algorithm of the data to be compressed.

Step S306: pre-compressing any one data block by utilizing the plurality of preset compression algorithms respectively; and evaluating the pre-compression result of each preset compression algorithm by using the set index, and determining the target compression algorithm corresponding to the data block according to the evaluation result.

Step S307: and sending each divided data block to a programmable controller through a preset network port bound by the target compression algorithm, so that the programmable controller compresses each data block by adopting the target compression algorithm.

The descriptions in steps S301 to S306 are that a target compression algorithm used for compressing the data to be compressed is determined from a plurality of preset compression algorithms based on a plurality of policies included in the set compression policy. Further, after a target compression algorithm is determined, sending each cut data block to a programmable controller through a preset network port bound by the target compression algorithm so that the programmable controller compresses each data block by using the target compression algorithm.

It is understood that the sequence numbers of step S302, step S304, and step S306 are merely examples, and the execution order of the various policies may not be strictly executed in the order of the numbers. Therefore, the target compression algorithm is determined through various compression strategies, and the flexibility and the efficiency of data compression are further improved.

The detailed description of step S301 to step S307 is identical to the description of step S101 to step S103, and is not repeated here.

As shown in fig. 4, an embodiment of the present invention provides an apparatus 400 for compressing data, including: a data splitting module 401, an algorithm determining module 402 and a data sending module 403; wherein the content of the first and second substances,

the data splitting module 401 is configured to split data to be compressed into a plurality of data blocks with set sizes;

the algorithm determining module 402 is configured to determine, based on a set compression policy, a target compression algorithm used for compressing the data to be compressed from multiple preset compression algorithms; each compression algorithm is respectively bound to a preset network port, and different compression algorithms are bound to different network ports;

the data sending module 403 is configured to send each divided data block to a programmable controller through a preset network port bound by the target compression algorithm, so that the programmable controller compresses each data block by using the target compression algorithm.

As shown in fig. 5, an embodiment of the present invention provides a programmable controller 500 for compressing data, including: a data receiving module 501, a data compressing module 502 and a message sending module 503; wherein, the first and the second end of the pipe are connected with each other,

the data receiving module 501 is configured to receive a plurality of data blocks with set sizes of data to be compressed, where the data blocks are transmitted through a preset network port connected to the programmable controller, the programmable controller is connected to a plurality of preset network ports, each preset network port is bound with one compression algorithm, and different preset network ports are bound with different compression algorithms;

the compressed data module 502 is configured to compress the plurality of data blocks through a compression algorithm bound to the preset network port;

the message sending module 503 is configured to encapsulate the compressed data into a protocol setting message, and send the protocol setting message to a data receiving party.

As shown in fig. 6, an embodiment of the present invention provides a system 600 for compressing data, including: an apparatus 400 for compressing data as described in fig. 4, and a programmable controller 500 for compressing data as described in fig. 5.

The device for compressing data comprises a module with multiple functions, which can be attributed to a client or a server, and a programmable controller for compressing data can be attributed to the client or the server; the device for compressing data and the programmable controller for compressing data can belong to the same physical machine; or may be attributed to a different physical machine.

An embodiment of the present invention further provides an electronic device for compressing data, including: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are enabled to realize the method provided by any one of the above embodiments.

Embodiments of the present invention further provide a computer-readable medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the method provided in any of the above embodiments.

An embodiment of the present invention further provides a computer program product, including a computer program, where the program is executed by a processor to implement the method provided in any of the above embodiments.

Fig. 7 illustrates an exemplary system architecture 700 to which the method of compressing data or the apparatus for compressing data of embodiments of the present invention may be applied.

As shown in fig. 7, the system architecture 700 may include

end devices

701, 702, 703, a network 704, and a server 707. The network 704 serves to provide a medium for communication links between the

terminal devices

701, 702, 703 and the server 707. Network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

A user may use the

terminal devices

701, 702, 703 to interact with a server 705 over a network 704, to receive or send messages or the like. Various client applications, such as an e-mall client application, a web browser application, a search application, an instant messaging tool, a mailbox client, etc., may be installed on the

terminal devices

701, 702, 703.

The

terminal devices

701, 702, 703 may be various electronic devices having a display screen and supporting various client applications, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 705 may be a server providing various services, such as a background management server providing support for client applications used by users with the

terminal devices

701, 702, 703. The background management server can receive a message containing compressed data from the client and process the compressed data contained in the message.

It should be noted that, the method for compressing data provided by the embodiment of the present invention is generally executed by the

terminal devices

701, 702, and 703, and accordingly, the apparatus for compressing data is generally disposed in the

terminal devices

701, 702, and 703, and the programmable controller for compressing data is generally disposed in the

terminal devices

701, 702, and 703; the device for compressing data is in communication connection with the programmable controller for compressing data.

It should be understood that the number of terminal devices, networks, and servers in fig. 7 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 8, shown is a block diagram of a computer system 800 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 8 is only an example, and should not bring any limitation to the functions and the use range of the embodiment of the present invention.

As shown in fig. 8, the computer system 800 includes a Central Processing Unit (CPU) 801 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data necessary for the operation of the system 800 are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.

The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a signal such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that the computer program read out therefrom is mounted on the storage section 808 as necessary.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program executes the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 801.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules and/or units described in the embodiments of the present invention may be implemented by software, and may also be implemented by hardware. The described modules and/or units may also be provided in a processor, and may be described as: a processor comprises a data segmentation module, an algorithm determination module and a data sending module; the names of these modules do not in some cases constitute a limitation to the module itself, for example, the split data module may also be described as a "module that splits data to be compressed into a plurality of data blocks of a set size".

As another aspect, the present invention also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be separate and not assembled into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: dividing data to be compressed into a plurality of data blocks with set sizes; determining a target compression algorithm used for compressing the data to be compressed from a plurality of preset compression algorithms based on a set compression strategy; each compression algorithm is respectively bound to a preset network port, and different compression algorithms are bound to different network ports; and sending each divided data block to a programmable controller through a preset network port bound by the target compression algorithm, so that the programmable controller compresses each data block by adopting the target compression algorithm.

According to the embodiment of the invention, a target compression algorithm used for compressing data to be compressed can be determined from a plurality of preset compression algorithms based on a set compression strategy; the method comprises the steps that each data block obtained by cutting data to be compressed is sent to a programmable controller through a preset network port bound by a target compression algorithm, the programmable controller receives a plurality of data blocks with set sizes of the data to be compressed transmitted through the preset network port, and the data blocks are compressed through the compression algorithm bound by the preset network port. The embodiment of the invention improves the flexibility and the universality of the compressed data, reduces the resource consumption of the compressed data and improves the efficiency of the compressed data.

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of compressing data, comprising:

dividing data to be compressed into a plurality of data blocks with set sizes;

determining a target compression algorithm used for compressing the data to be compressed from a plurality of preset compression algorithms based on a set compression strategy; each compression algorithm is respectively bound to a preset network port, and different compression algorithms are bound to different network ports;

and sending each divided data block to a programmable controller through a preset network port bound by the target compression algorithm, so that the programmable controller compresses each data block by adopting the target compression algorithm.

2. The method of claim 1, wherein setting the compression policy comprises:

pre-compressing any one data block by utilizing the plurality of preset compression algorithms respectively;

evaluating the pre-compression result of each preset compression algorithm by using a set index, and determining a target compression algorithm corresponding to the data block according to the evaluation result;

the set index comprises any one or more of compression ratio, time consumption of compression, fault tolerance of compressed data, resource consumption, data type matching degree, parallelization degree and self-defined rule.

3. The method of claim 1, wherein setting the compression policy further comprises:

in a preset compression period, taking a target compression algorithm of other compressed data as a target compression algorithm of the data to be compressed; wherein the predetermined compression period is determined by a time period between establishment and revocation of a protocol stream transmitting one or more data blocks.

4. The method of claim 1, wherein setting the compression policy further comprises:

and under the condition that the data to be compressed is smaller than the set size, directly selecting a set compression algorithm from the multiple preset compression algorithms as the target compression algorithm.

5. The method of claim 1,

the step of sending each cut data block to a programmable controller through the preset network port bound by the target compression algorithm includes:

for each of the data blocks, performing:

packaging the data block into a set protocol format to generate a set protocol message; adding the port number of the preset network port bound by the target compression algorithm into the set protocol message;

and sending the set protocol message to the programmable controller through a preset network port bound by the target compression algorithm.

6. A method of compressing data for use in a programmable controller, comprising:

receiving a plurality of data blocks with set sizes of data to be compressed transmitted through preset network ports connected with the programmable controller, wherein the programmable controller is connected with a plurality of preset network ports, each preset network port is bound with one compression algorithm, and different network ports are bound with different compression algorithms;

compressing the data blocks by a compression algorithm bound by the preset network port;

and encapsulating the compressed data into a set protocol message, and sending the set protocol message to a data receiving party.

7. The method of claim 6,

the receiving a plurality of data blocks with set sizes of data to be compressed transmitted through a preset network port connected with the programmable controller comprises:

and receiving a plurality of messages which are transmitted by a preset network port connected with the programmable controller and are packaged with data blocks with set sizes and port numbers of the network ports for transmitting the data blocks.

8. The method of claim 7, wherein after receiving a plurality of messages encapsulating a data block of a set size and a port number of a network port through which the data block is transmitted, the messages being transmitted via a predetermined network port connected to the programmable controller, the method further comprises:

analyzing a preset network port from the message;

and searching a compression algorithm bound with the preset network port by using the preset network port, and acquiring a compression engine corresponding to the compression algorithm so as to perform the step of compressing the plurality of data blocks by using the compression engine.

9. The method of claim 6,

encapsulating the compressed data into a set protocol message, and sending the set protocol message to a data receiver, wherein the method comprises the following steps:

when the compressed data is packaged into a set protocol message, a target compression algorithm identifier used for compressing the compressed data is added into the set protocol message, so that the data receiving party determines a compression algorithm used for decompressing the set protocol message based on the target compression algorithm identifier.

10. An apparatus for compressing data, comprising: the device comprises a data segmentation module, an algorithm determination module and a data sending module; wherein the content of the first and second substances,

11. A programmable controller for compressing data, comprising: the device comprises a data receiving module, a data compressing module and a message sending module; wherein the content of the first and second substances,

the data receiving module is used for receiving a plurality of data blocks with set sizes of data to be compressed transmitted through a preset network port connected with the programmable controller, wherein the programmable controller is connected with a plurality of preset network ports, each preset network port is bound with one compression algorithm, and different preset network ports are bound with different compression algorithms;

the compressed data module is used for compressing the data blocks through a compression algorithm bound by the preset network port;

12. A system for compressing data, comprising: apparatus for compressing data according to claim 9, and a programmable controller for compressing data according to claim 11.

13. An electronic device, comprising:

one or more processors;

a storage device to store one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-9.

14. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-9.

15. A computer program product comprising a computer program, characterized in that the program, when executed by a processor, implements the method of any one of claims 1-9.