CN114125071B - Data compression transmission method and device - Google Patents
Data compression transmission method and device Download PDFInfo
- Publication number
- CN114125071B CN114125071B CN202111396502.7A CN202111396502A CN114125071B CN 114125071 B CN114125071 B CN 114125071B CN 202111396502 A CN202111396502 A CN 202111396502A CN 114125071 B CN114125071 B CN 114125071B
- Authority
- CN
- China
- Prior art keywords
- data message
- message
- data
- compression
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 230000005540 biological transmission Effects 0.000 title claims abstract description 52
- 238000013144 data compression Methods 0.000 title claims abstract description 30
- 238000007906 compression Methods 0.000 claims abstract description 255
- 230000006835 compression Effects 0.000 claims abstract description 253
- 238000004891 communication Methods 0.000 claims abstract description 106
- 238000012545 processing Methods 0.000 claims description 24
- 230000006837 decompression Effects 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 8
- 238000004458 analytical method Methods 0.000 claims description 3
- 230000000903 blocking effect Effects 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000012360 testing method Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 4
- 230000003993 interaction Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 239000002699 waste material Substances 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000015556 catabolic process Effects 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/146—Markers for unambiguous identification of a particular session, e.g. session cookie or URL-encoding
Abstract
The invention provides a data compression transmission method, and relates to the technical field of databases. The method is applied to a client and comprises the steps of sending a first message to a server, wherein the first message is used for requesting to establish connection with the server; the first message includes: communication compression control parameters and compression threshold parameters; if the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter, compressing the first data message according to a preset compression algorithm to generate a second data message; the second data message carries compression identification information; the first data message is a message with a fixed format obtained by encoding request data sent by an application layer according to a preset communication protocol; and sending the second data message to the server. By adopting the method, under the condition of limited network bandwidth, when the database server is queried and acquires data with large data volume, the problem of reduced data transmission performance caused by network blocking can be avoided.
Description
Technical Field
The present invention relates to the field of database technologies, and in particular, to a data compression transmission method and apparatus.
Background
With the development of the internet and the information age, data has penetrated into various industries, and the current society enterprises generate huge amounts of data to be stored and data interaction, which brings serious challenges to the network and the information system. When the information system application needs to inquire the data with large data volume through the network, the problem of network blocking easily occurs, and the condition that the CPU (central processing unit ) at the server side is idle and the network is busy is caused.
In the existing scheme, one way is to increase network bandwidth through operators to solve the problems, but the cost of the method is higher, and the operation cost burden of enterprises is increased; another way is to perform data operations with large traffic while the network is idle, but this way only solves a small part of the traffic problems.
Therefore, under the condition of limited network bandwidth, when a database server is queried and obtained by concurrency, how to avoid the degradation of data transmission performance caused by network congestion is a problem to be solved.
Disclosure of Invention
In order to solve the technical problems or at least partially solve the technical problems, the invention provides a data compression transmission method and a data compression transmission device, which solve the problem that when a database server is queried and acquires a large amount of data in parallel under the condition of limited network bandwidth, the data transmission performance is reduced due to network congestion.
In order to achieve the above object, the embodiment of the present invention provides the following technical solutions:
in a first aspect, an embodiment of the present invention provides a data compression transmission method, applied to a client, where the method includes:
a first message is sent to a server, wherein the first message is used for requesting to establish connection with the server; the first message includes: communication compression control parameters and compression threshold parameters;
if the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter, compressing the first data message according to a preset compression algorithm to generate a second data message; the first data message is a message with a fixed format obtained by encoding request data sent by an application layer according to a preset communication protocol; the second data message is a data message after the first data message is compressed and carries compression identification information;
and sending the second data message to the server.
As an optional implementation manner of the embodiment of the present invention, after sending the first packet to the server, the method further includes:
receiving a second message sent by the server, wherein the second message is used for indicating whether the client and the server are successfully connected;
Judging whether the connection between the client and the server is successfully established or not according to the second message;
if the connection is successful, the first data message is compressed according to a preset compression algorithm to generate a second data message when the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter.
As an optional implementation manner of the embodiment of the present invention, after sending the second data packet to the server, the method further includes:
receiving a fourth data message sent by the server; the fourth data message is a decompressed data message of the third data message and carries compression identification information; the third data message is generated after the first data message is subjected to basic database operation;
decompressing the fourth data message according to a preset compression algorithm, and recovering to obtain a third data message;
and sending the third data message to an application layer.
As an optional implementation manner of the embodiment of the present invention, if the communication compression control parameter is zero, or the packet size of the first data packet is smaller than or equal to the value of the compression threshold parameter, the first data packet is sent to the server.
As an optional implementation manner of the embodiment of the present invention, a data compression transmission method is provided, and the method is applied to a server, and includes:
receiving a first message sent by a client, wherein the first message is used for requesting to establish connection with a server; the first message includes: communication compression control parameters and compression threshold parameters;
analyzing the first message to obtain the communication compression control parameter and the compression threshold parameter;
receiving a second data message sent by a client; the second data message is a data message after the first data message is compressed and carries compression identification information; the first data message is a message with a fixed format obtained by encoding request data sent by an application layer according to a preset communication protocol;
decompressing the second data message according to a preset compression algorithm, and recovering to obtain a first data message;
and processing the first data message according to the decompression result to generate a third data message, wherein the third data message is generated after the first data message is subjected to database basic operation.
As an optional implementation manner of the embodiment of the present invention, after receiving the first packet sent by the client, the method further includes:
And sending a second message to the client, wherein the second message is used for indicating whether the client and the server are successfully connected.
As an optional implementation manner of the embodiment of the present invention, after processing the first data packet according to the decompression result to generate a third data packet, the method further includes:
if the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter, decompressing the third data message according to a preset compression algorithm to generate a fourth data message; the fourth data message is the data message decompressed by the third data message and carries compression identification information;
and sending the fourth data message to the client.
In a second aspect, an embodiment of the present invention provides a client, including:
the starting module is used for sending a first message to the server, wherein the first message is used for requesting to establish connection with the server; the first message includes: communication compression control parameters and compression threshold parameters;
the compression module is used for compressing the first data message according to a preset compression algorithm to generate a second data message if the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter; the first data message is a message with a fixed format obtained by encoding request data sent by an application layer according to a preset communication protocol; the second data message is a data message after the first data message is compressed and carries compression identification information;
And the sending module is used for sending the second data message to the server.
As an optional implementation manner of the embodiment of the present invention, the client further includes:
the receiving response module is used for receiving a second message sent by the server side, wherein the second message is used for indicating whether the client side and the server side are successfully connected or not;
the judging and connecting module is used for judging whether the connection between the client and the server is successfully established according to the second message; if the connection is successful, the first data message is compressed according to a preset compression algorithm to generate a second data message when the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter.
As an optional implementation manner of the embodiment of the present invention, the client further includes:
the message receiving module is used for receiving a fourth data message sent by the server; the fourth data message is a decompressed data message of the third data message and carries compression identification information; the third data message is generated after the first data message is subjected to basic database operation;
the message decompressing module is used for decompressing the fourth data message according to a preset compression algorithm and recovering to obtain a third data message;
And the message sending module is used for sending the third data message to an application layer.
As an optional implementation manner of the embodiment of the present invention, the client further includes:
and the parameter judging module is used for sending the first data message to the server if the communication compression control parameter is zero or the data packet size of the first data message is smaller than or equal to the value of the compression threshold parameter.
In a third aspect, an embodiment of the present invention provides a server, including:
the connection module is used for receiving a first message sent by the client, wherein the first message is used for requesting to establish connection with the server; the first message includes: communication compression control parameters and compression threshold parameters;
the analysis module is used for analyzing the first message and acquiring the communication compression control parameter and the compression threshold parameter;
the receiving module is used for receiving the second data message sent by the client; the second data message is a data message after the first data message is compressed and carries compression identification information; the first data message is a message with a fixed format obtained by encoding request data sent by an application layer according to a preset communication protocol;
The decompression module is used for decompressing the second data message according to a preset compression algorithm and recovering to obtain a first data message;
the processing module is used for processing the first data message according to the decompression result to generate a third data message, wherein the third data message is generated after the first data message is subjected to database basic operation.
As an optional implementation manner of the embodiment of the present invention, the server further includes:
and the sending response module is used for sending a second message to the client, wherein the second message is used for indicating whether the client and the server are successfully connected or not.
As an optional implementation manner of the embodiment of the present invention, the server further includes:
the compression processing module is used for decompressing the third data message according to a preset compression algorithm to generate a fourth data message if the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter; the fourth data message is the data message decompressed by the third data message and carries compression identification information;
and the sending result module is used for sending the fourth data message to the client.
In a fourth aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, where the memory stores a computer program, and the processor implements the data compression transmission method according to the first aspect or any implementation manner of the first aspect when executing the computer program.
The data compression transmission method provided by the embodiment of the invention is applied to a client, and is characterized in that first a first message comprising a communication compression control parameter and a compression threshold parameter is sent to a server, then whether the first data message is compressed or not is determined according to the communication compression control parameter and the compression threshold parameter, the first data message is compressed according to a preset compression algorithm, a second data message is generated, and the second data message is sent to the server. Because the first message comprises the communication compression control parameter and the compression threshold parameter, and the first data message is a message with a fixed format obtained by encoding request data sent by the application layer according to a preset communication protocol, before the request message is transmitted, the problem of CPU resource waste caused by compression processing of small data packets can be avoided by judging the relation between the data packet size of the first data message and the compression threshold parameter, and whether the first data message is compressed and the compression ratio size can be determined by the specific value of the communication compression control parameter, so that the data transmission quantity is reduced, the data transmission density is increased, and the data transmission performance under the network bottleneck is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, and it will be obvious to a person skilled in the art that other drawings can be obtained from these drawings without inventive effort.
Fig. 1 is a schematic diagram of an application scenario of a data compression transmission method in one embodiment;
FIG. 2 is a schematic diagram of a client-server interaction flow of a data compression and transmission method in one embodiment;
FIG. 3 is a schematic diagram of a client-server interaction flow of a data compression and transmission method in another embodiment;
fig. 4 is a schematic structural diagram of a data compression transmission method applied to a client in one embodiment;
FIG. 5 is a schematic diagram of a data compression transmission method applied to a server in one embodiment;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order that the above objects, features and advantages of the invention will be more clearly understood, a further description of the invention will be made. It should be noted that, without conflict, the embodiments of the present invention and features in the embodiments may be combined with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present invention may be practiced otherwise than as described herein; it will be apparent that the embodiments in the specification are only some, but not all, embodiments of the invention.
Relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
In embodiments of the invention, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g." in an embodiment should not be taken as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion. Furthermore, in the description of the embodiments of the present invention, unless otherwise indicated, the meaning of "plurality" means two or more.
The overall conception of the embodiment of the invention is as follows: under the condition of limited bandwidth, when a business layer needs to perform concurrent query on a database server to acquire a large number of large result sets, the problem of data transmission performance degradation caused by network blocking is solved by sending a first message comprising a communication compression control parameter and a compression threshold parameter to a server, determining whether to compress the first data message according to the communication compression control parameter and the compression threshold parameter, compressing the first data message according to a preset compression algorithm to generate a second data message, and finally sending the second data message to the server.
Network congestion refers to the phenomenon that when a certain part of a communication subnetwork is too many packets to be handled by that part of the network, this part or even the whole network is degraded.
The application provides a data compression transmission method which can be applied to an application environment shown in fig. 1. FIG. 1 is an application scenario diagram of a data compression transmission method in one embodiment; the application environment includes a client 11 and a server 12. Wherein, the client and the server communicate data through a network. Specifically, a client sends a first message to a server, wherein the first message is used for requesting to establish connection with the server; the first message includes: communication compression control parameters and compression threshold parameters; if the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter, compressing the first data message according to a preset compression algorithm to generate a second data message; the first data message is a message with a fixed format obtained by encoding request data sent by an application layer according to a preset communication protocol; the second data message is a data message after the first data message is compressed and carries compression identification information; and sending the second data message to the server. The client may be, but not limited to, various tablet computers, notebook computers, and desktop computers, and the server may be implemented by an independent server or a server cluster formed by a plurality of servers.
Based on the application scenario shown in fig. 1, an embodiment of the present invention provides a data compression transmission method, which is shown in fig. 2. In this embodiment, the data compression transmission method includes the steps of:
s21, sending a first message to the server.
The first message is used for requesting to establish connection with a server; the first message includes: communication compression control parameters and compression threshold parameters.
Specifically, the first message may be understood as a start message, where the start message has two roles, and the first message is a connection opening message when the client communicates with the database server. When the client side inquires data through the database server, user authority verification is needed first, and the user name and the password are verified and then connected with the database. Second, the communication compression control parameter and the compression threshold parameter are designed to be added in the starting message format. In addition, the first message may further include: application name, protocol version number, time format, precision of floating point number, type, etc.
The communication compression control parameter represents a transmission compression level of the communication between the client and the server, and the value range of the communication compression control parameter can be any integer from 0 to 9. Specifically, 0 represents closing communication compression, 1-9 represents opening compression level, the larger the number represents the higher compression level, the smaller the compressed transmission data amount is, and the higher the corresponding CPU consumption is. The value of the compression threshold parameter may be set according to the network situation and the data transmission scenario, for example, the value of the compression threshold parameter may be set to 100M.
Illustratively, in one application scenario, the client communicates with the server in compliance with a front-end and back-end communication protocol of a company, i.e., in the form of protocol data packets. For example, the first message may include: host name, host IP address, host port, operation user name, operation user password, database name, compression level, compression byte, etc., examples are as follows: "host=myhost, host=10.11.12.13, port=54388, user= wli, password=asdfg123|@ #, dbname=kingbase, comm_compression=2, cmps_min_size=1 kbyte". In practical application, the communication protocol formats used by the client and the server in different application scenarios can be agreed by themselves, and the specific limitation is not imposed here.
Correspondingly, the server receives a first message sent by the client.
It can be understood that, in this embodiment, the KCI interface provided by a certain company (providing a native efficient C access interface) is used to implement access to the database, and mainly operations such as connection, disconnection, performing addition and deletion, transaction opening, transaction rollback, and transaction commit are provided for the database. In addition, a database connection pool, a monitoring thread and a maintenance thread are also provided, so that the access efficiency of the database is provided.
Specifically, after receiving the first message sent by the client, the server executes step S31.
S31, analyzing the first message, and acquiring the communication compression control parameter and the compression threshold parameter.
Specifically, the server obtains the communication compression control parameter and the compression threshold parameter by analyzing the first message.
S22, if the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter, compressing the first data message according to a preset compression algorithm to generate a second data message.
The first data message is a message with a fixed format obtained by encoding request data sent by the application layer according to a preset communication protocol.
For example, the data communication compression function of a database of a company needs to select a suitable compression algorithm to complete the compression and decompression functions. The data compression algorithm is mainly divided into two types of lossy compression and lossless compression, and because the basic requirements of data in a database are accuracy and integrity, the requirements of data communication compression of a certain company on the compression algorithm can be completely restored, namely, a lossless compression algorithm is required. Meanwhile, the content to be compressed in the data communication compression of the database of a certain company is mainly text data content, and high compression rate and compression efficiency are required, so that an LZ77 algorithm can be selected by a preset compression algorithm, and the compression and decompression tasks of the data content of the communication messages of the client and the server are completed. The LZ77 algorithm uses a virtual window that can slide along with the compression process as a term dictionary, and if a character string to be compressed appears in the window, the appearance position and the length thereof are output. The term matching is performed using a fixed size window, instead of matching in all the already encoded information, because the time consumption of the matching algorithm is often great, and the size of the dictionary must be limited to ensure the efficiency of the algorithm; the dictionary window is slid as compression progresses so that the most recently encoded information is always contained therein, as for most information, the string to be encoded tends to find the matching string more easily in the most recent context.
In addition, it will be appreciated that in different data communication scenarios, suitable compression and decompression algorithms may be selected according to requirements, which is only an example and not particularly limited herein.
S23, sending the second data message to the server.
The second data message is a data message after the first data message is compressed, and carries compression identification information.
For example, for implementation of the data communication compression function, the data message structure of the communication protocol between the client and the server of a database of a company may be as shown in table 1.
TABLE 1
Specifically, the compressed identification information may be represented by a character 'M', and after the message is decompressed, if the compressed identification bit of the data is 'M', the message is represented as a compressed message; if the compression mark of the data is other characters or numbers, the message is an uncompressed message.
Correspondingly, the server receives a second data message sent by the client.
Specifically, after receiving the second data packet sent by the client, the server executes step S32.
S32, decompressing the second data message according to a preset compression algorithm, and recovering to obtain a first data message.
The second data message is a data message after the first data message is compressed, and carries compression identification information.
S33, processing the first data message according to the decompression result to generate a third data message.
The third data message is generated after the first data message is subjected to database basic operation.
Specifically, the basic database operations are: basic operations of the library, such as creating a database, viewing all information of the database, modifying a database encoding set, and removing the database; basic operations of the table, such as creating the table, looking up table information, modifying field attributes, removing the table; basic operation of fields, adding fields, deleting fields, inserting fields, modifying fields, looking up fields, etc.
For example, after the first data message is parsed, the function of obtaining the message is to add a field, and then the field adding operation is performed in the database corresponding to the server side, so as to obtain a third data message, where the data of the third data message includes a newly added field.
The data compression transmission method provided by the embodiment of the invention is applied to a client, and is characterized in that first a first message comprising a communication compression control parameter and a compression threshold parameter is sent to a server, then whether the first data message is compressed or not is determined according to the communication compression control parameter and the compression threshold parameter, the first data message is compressed according to a preset compression algorithm, a second data message is generated, and the second data message is sent to the server. Because the first message comprises the communication compression control parameter and the compression threshold parameter, and the first data message is a message with a fixed format obtained by encoding request data sent by the application layer according to a preset communication protocol, before the request message is transmitted, the problem of CPU resource waste caused by compression processing of small data packets can be avoided by judging the relation between the value of the compression threshold parameter and the size of the data packet of the first data message, and whether the first data message is compressed and the compression ratio can be determined by the specific value of the communication compression control parameter, so that the data transmission quantity is reduced, the data transmission density is increased, and the data transmission performance under the network bottleneck is improved.
In one embodiment, based on fig. 2, the data compression transmission method provided by the embodiment of the present invention, as shown in fig. 3, includes the following steps:
s21, sending a first message to the server.
The first message is used for requesting to establish connection with a server; the first message includes: communication compression control parameters and compression threshold parameters.
Correspondingly, the server receives a first message sent by the client.
Specifically, after receiving the first message sent by the client, the server executes step S31.
S31, analyzing the first message, and acquiring the communication compression control parameter and the compression threshold parameter.
Specifically, the server receives a first message for requesting connection, parses the first message, and stores a communication compression control parameter and a compression threshold parameter, wherein the two compression parameters are used as the basis for data compression processing in a later data interaction process.
Before executing step S31, step S301 is further executed to send a second message to the client.
The second message is used for indicating whether the client and the server are successfully connected.
Specifically, whether the connection between the client and the server is successfully established is judged according to the second message.
If the connection is successful, step S22 is performed.
S22, if the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter, compressing the first data message according to a preset compression algorithm to generate a second data message.
The first data message is a message with a fixed format obtained by encoding request data sent by an application layer according to a preset communication protocol; the second data message is a data message after the first data message is compressed, and carries compression identification information.
Specifically, when the value of the communication compression control parameter is set to be non-zero in the first message sent by the client, the client needs to establish a connection for compressing transmission data, and before the subsequent message is sent, if the size of the data packet is greater than the value of the set compression threshold parameter, compression processing is adopted.
S23, sending the second data message to the server.
Correspondingly, the server receives a second data message sent by the client.
Specifically, after receiving the second data packet sent by the client, the server executes step S32.
S32, decompressing the second data message according to a preset compression algorithm, and recovering to obtain a first data message.
Specifically, after the server receives the second data message after compressing the first data message, which is sent by the client, the second data message carries compression identification information, and the second data message is judged to be a compressed message according to the compression identification information, so that decompression processing is performed on the second data message, and the first data message is recovered.
S33, processing the first data message according to the decompression result to generate a third data message, wherein the third data message is generated after the first data message is subjected to database basic operation.
And S34, if the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter, decompressing the third data message according to a preset compression algorithm to generate a fourth data message.
The fourth data message is the data message decompressed by the third data message and carries compression identification information.
And S35, sending the fourth data message to the client.
Correspondingly, the client receives a fourth data message sent by the server.
Specifically, after the client receives the fourth data packet sent by the server, step S24 is executed.
S24, decompressing the fourth data message according to a preset compression algorithm, and recovering to obtain a third data message.
S25, the third data message is sent to an application layer.
Specifically, after the client receives the fourth data message after compressing the third data message, which is sent by the server, the fourth data message carries compression identification information, and the fourth data message is judged to be a compressed message according to the compression identification information, so that decompression processing is performed on the fourth data message, the third data message is recovered, and then the third data message is sent to the application layer.
By way of example, when the client and the server transmit a large result set through the network concurrently, the transmission efficiency can be improved by about 30% by starting the communication compression mode when the CPU is idle and the bandwidth is insufficient. Taking the test scenario of multi-thread concurrency acquisition of the simulation power POC as an example, the test is performed:
the test database environment is as follows: 64-core central processing unit, x86+220G memory, hundred megabandwidth. The test data size was: test table 500 row 6 columns, data type is int (integer) or varchar (variable length string), each row is about 16 kbytes in size. The test case environment is: JDK1.8, multithreading concurrently executes the query acquisition results. The compression effect table is tested to obtain that: when the opening compression level is 2, the efficiency is improved the highest and is about 25% -30%.
In addition, it should be noted that, in this embodiment, the test is to replace network resources with the CPU, and a high performance physical machine is required to process compression and decompression, otherwise, the test effect will be greater.
In the process of data communication between the client and the server, a communication layer compression method is adopted to compress communication messages, so that the data transmission quantity is reduced, the data transmission density is increased, and the transmission performance under the network bottleneck is improved.
In one embodiment, if the communication compression control parameter is zero, or the packet size of the first data packet is less than or equal to the value of the compression threshold parameter, the first data packet is sent to the server.
Specifically, when the first message of the client sets the communication compression control parameter to zero, which means that the communication compression is closed, only the original data message sent by the application layer needs to be transmitted. Or when the size of the data packet of the first data packet is smaller than or equal to the value of the compression threshold parameter, the data volume of the transmitted data packet is small, and the problem of excessive network bandwidth consumption in the transmission process is avoided, so that compression is not needed, and only the original data packet is needed to be sent to the server at the same time.
In this case, the server receives the first data packet, where the compressed identification information carried in the first data packet is other characters or numbers except for the character 'M', which indicates that the first data packet is an uncompressed packet.
Correspondingly, the server executes step S33, processes the first data message according to the decompression result, and generates a third data message, wherein the third data message is generated after the first data message is subjected to the basic database operation.
Specifically, since the first data message is an uncompressed message, the first data message does not need to be decompressed, and the first data message is processed directly according to the analysis result of the first data message. The specific processing manner is described in the above embodiments, and will not be described herein.
In one embodiment, if the communication compression control parameter is zero, or the packet size of the third data packet is less than or equal to the value of the compression threshold parameter, the third data packet is sent to the client.
Specifically, before the server sends the data message to the client, it first determines whether the communication compression control parameter is zero, if so, it indicates that the communication compression is closed, and then the third data message is transmitted. And judging whether the data packet size of the third data packet is smaller than or equal to the value of the compression threshold parameter, if so, indicating that the data volume of the transmitted data packet is small, and not causing the problem of excessive network bandwidth consumption in the transmission process, so that the third data packet is not compressed, and only the uncompressed third data packet is needed to be sent to a server at the same time.
Correspondingly, the client receives the third data message, and the compressed identification information carried by the third data message is other characters or numbers except the character 'M', which indicates that the third data message is an uncompressed message, so that the third data message is not required to be decompressed, and the third data message is directly sent to the application layer.
In the process of data communication between the client and the server, the method for judging the communication compression control parameter and the compression threshold parameter before the client transmits data and judging the communication compression control parameter and the compression threshold parameter before the server transmits data can effectively avoid the problem of CPU resource waste caused by processing small data packets.
The data compression transmission method provided by the embodiment of the invention is applied to a client, and is characterized in that first a first message comprising a communication compression control parameter and a compression threshold parameter is sent to a server, then whether the first data message is compressed or not is determined according to the communication compression control parameter and the compression threshold parameter, the first data message is compressed according to a preset compression algorithm, a second data message is generated, and the second data message is sent to the server. Because the first message comprises the communication compression control parameter and the compression threshold parameter, and the first data message is a message with a fixed format obtained by encoding request data sent by the application layer according to a preset communication protocol, before the request message is transmitted, the problem of CPU resource waste caused by compression processing of small data packets can be avoided by judging the relation between the data packet size of the first data message and the compression threshold parameter, and whether the first data message is compressed and the compression ratio size can be determined by the specific value of the communication compression control parameter, so that the data transmission quantity is reduced, the data transmission density is increased, and the data transmission performance under the network bottleneck is improved.
In one embodiment, as shown in FIG. 4, there is provided a client 400 comprising:
the starting module 410 is configured to send a first message to the server, where the first message is used to request to establish a connection with the server; the first message includes: communication compression control parameters and compression threshold parameters;
the compression module 420 is configured to compress the first data packet according to a preset compression algorithm to generate a second data packet if the communication compression control parameter is not zero and the packet size of the first data packet is greater than the value of the compression threshold parameter; the first data message is a message with a fixed format obtained by encoding request data sent by an application layer according to a preset communication protocol; the second data message is a data message after the first data message is compressed and carries compression identification information;
and a sending module 430, configured to send the second data packet to the server.
As an optional implementation manner of the embodiment of the present invention, the client further includes:
the receiving response module is used for receiving a second message sent by the server side, wherein the second message is used for indicating whether the client side and the server side are successfully connected or not;
The judging and connecting module is used for judging whether the connection between the client and the server is successfully established according to the second message; if the connection is successful, the first data message is compressed according to a preset compression algorithm to generate a second data message when the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter.
As an optional implementation manner of the embodiment of the present invention, the client further includes:
the message receiving module is used for receiving a fourth data message sent by the server; the fourth data message is a decompressed data message of the third data message and carries compression identification information; the third data message is generated after the first data message is subjected to basic database operation;
the message decompressing module is used for decompressing the fourth data message according to a preset compression algorithm and recovering to obtain a third data message;
and the message sending module is used for sending the third data message to an application layer.
As an optional implementation manner of the embodiment of the present invention, the client further includes:
and the parameter judging module is used for sending the first data message to the server if the communication compression control parameter is zero or the data packet size of the first data message is smaller than or equal to the value of the compression threshold parameter.
In one embodiment, as shown in FIG. 5, a server 500 is provided, comprising:
the connection module 510 is configured to receive a first message sent by a client, where the first message is used to request to establish connection with a server; the first message includes: communication compression control parameters and compression threshold parameters;
the parsing module 520 is configured to parse the first packet, and obtain the communication compression control parameter and the compression threshold parameter;
a receiving module 530, configured to receive a second data packet sent by the client; the second data message is a data message after the first data message is compressed and carries compression identification information; the first data message is a message with a fixed format obtained by encoding request data sent by an application layer according to a preset communication protocol;
the decompression module 540 is configured to decompress the second data packet according to a preset compression algorithm, and recover to obtain a first data packet;
and the processing module 550 is configured to process the first data packet according to the decompression result, and generate a third data packet, where the third data packet is a data packet generated after performing a database basic operation on the first data packet.
As an optional implementation manner of the embodiment of the present invention, the server further includes:
and the sending response module is used for sending a second message to the client, wherein the second message is used for indicating whether the client and the server are successfully connected or not.
As an optional implementation manner of the embodiment of the present invention, the server further includes:
the compression processing module is used for decompressing the third data message according to a preset compression algorithm to generate a fourth data message if the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter; the fourth data message is the data message decompressed by the third data message and carries compression identification information;
and the sending result module is used for sending the fourth data message to the client.
The embodiment of the invention also provides an electronic device, and fig. 6 is a schematic structural diagram of the electronic device for data compression and transmission. As shown in fig. 6, the data compression transmission electronic device provided in this embodiment includes: a memory 61 and a processor 62, the memory 61 for storing a computer program; the processor 62 is configured to perform the steps performed by any of the embodiments of the data compression transmission methods provided by the method embodiments described above when the computer program is invoked.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein.
The processor may be a central decision unit (CentralProcessingUnit, CPU), but may also be other general purpose processors, digital signal processors (DigitalSignalProcessor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash memory (flashRAM). Memory is an example of a computer-readable medium.
Computer readable media include both non-transitory and non-transitory, removable and non-removable storage media. Storage media may embody any method or technology for storage of information, which may be computer readable instructions, data structures, program modules, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transshipment) such as modulated data signals and carrier waves.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing is merely a specific embodiment of the disclosure to enable one skilled in the art to understand or practice the disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown and described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (8)
1. A data compression transmission method, applied to a client, the method comprising:
a first message is sent to a server, wherein the first message is used for requesting to establish connection with the server; the first message includes: communication compression control parameters and compression threshold parameters;
if the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter, compressing the first data message according to a preset compression algorithm to generate a second data message; the first data message is a message with a fixed format obtained by encoding request data sent by an application layer according to a preset communication protocol; the second data message is a data message after the first data message is compressed and carries compression identification information;
The second data message is sent to the server side, so that the server side decompresses the second data message according to the preset compression algorithm, and the first data message is recovered;
receiving a fourth data message sent by the server; the fourth data message is a data message compressed by the third data message and carries compression identification information; the third data message is generated after the first data message is subjected to basic database operation;
decompressing the fourth data message according to a preset compression algorithm, and recovering to obtain a third data message;
and sending the third data message to an application layer.
2. The method of claim 1, wherein after sending the first message to the server, the method further comprises:
receiving a second message sent by the server, wherein the second message is used for indicating whether the client and the server are successfully connected;
judging whether the connection between the client and the server is successfully established or not according to the second message;
if the connection is successful, the first data message is compressed according to a preset compression algorithm to generate a second data message when the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter.
3. The method according to claim 1, wherein the method further comprises:
and if the communication compression control parameter is zero or the data packet size of the first data message is smaller than or equal to the value of the compression threshold parameter, sending the first data message to the server.
4. A data compression transmission method, which is characterized by being applied to a server, the method comprising:
receiving a first message sent by a client, wherein the first message is used for requesting to establish connection with a server; the first message includes: communication compression control parameters and compression threshold parameters;
analyzing the first message to obtain the communication compression control parameter and the compression threshold parameter;
receiving a second data message sent by a client; the second data message is a data message after the first data message is compressed and carries compression identification information; the first data message is a message with a fixed format obtained by encoding request data sent by an application layer according to a preset communication protocol;
decompressing the second data message according to a preset compression algorithm, and recovering to obtain a first data message;
processing the first data message according to the decompression result to generate a third data message, wherein the third data message is generated after the first data message is subjected to database basic operation;
If the communication compression control parameter is not zero and the data packet size of the third data message is larger than the value of the compression threshold parameter, compressing the third data message according to a preset compression algorithm to generate a fourth data message; the fourth data message is the data message compressed by the third data message and carries compression identification information;
and sending the fourth data message to the client.
5. The method of claim 4, wherein after receiving the first message sent by the client, the method further comprises:
and sending a second message to the client, wherein the second message is used for indicating whether the client and the server are successfully connected.
6. A client, comprising:
the starting module is used for sending a first message to the server, wherein the first message is used for requesting to establish connection with the server; the first message includes: communication compression control parameters and compression threshold parameters;
the compression module is used for compressing the first data message according to a preset compression algorithm to generate a second data message if the communication compression control parameter is not zero and the data packet size of the first data message is larger than the value of the compression threshold parameter; the first data message is a message with a fixed format obtained by encoding request data sent by an application layer according to a preset communication protocol; the second data message is a data message after the first data message is compressed and carries compression identification information;
The sending module is used for sending the second data message to the server side so that the server side decompresses the second data message according to the preset compression algorithm and recovers the first data message;
the message receiving module is used for receiving a fourth data message sent by the server; the fourth data message is a data message compressed by the third data message and carries compression identification information; the third data message is generated after the first data message is subjected to basic database operation;
the message decompressing module is used for decompressing the fourth data message according to a preset compression algorithm and recovering to obtain a third data message;
and the message sending module is used for sending the third data message to an application layer.
7. A server, comprising:
the connection module is used for receiving a first message sent by the client, wherein the first message is used for requesting to establish connection with the server; the first message includes: communication compression control parameters and compression threshold parameters;
the analysis module is used for analyzing the first message and acquiring the communication compression control parameter and the compression threshold parameter;
The receiving module is used for receiving the second data message sent by the client; the second data message is a data message after the first data message is compressed and carries compression identification information; the first data message is a message with a fixed format obtained by encoding request data sent by an application layer according to a preset communication protocol;
the decompression module is used for decompressing the second data message according to a preset compression algorithm and recovering to obtain a first data message;
the processing module is used for processing the first data message according to the decompression result to generate a third data message, wherein the third data message is generated after the first data message is subjected to basic database operation;
the compression processing module is used for compressing the third data message according to a preset compression algorithm to generate a fourth data message if the communication compression control parameter is not zero and the data packet size of the third data message is larger than the value of the compression threshold parameter; the fourth data message is the data message compressed by the third data message and carries compression identification information;
and the sending result module is used for sending the fourth data message to the client.
8. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the data compression transmission method of any one of claims 1 to 5 when the computer program is executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111396502.7A CN114125071B (en) | 2021-11-23 | 2021-11-23 | Data compression transmission method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111396502.7A CN114125071B (en) | 2021-11-23 | 2021-11-23 | Data compression transmission method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114125071A CN114125071A (en) | 2022-03-01 |
| CN114125071B true CN114125071B (en) | 2024-02-09 |
Family
ID=80440064
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111396502.7A Active CN114125071B (en) | 2021-11-23 | 2021-11-23 | Data compression transmission method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114125071B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102111437A (en) * | 2010-12-30 | 2011-06-29 | 浪潮集团山东通用软件有限公司 | Simple object access protocol (SOAP) extension-based WEB service transmission compression method |
| CN102428483A (en) * | 2009-03-23 | 2012-04-25 | 生命力有限公司 | System and Method for Multi-Stream Video Compression |
| CN102790766A (en) * | 2012-06-29 | 2012-11-21 | 华为技术有限公司 | Object query method, object query system, object query device and object query acquisition device |
| US9532268B2 (en) * | 2014-11-19 | 2016-12-27 | Qualcomm Incorporated | Methods and apparatus for synchronizing a user equipment with an HFN offset |
| WO2017124837A1 (en) * | 2016-01-19 | 2017-07-27 | 深圳前海达闼云端智能科技有限公司 | Proxy method, server and client for sslvpn, and processing method thereof |
| CN107395716A (en) * | 2017-07-19 | 2017-11-24 | 河南神州数码索贝科技有限公司 | A kind of mobile terminal picture compression method for uploading and device |
| CN110572375A (en) * | 2019-08-21 | 2019-12-13 | 北京金堤科技有限公司 | IP address proxy method and device |
| CN110597461A (en) * | 2019-08-14 | 2019-12-20 | 阿里巴巴集团控股有限公司 | Data storage method, device and equipment in block chain type account book |
| CN110611680A (en) * | 2019-09-25 | 2019-12-24 | 腾讯科技（深圳）有限公司 | Data packet compression control method and device |
| US10694002B1 (en) * | 2017-04-27 | 2020-06-23 | EMC IP Holding Company LLC | Data compression optimization based on client clusters |
| CN111935261A (en) * | 2020-07-30 | 2020-11-13 | 北京达佳互联信息技术有限公司 | Response message processing method and device, electronic equipment and storage medium |
| CN112887429A (en) * | 2021-03-08 | 2021-06-01 | 深圳市车宝信息科技有限公司 | Data transmission method and device, electronic equipment and storage medium |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2365571C (en) * | 2001-12-18 | 2006-06-27 | Cognos Incorporated | System and method for retrieving data from a database system |
| US20080077702A1 (en) * | 2006-09-27 | 2008-03-27 | Joshua Posamentier | Dynamic server stream allocation |
| US9235618B2 (en) * | 2013-04-06 | 2016-01-12 | Citrix Systems, Inc. | Systems and methods for caching of SQL responses using integrated caching |
| CN103561082B (en) * | 2013-10-30 | 2017-01-18 | 北京奇虎科技有限公司 | Compressing request processing method and server |
| US9961157B2 (en) * | 2014-11-11 | 2018-05-01 | Dell Products L.P. | Adaptive compression management for web services |
| US11249987B2 (en) * | 2019-08-14 | 2022-02-15 | Advanced New Technologies Co., Ltd. | Data storage in blockchain-type ledger |
| CN111224864A (en) * | 2020-01-14 | 2020-06-02 | 平安国际智慧城市科技股份有限公司 | Message transmission method and device, computer equipment and storage medium |
-
2021
- 2021-11-23 CN CN202111396502.7A patent/CN114125071B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102428483A (en) * | 2009-03-23 | 2012-04-25 | 生命力有限公司 | System and Method for Multi-Stream Video Compression |
| CN102111437A (en) * | 2010-12-30 | 2011-06-29 | 浪潮集团山东通用软件有限公司 | Simple object access protocol (SOAP) extension-based WEB service transmission compression method |
| CN102790766A (en) * | 2012-06-29 | 2012-11-21 | 华为技术有限公司 | Object query method, object query system, object query device and object query acquisition device |
| US9532268B2 (en) * | 2014-11-19 | 2016-12-27 | Qualcomm Incorporated | Methods and apparatus for synchronizing a user equipment with an HFN offset |
| WO2017124837A1 (en) * | 2016-01-19 | 2017-07-27 | 深圳前海达闼云端智能科技有限公司 | Proxy method, server and client for sslvpn, and processing method thereof |
| US10694002B1 (en) * | 2017-04-27 | 2020-06-23 | EMC IP Holding Company LLC | Data compression optimization based on client clusters |
| CN107395716A (en) * | 2017-07-19 | 2017-11-24 | 河南神州数码索贝科技有限公司 | A kind of mobile terminal picture compression method for uploading and device |
| CN110597461A (en) * | 2019-08-14 | 2019-12-20 | 阿里巴巴集团控股有限公司 | Data storage method, device and equipment in block chain type account book |
| CN110572375A (en) * | 2019-08-21 | 2019-12-13 | 北京金堤科技有限公司 | IP address proxy method and device |
| CN110611680A (en) * | 2019-09-25 | 2019-12-24 | 腾讯科技（深圳）有限公司 | Data packet compression control method and device |
| CN111935261A (en) * | 2020-07-30 | 2020-11-13 | 北京达佳互联信息技术有限公司 | Response message processing method and device, electronic equipment and storage medium |
| CN112887429A (en) * | 2021-03-08 | 2021-06-01 | 深圳市车宝信息科技有限公司 | Data transmission method and device, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 微影用户中心性能调优探索与实践;靳松;现代电影技术;第38-42页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114125071A (en) | 2022-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9727574B2 (en) | System and method for applying an efficient data compression scheme to URL parameters | |
| US9426219B1 (en) | Efficient multi-part upload for a data warehouse | |
| CN106161633B (en) | Transmission method and system for packed files based on cloud computing environment | |
| EP3013015B1 (en) | Packet compression method and apparatus | |
| CN112261094B (en) | Message processing method and proxy server | |
| US11461276B2 (en) | Method and device for deduplication | |
| US11494437B1 (en) | System and method for performing object-modifying commands in an unstructured storage service | |
| CN115065725B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN113238912B (en) | Aggregation processing method for network security log data | |
| US11899624B2 (en) | System and method for random-access manipulation of compacted data files | |
| WO2013097812A1 (en) | Method and system for downloading font file | |
| EP2380098A1 (en) | Dictionary-based data compression and subsequent data transmission in a server / client architecture | |
| CN114125071B (en) | Data compression transmission method and device | |
| US11868616B2 (en) | System and method for low-distortion compaction of floating-point numbers | |
| CN115080826A (en) | Search query method, middleware, equipment and storage medium | |
| US9160820B2 (en) | Large volume data transfer | |
| US10168909B1 (en) | Compression hardware acceleration | |
| CN114070471B (en) | Test data packet transmission method, device, system, equipment and medium | |
| US11853262B2 (en) | System and method for computer data type identification | |
| WO2023019712A1 (en) | Zlib compression algorithm-based cloud computing resource manager communication delay optimization method | |
| US20230289354A1 (en) | Endpoint scan and profile generation | |
| CN117880389A (en) | Data monitoring method, device, equipment and medium for HTTP2 protocol | |
| Choi et al. | Network Traffic Reduction by Hypertext Compression. | |
| CN117215807A (en) | Caching method and device of interface query result, storage medium and computer equipment | |
| CN117135229A (en) | Log processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |