CN114449579A - Method, device and equipment for data compression - Google Patents
Method, device and equipment for data compression Download PDFInfo
- Publication number
- CN114449579A CN114449579A CN202011212245.2A CN202011212245A CN114449579A CN 114449579 A CN114449579 A CN 114449579A CN 202011212245 A CN202011212245 A CN 202011212245A CN 114449579 A CN114449579 A CN 114449579A
- Authority
- CN
- China
- Prior art keywords
- compression
- dictionary
- data
- model
- compression algorithm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 80
- 238000013144 data compression Methods 0.000 title claims abstract description 71
- 238000007906 compression Methods 0.000 claims abstract description 911
- 230000006835 compression Effects 0.000 claims abstract description 911
- 230000005540 biological transmission Effects 0.000 claims abstract description 202
- 238000013473 artificial intelligence Methods 0.000 claims description 217
- 238000004590 computer program Methods 0.000 claims description 23
- 238000000605 extraction Methods 0.000 claims description 20
- 238000003860 storage Methods 0.000 claims description 15
- 238000012549 training Methods 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 23
- 230000008569 process Effects 0.000 description 16
- 238000012545 processing Methods 0.000 description 14
- 230000006837 decompression Effects 0.000 description 12
- 230000007246 mechanism Effects 0.000 description 12
- 230000006870 function Effects 0.000 description 10
- 230000006854 communication Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 230000007774 longterm Effects 0.000 description 5
- 230000003993 interaction Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 239000003795 chemical substances by application Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 240000007594 Oryza sativa Species 0.000 description 1
- 235000007164 Oryza sativa Nutrition 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 235000009566 rice Nutrition 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W28/00—Network traffic management; Network resource management
- H04W28/02—Traffic management, e.g. flow control or congestion control
- H04W28/06—Optimizing the usage of the radio link, e.g. header compression, information sizing, discarding information
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a method, a device and equipment for data compression, wherein the method comprises the following steps: in the service transmission process, determining a currently adopted compression dictionary and a compression algorithm, wherein an initialized compression dictionary and an initialized compression algorithm are initially adopted, and then when determining that an updating condition is met, respectively and correspondingly updating the currently adopted compression dictionary and the currently adopted compression algorithm by using the compression dictionary and the compression algorithm output by the AI model; and based on the currently adopted compression dictionary, compressing or decompressing the transmitted service data by using the currently adopted compression algorithm. By utilizing the method disclosed by the invention, the compression dictionary and the compression algorithm are optimized through the learning and training of data, and the compression rate is improved.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a method, an apparatus, and a device for data compression.
Background
In an LTE (Long-Term Evolution)/LTE-a (Long-Term Evolution-advanced ) system, a network may configure a UE (User Equipment) to compress and transmit Uplink Data using a UDC (Uplink Data Compression) function, so as to reduce air interface resource overhead.
When uplink data compression is carried out, sending UE (user equipment) compresses data to be transmitted by using a preset dictionary or by using contents in a compression cache as a dictionary, so that the compression rate is further improved; correspondingly, the base station side decompresses the received data according to a preset dictionary or by using the previously received data as a dictionary.
In the existing UDC mechanism, the dictionary is generated by adopting the content in a compression cache as the dictionary, wherein the dictionary can be preset or can be all zero based on configuration in the compression cache, and when data is transmitted, the compression cache adopts a first-in first-out strategy and uses new data to replace the original data as a new dictionary. Although the above method utilizes the correlation between data, the best compression effect cannot be achieved. In addition, configured compression algorithms are used in the prior art UDC mechanism, and flexible compression algorithms are not considered for improving the compression rate.
Disclosure of Invention
The invention provides a method, a device and equipment for data compression, which solve the problems that in the prior art, the best compression effect cannot be achieved by an algorithm for generating a compression dictionary, and the adoption of a flexible compression algorithm is not considered so as to improve the compression ratio.
In a first aspect, the present invention provides a method for data compression, applied to a data transmission device, the method including:
in the service transmission process, determining a currently adopted compression dictionary and a compression algorithm, wherein an initialized compression dictionary and an initialized compression algorithm are initially adopted, and then when determining that an updating condition is met, respectively and correspondingly updating the currently adopted compression dictionary and the currently adopted compression algorithm by using the compression dictionary and the compression algorithm output by the AI model;
and based on the currently adopted compression dictionary, compressing or decompressing the transmitted service data by using the currently adopted compression algorithm.
Optionally, the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted feature and the service data, compress the current compression dictionary by using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust a model parameter of the AI model by using the highest compression ratio as a feedback input.
Optionally, the method further comprises:
initializing a currently adopted compression dictionary and a compression algorithm according to preset information; or
And when the service connection is established, initializing the currently adopted compression dictionary and compression algorithm according to the compression dictionary and compression algorithm output by the AI model when the service transmission is completed last time.
Optionally, determining that the update condition is satisfied includes at least one of:
when the service connection is established, the updating condition is met;
after the service connection is established, when a set updating period is reached, the updating condition is met;
and when determining that the event triggering condition is met, meeting the updating condition.
Optionally, determining that the event trigger condition is satisfied includes at least one of:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
Optionally, in the service transmission process, the method further includes:
acquiring a compression dictionary and a compression algorithm output by a local AI model; or
Acquiring a compression dictionary and a compression algorithm output by a local AI model, and sending the compression dictionary and the compression algorithm to opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from third-party equipment, wherein the third-party equipment is equipment of a functional node positioned on a cloud or edge; or
And acquiring a compression dictionary and a compression algorithm output by using the AI model from a third-party device, and sending the compression dictionary and the compression algorithm to an opposite-end data transmission device, wherein the third-party device is a device of a functional node positioned in the cloud or the edge.
Optionally, the method further comprises:
and sending the compression rate of the latest transmitted service data and the currently transmitted service data to the third-party equipment.
In a second aspect, the present invention provides a method for data compression, which is applied to a third-party device, and the method includes:
responding to the request of the data transmission equipment, and acquiring the latest service data transmitted by the current service and the compression rate of the service data which is currently transmitted;
inputting the latest transmitted service data into an AI model, and outputting a compression dictionary and a compression algorithm by using the AI model;
and sending the compression dictionary and the compression algorithm to the data transmission equipment.
Optionally, the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted feature and the service data, perform compression with a current compression dictionary by using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust a model parameter of the AI model by using the highest compression ratio as a feedback input.
Optionally, sending the compression dictionary and the compression algorithm to the data transmission device specifically includes:
and when the updating condition is determined to be met, sending the compression dictionary and the compression algorithm to the data transmission equipment.
Optionally, determining that the update condition is satisfied includes at least one of:
determining that the data transmission equipment meets the updating condition when establishing service connection;
determining that the data transmission equipment meets the updating condition when a set updating period is reached after the data transmission equipment establishes service connection;
and when determining that the event triggering condition is met, meeting the updating condition.
Optionally, determining that the event trigger condition is satisfied includes at least one of:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
In a third aspect, the present invention provides a data transmission device for data compression, comprising a memory, a transceiver, a processor:
a memory for storing a computer program; a transceiver for transceiving data under control of the processor; a processor for reading the computer program in the memory and performing the following operations:
in the service transmission process, determining a currently adopted compression dictionary and a compression algorithm, wherein an initialized compression dictionary and an initialized compression algorithm are initially adopted, and then when determining that an updating condition is met, respectively and correspondingly updating the currently adopted compression dictionary and the currently adopted compression algorithm by using the compression dictionary and the compression algorithm output by the AI model;
and based on the currently adopted compression dictionary, compressing or decompressing the transmitted service data by using the currently adopted compression algorithm.
Optionally, the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted feature and the service data, compress the current compression dictionary by using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust a model parameter of the AI model by using the highest compression ratio as a feedback input.
Optionally, the processor is further configured to:
initializing a currently adopted compression dictionary and a compression algorithm according to preset information; or
And when the service connection is established, initializing the currently adopted compression dictionary and compression algorithm according to the compression dictionary and compression algorithm output by the AI model when the service transmission is completed last time.
Optionally, the processor determines that the update condition is satisfied, including at least one of:
when the service connection is established, the updating condition is met;
after the service connection is established, when a set updating period is reached, the updating condition is met;
and when determining that the event triggering condition is met, meeting the updating condition.
Optionally, the processor determines that an event trigger condition is satisfied, including at least one of:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
Optionally, in the service transmission process, the processor is further configured to:
acquiring a compression dictionary and a compression algorithm output by a local AI model; or
Acquiring a compression dictionary and a compression algorithm output by a local AI model, and sending the compression dictionary and the compression algorithm to opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI model from opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from third-party equipment, wherein the third-party equipment is equipment of a functional node positioned on a cloud or edge; or
And acquiring a compression dictionary and a compression algorithm output by using the AI model from a third-party device, and sending the compression dictionary and the compression algorithm to an opposite-end data transmission device, wherein the third-party device is a device of a functional node positioned in the cloud or the edge.
Optionally, the processor is further configured to:
and sending the compression rate of the latest transmitted service data and the currently transmitted service data to the third-party equipment.
In a fourth aspect, the present invention provides a third party device for data compression, comprising:
a memory for storing a computer program; a transceiver for transceiving data under control of the processor; a processor for reading the computer program in the memory and performing the following operations:
responding to the request of the data transmission equipment, and acquiring the latest service data transmitted by the current service and the compression rate of the service data which is currently transmitted;
inputting the latest transmitted service data into an AI model, and outputting a compression dictionary and a compression algorithm by using the AI model;
and sending the compression dictionary and the compression algorithm to the data transmission equipment.
Optionally, the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted feature and the service data, compress the current compression dictionary by using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust a model parameter of the AI model by using the highest compression ratio as a feedback input.
Optionally, the compression dictionary and the compression algorithm are sent to the data transmission device, and the processor is specifically configured to:
and when the updating condition is determined to be met, sending the compression dictionary and the compression algorithm to the data transmission equipment.
Optionally, the processor determines that the update condition is satisfied, including at least one of:
determining that the data transmission equipment meets the updating condition when establishing service connection;
determining that the data transmission equipment meets the updating condition when a set updating period is reached after the data transmission equipment establishes service connection;
and when determining that the event triggering condition is met, meeting the updating condition.
Optionally, the processor determines that the event trigger condition is satisfied, including at least one of:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
In a fifth aspect, the present invention provides an apparatus for data compression, comprising:
the dictionary algorithm determining unit is used for determining a compression dictionary and a compression algorithm which are adopted currently in the service transmission process, wherein the compression dictionary and the compression algorithm which are initialized are adopted initially, and then when the updating condition is met, the compression dictionary and the compression algorithm which are output by the AI model are utilized to respectively correspondingly update the compression dictionary and the compression algorithm which are adopted currently;
and the compression unit is used for compressing or decompressing the transmitted service data by using the currently adopted compression algorithm based on the currently adopted compression dictionary.
Optionally, the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted feature and the service data, compress the current compression dictionary by using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust a model parameter of the AI model by using the highest compression ratio as a feedback input.
Optionally, the dictionary algorithm determining unit is further configured to:
initializing a currently adopted compression dictionary and a compression algorithm according to preset information; or
And when the service connection is established, initializing the currently adopted compression dictionary and compression algorithm according to the compression dictionary and compression algorithm output by the AI model when the service transmission is completed last time.
Optionally, the dictionary algorithm determining unit determines that the update condition is satisfied, and includes at least one of the following steps:
when the service connection is established, the updating condition is met;
after the service connection is established, when a set updating period is reached, the updating condition is met;
and when determining that the event triggering condition is met, meeting the updating condition.
Optionally, the dictionary algorithm determination unit determines that the event triggering condition is satisfied, and includes at least one of the following steps:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
Optionally, in the service transmission process, the dictionary algorithm determining unit is further configured to:
acquiring a compression dictionary and a compression algorithm output by a local AI model; or
Acquiring a compression dictionary and a compression algorithm output by a local AI model, and sending the compression dictionary and the compression algorithm to opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from third-party equipment, wherein the third-party equipment is equipment of a functional node positioned on a cloud or edge; or
And acquiring a compression dictionary and a compression algorithm output by using the AI model from a third-party device, and sending the compression dictionary and the compression algorithm to an opposite-end data transmission device, wherein the third-party device is a device of a functional node positioned in the cloud or the edge.
Optionally, the compression unit is further configured to:
and sending the compression rate of the latest transmitted service data and the currently transmitted service data to the third-party equipment.
In a sixth aspect, the present invention provides an apparatus for data compression, comprising:
the data receiving unit is used for responding to the request of the data transmission equipment and acquiring the latest transmitted service data of the current service and the compression rate of the service data which is currently transmitted;
the dictionary algorithm generating unit is used for inputting the latest transmitted service data into the AI model and outputting a compression dictionary and a compression algorithm by utilizing the AI model;
and the data sending unit is used for sending the compression dictionary and the compression algorithm to the data transmission equipment.
Optionally, the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted feature and the service data, compress the current compression dictionary by using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust a model parameter of the AI model by using the highest compression ratio as a feedback input.
Optionally, the compression dictionary and the compression algorithm are sent to the data transmission device, and the data sending unit is specifically configured to:
and when the updating condition is determined to be met, sending the compression dictionary and the compression algorithm to the data transmission equipment.
Optionally, the determining, by the data sending unit, that the update condition is satisfied includes at least one of:
determining that the data transmission equipment meets the updating condition when establishing service connection;
determining that the data transmission equipment meets the updating condition when a set updating period is reached after the data transmission equipment establishes service connection;
and when determining that the event triggering condition is met, meeting the updating condition.
Optionally, the data sending unit determines that the event trigger condition is satisfied, and includes at least one of the following steps:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
In a seventh aspect, the present invention provides a computer program medium having a computer program stored thereon, which when executed by a processor, performs the steps of a method for data compression as provided in the first aspect above.
In an eighth aspect, the present invention provides a chip, where the chip is coupled to a memory in a device, so that the chip invokes, when running, program instructions stored in the memory to implement the above aspects of the embodiments of the present application and any method for data compression that may be involved in the aspects.
In a ninth aspect, the present invention provides a computer program product, which, when run on an electronic device, causes the electronic device to execute a method for performing data compression, which implements the above aspects of the embodiments of the present application and any one of the aspects related to the above aspects.
The method, the device and the equipment for data compression have the following beneficial effects that:
in the data transmission process, the compression dictionary and the compression algorithm output by the AI model are respectively and correspondingly updated, the compression dictionary and the compression algorithm which are currently adopted are optimized, the flexible compression algorithm is adopted, the compression dictionary and the compression algorithm are synchronously updated by the sending end and the receiving end of the data, the data are compressed and decompressed by the updated compression dictionary and compression algorithm, and the compression ratio is improved.
Drawings
FIG. 1 is a schematic diagram of a system for data compression according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a compression dictionary and a compression algorithm that are output by a base station and used for acquiring a local AI model according to an embodiment of the present invention to perform data compression;
fig. 3 is a schematic diagram of data compression performed by obtaining a compression dictionary and a compression algorithm output by an AI model from a third-party device according to an embodiment of the present invention;
fig. 4 is a schematic diagram of data compression performed by acquiring a compression dictionary and a compression algorithm output by an AI model from a third-party device and updating conditions for determining establishment of a service connection according to an embodiment of the present invention;
fig. 5 is a flowchart of a method for data compression by a data transmission device according to an embodiment of the present invention;
fig. 6 is a flowchart of a method for data compression performed by a third-party device according to an embodiment of the present invention;
fig. 7 is a schematic diagram of a data transmission device for performing data compression according to an embodiment of the present invention;
fig. 8 is a schematic diagram of a third-party device for data compression according to an embodiment of the present invention;
fig. 9 is a schematic diagram of an apparatus for data compression by a data transmission device according to an embodiment of the present invention;
fig. 10 is a schematic diagram of an apparatus for performing data compression by a third-party device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In an LTE (Long-Term Evolution)/LTE-a (Long-Term Evolution-advanced ) system, a network may configure a UE (User Equipment) to compress and transmit Uplink Data using a UDC (Uplink Data Compression) function, so as to reduce air interface resource overhead.
When uplink data compression is carried out, sending UE (user equipment) compresses data to be transmitted by using a preset dictionary or by using contents in a compression cache as a dictionary, so that the compression rate is further improved; correspondingly, the base station side decompresses the received data according to a preset dictionary or by using the previously received data as a dictionary.
The UDC compression mechanism and the preset dictionary based compression mechanism are described in detail below:
1) UDC compression mechanism
1.1) a sending end maintains a compression cache, a receiving end maintains a decompression cache, and the compression cache and the decompression cache are both first-in first-out queues;
1.2) before sending data, the sending end compresses the data to be sent:
a) in a data packet to be sent, a target field meeting the following characteristics is searched for:
the length exceeds a preset threshold;
the destination field is the same as some field in the compression cache or in the packet that precedes the destination field.
b) If the target field is found, replacing the target field with an offset and length combination:
the offset is the position offset between the target field and the same field before the target field;
the length is the length of the target field;
because the length of the offset and length combination is shorter than the length of the target field itself, the effect of compression is achieved. Within a packet, there may be multiple fields that meet the above characteristics, and all of these fields may be compressed.
1.3) the sending end sends the compressed data packet to the opposite end; meanwhile, filling the corresponding original data packet, namely uncompressed data, into a compression cache;
1.4) the receiving end decompresses the received data packet based on the offset and the length and the decompression buffer; and then, filling the decompressed data packet into a decompression buffer.
2) Preset dictionary based compression mechanism
As an optimization of the UDC, a preset dictionary based compression mechanism can write a preset dictionary of fields appearing in high frequency based on service characteristics, and before the UDC is started, the preset dictionary is respectively stored in compression and decompression caches of a compression end and a decompression section.
Therefore, when the UDC is just started, the compression and decompression cache is not empty any more, but a preset dictionary with high-frequency fields is stored, the discovery probability of the target fields can be effectively improved, and the compression rate is improved.
Obviously, in order to implement the above mechanism, before the UDC is started, the terminal and the base station need to respectively acquire the preset dictionaries to be used, that is, the preset dictionary synchronization process is completed.
Based on the principle of the preset dictionary-based compression mechanism, when the UDC is just started, the compression and decompression cache is possibly configured to be empty, the probability that the compression end finds the target field in the current packet to be sent is low, and the compression rate is correspondingly low; after the UDC is operated for a period of time, the compression cache is gradually increased, the probability that the compression end finds the target field in the current packet to be sent is improved, and the compression rate can be correspondingly improved.
In the existing UDC mechanism, the dictionary is generated by adopting the content in a compression cache as the dictionary, wherein the dictionary can be preset or can be all zero based on configuration in the compression cache, when data is transmitted, the compression cache adopts a first-in first-out strategy, and new data is used for replacing original data to serve as a new dictionary. Although the above method utilizes the correlation between data, the best compression effect cannot be achieved. In addition, configured compression algorithms are used in the prior art UDC mechanism, and flexible compression algorithms are not considered for improving the compression rate.
In view of the above problems, embodiments of the present application provide a method, an apparatus, and a device for data compression, which optimize a compression dictionary and a compression algorithm by learning and training data, thereby improving a compression rate. Embodiments of a method, an apparatus, and a device for data compression according to embodiments of the present invention are given below.
Example 1
As shown in fig. 1, an embodiment of the present invention provides a schematic diagram of a system for data compression, including:
the first data transmission device 101 serving as a sending end is used for determining a currently-used compression dictionary and compression algorithm in a service transmission process, wherein the initialized compression dictionary and compression algorithm are initially used, and then when an update condition is met, the currently-used compression dictionary and compression algorithm are updated correspondingly by using the compression dictionary and compression algorithm output by the AI model;
based on the currently adopted compression dictionary, compressing the transmitted service data by using the currently adopted compression algorithm;
it should be noted that, when the first data transmission device 101 serves as a sending end, a compression operation is performed; when the first data transmission apparatus 101 described above functions as a receiving side, a decompression operation is performed.
The second data transmission device 102 serving as the receiving end is configured to determine a currently-used compression dictionary and compression algorithm in a service transmission process, where the compression dictionary and compression algorithm that are initialized are initially used, and then when it is determined that an update condition is satisfied, the compression dictionary and compression algorithm that are output by the AI model are used to respectively update the currently-used compression dictionary and compression algorithm;
based on the currently adopted compression dictionary, decompressing the transmitted service data by using the currently adopted compression algorithm;
it should be noted that, when the second data transmission device 102 serves as a sending end, a compression operation is performed; when the second data transmission device 102 is used as a receiving end, a decompression operation is performed.
It should be noted that the AI model is used to extract the features of the latest transmitted service data, output a compression dictionary according to the association between the extracted features and the service data, compress the current compression dictionary by using different compression algorithms, output the compression algorithm corresponding to the highest compression ratio, and adjust the model parameters of the AI model by using the highest compression ratio as the feedback input.
It should be noted that, during service transmission, identities of the first data transmission device 101 and the second data transmission device 102 as a sending end and a receiving end may be changed, for example, after the first data transmission device 101 compresses service data by using a currently-used compression dictionary and compression algorithm, the service data is sent to the second data transmission device 102, and the second data transmission device 102 decompresses the transmitted service data by using the currently-used compression dictionary and compression algorithm, at this time, the first data transmission device 101 is a sending end of data, and the second data transmission device 102 is a receiving end of data; the second data transmission device 102 compresses the service data using the currently used compression dictionary and compression algorithm, and then sends the compressed service data to the first data transmission device 101, where the first data transmission device 101 decompresses the transmitted service data using the currently used compression dictionary and compression algorithm, and at this time, the first data transmission device 101 is a data receiving end, and the second data transmission device 102 is a data sending end.
As an optional implementation manner, in the service transmission process, the method further includes:
(1) the first data transmission device 101 obtains a compression dictionary and a compression algorithm output by a local AI model, and sends the obtained compression dictionary and compression algorithm to the second data transmission device 102;
(2) the second data transmission device 102 obtains a compression dictionary and a compression algorithm output by the local AI model, and sends the compression dictionary and the compression algorithm to the first data transmission device 101;
(3) the first data transmission device 101 and/or the second data transmission device 102 obtain a compression dictionary and a compression algorithm output by using the AI model from a third-party device, where the third-party device is a device located in a functional node of a cloud or an edge.
It should be noted that, when the third-party device sends the output compression dictionary and compression algorithm to any one of the first data transmission device 101 and the second data transmission device 102, the data transmission device that receives the compression dictionary and compression algorithm sends the compression dictionary and compression algorithm to the opposite-end data transmission device.
As an alternative embodiment, the transmitting of the compression dictionary and the compression algorithm includes:
transmitting a compression dictionary and a compression algorithm through RRC message;
transmitting the compression dictionary and the compression algorithm through the control unit MAC CE;
sending a compression dictionary and a compression algorithm by carrying the compression dictionary and the compression algorithm in a Packet Data Convergence Protocol (PDCP) header of a first data packet;
and transmitting the compression dictionary and the compression algorithm by carrying the compression dictionary and the compression algorithm indication information in the PDCP header of the first data packet and carrying the compression dictionary and the compression algorithm by using the PDCP subPDU.
It should be noted that, for any one of the sending end and the receiving end of the data transmission device, the sending end and the receiving end itself output a compression dictionary and a compression algorithm by using the AI model, and the new compression dictionary and/or compression algorithm are directly transmitted between the sending end and the receiving end; for a case where a third-party device outputs a compression dictionary and a compression algorithm by using an AI model, for example, the third-party device is a functional node of a cloud or an edge, and the transmitting end and the receiving end acquire the compression dictionary and/or the compression algorithm from the functional node.
When the embodiment of (3) above is adopted, wherein the compression dictionary and the compression algorithm are received from the third-party device, the system further includes:
the third party equipment 103 is used for responding to the request of the data transmission equipment and acquiring the latest transmission service data of the current service and the compression rate of the service data which finishes transmission at present; inputting the latest transmitted service data into an AI model, and outputting a compression dictionary and a compression algorithm by using the AI model; and sending the compression dictionary and the compression algorithm to the data transmission equipment.
It should be noted that the AI model is used to extract the features of the latest transmitted service data, output a compression dictionary according to the association between the extracted features and the service data, compress the current compression dictionary by using different compression algorithms, output the compression algorithm corresponding to the highest compression ratio, and adjust the model parameters of the AI model by using the highest compression ratio as the feedback input.
As an alternative embodiment, the data transmission device sends a request message requesting the compression dictionary and the compression algorithm to the third-party device.
As an optional implementation, for the above implementation of receiving the compression dictionary and the compression algorithm from the third-party device, the method further includes:
and sending the compression rate of the latest transmitted service data and the currently transmitted service data to the third-party equipment.
It should be noted that the operation of sending the compression ratios of the latest transmitted service data and the currently transmitted service data to the third-party device may be executed by any one of the sending end and the receiving end of the data transmission device.
As an optional implementation manner, the third party device actively senses compression rates of the latest transmitted service data and the currently completed transmission service data.
As an optional implementation manner, sending the compression dictionary and the compression algorithm to the data transmission device specifically includes:
and when the updating condition is determined to be met, sending the compression dictionary and the compression algorithm to the data transmission equipment.
It should be noted that the above system architecture is only an example of the system architecture applicable to the embodiment of the present invention, and the system architecture applicable to the embodiment of the present invention may also add other entities or reduce part of the entities compared to the system architecture shown in fig. 1.
As an optional implementation manner, the data transmission device 101 is a user terminal UE, the data transmission device 102 is a base station, and the third-party device 103 is a cloud or edge function node, such as an AI compression server, in which the AI model is deployed.
The user terminal UE according to the embodiments of the present application may be a device providing voice and/or data connectivity to a user, a handheld device having a wireless connection function, or another processing device connected to a wireless modem. The names of the terminal devices may also be different in different systems, for example, in a 5G system, the terminal devices may be referred to as User Equipments (UEs). Wireless terminal devices, which may be mobile terminal devices such as mobile telephones (or "cellular" telephones) and computers having mobile terminal devices, for example, portable, pocket, hand-held, computer-included, or vehicle-mounted mobile devices, may communicate with one or more core networks via a Radio Access Network (RAN). Examples of such devices include Personal Communication Service (PCS) phones, cordless phones, Session Initiated Protocol (SIP) phones, Wireless Local Loop (WLL) stations, Personal Digital Assistants (PDAs), and the like. The wireless terminal device may also be referred to as a system, a subscriber unit (subscriber unit), a subscriber station (subscriber station), a mobile station (mobile), a remote station (remote station), an access point (access point), a remote terminal device (remote terminal), an access terminal device (access terminal), a user terminal device (user terminal), a user agent (user agent), and a user device (user device), which are not limited in this embodiment of the present application.
The base stations referred to in embodiments of the present application may also be referred to as access points, or may refer to devices in an access network that communicate over the air-interface, through one or more sectors, with wireless terminal devices, or by other names, depending on the particular application. The network device may be configured to interconvert received air frames and Internet Protocol (IP) packets as a router between the wireless terminal device and the rest of the access network, which may include an Internet Protocol (IP) communication network. The network device may also coordinate attribute management for the air interface. For example, the network device according to the embodiment of the present application may be a Base Transceiver Station (BTS) in a Global System For Mobile communications (GSM) or a Code Division Multiple Access (CDMA), may also be a network device (NodeB) in a Wideband Code Division Multiple Access (WCDMA), may also be an evolved Node b (eNB or e-NodeB) in a Long Term Evolution (LTE) System, a 5G Base Station in a 5G network architecture (next generation System), and may also be a home evolved Node b (HeNB), a Relay Node (Relay Node), a home Base Station (femto), a pico Base Station (pico), and the like, which are not limited in the embodiment of the present application.
As an optional implementation, the AI model is integrated in an AI module, and the AI module may implement:
(1) generating an AI model of a compression dictionary and a compression algorithm;
(2) compression rate feedback, namely compressing the transmission data by using an AI compression dictionary and a compression algorithm, calculating the compression rate and feeding the compression rate back to the AI training model;
(3) and performing AI learning on the new transmission data by using an AI training model, and generating a new compression dictionary and a new compression algorithm.
It should be noted that the AI module may be located at any one of a sending end and a receiving end of the data transmission device, or located in a third-party device, for example, a cloud or an edge functional node.
As an optional implementation manner, in the above three implementation manners, the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted feature and the service data, perform compression with a current compression dictionary by using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust a model parameter of the AI model by using the highest compression ratio as a feedback input.
It should be noted that, the AI model may adopt an existing feedback neural network model, continuously adjust parameters of the AI model through a self-learning mechanism, continuously enhance the capability of performing dictionary word extraction by analyzing the correlation characteristics of input data, and adjust the dictionary recognition effect to the forward direction of improving the compression ratio along with the continuous input of business data. The AI model calculates the data compression rate using different compression algorithms that are existing algorithms, such as huffman coding, Rice coding, run-length coding, etc. In addition, the compression rate feedback is specifically input into the AI model, and the process of adjusting the model parameters may be performed in an existing manner, which is not described in detail herein.
Particularly, when the data transmission device initially establishes a connection and performs service transmission for the first time, it needs to determine that an initialized compression dictionary and compression algorithm are initially used.
As an optional implementation manner, according to preset information, initializing a currently adopted compression dictionary and a compression algorithm; or
And when the service connection is established, initializing the currently adopted compression dictionary and compression algorithm according to the compression dictionary and compression algorithm output by the AI model when the service transmission is completed last time.
It should be noted that, the initializing a currently used compression dictionary according to the preset information includes:
initializing a currently adopted compression dictionary as a compression dictionary configured based on business characteristics according to preset information; or
And initializing the currently adopted compression dictionary to be empty according to the preset condition.
It should be noted that, the initializing a currently adopted compression algorithm according to the preset information includes:
initializing a currently adopted compression algorithm as a default compression algorithm according to preset information; or
Initializing the currently adopted compression algorithm into a pre-configured compression algorithm according to preset information; or
And initializing the currently adopted compression algorithm into the compression algorithm selected according to the service characteristics according to the preset information.
It should be noted that, the AI model extracts features of each transmitted service data, outputs a compression dictionary according to the correlation between the extracted features and the service data, and compresses the current compression dictionary by using different compression algorithms to output a compression algorithm corresponding to the highest compression ratio, that is, adjusts the compression dictionary and the compression algorithm according to each service transmitted data, but only when an update condition is satisfied, the compression dictionary and the compression algorithm output by the AI model are used to respectively update the currently used compression dictionary and compression algorithm.
As an optional implementation manner, determining that the update condition is satisfied includes at least one of the following steps:
when the service connection is established, the updating condition is met;
after the service connection is established, when a set updating period is reached, the updating condition is met;
and when determining that the event triggering condition is met, meeting the updating condition.
It should be noted that the determination that the update condition is satisfied includes any one or more of the above three conditions, that is, the determination that the update condition is satisfied is to determine that one of the above three conditions or any combination of any two of the above three conditions or three conditions is satisfied.
Specific embodiments are given for the three methods for determining that the update condition is satisfied:
embodiment a: the update is determined by determining to establish a traffic connection.
And for each service for compression transmission, synchronizing a compression dictionary and a compression algorithm for the service once after the service connection is established, and not updating the compression dictionary and the compression algorithm in the communication process.
And updating the compression dictionary and the compression algorithm based on the transmission data and the compression rate for the communication of the service next time, namely, updating the compression dictionary and the compression algorithm once only after the service connection is established, wherein the compression dictionary and the compression algorithm used in the service connection process are kept unchanged.
Embodiment b: and determining to update the service connection by determining that the set update period is reached after the service connection is established.
After the service connection is established for each service for data compression transmission, the compression dictionary and the compression algorithm are synchronized once for the service, the compression dictionary and the compression algorithm are updated periodically in the communication process, when the period is reached, the compression dictionary and the compression algorithm output by the current model are used for updating, and the compression dictionary and the compression algorithm used in one period are kept unchanged.
Embodiment c: the update is determined by determining that an eventing trigger condition is satisfied.
As an optional implementation, determining that the event trigger condition is satisfied includes at least one of the following steps:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
It should be noted that the expected compression ratio of the compression dictionary and the compression algorithm output by the current AI model is an expected compression ratio for compressing the latest transmitted service data by using the compression dictionary and the compression algorithm output by the current AI model.
It should be noted that the event triggering condition is only an example, and is not specifically limited, and the specific event triggering condition may be set according to a specific implementation, for example, the event triggering condition is set to a compression ratio expected by a compression dictionary and a compression algorithm output by the current AI model, and when the expected compression ratio is greater than a preset value, it is determined that the event triggering condition is satisfied.
Based on the foregoing embodiments, the embodiments of the present invention provide three specific implementation manners, and specifically describe the foregoing method for performing data compression.
The first implementation mode comprises the following steps: and the base station acquires a compression dictionary and a compression algorithm output by the local AI model.
As shown in fig. 2, an embodiment of the present invention provides a schematic diagram of a compression dictionary and a compression algorithm, which are output by a base station to obtain a local AI model, for data compression.
The PDCP (Packet Data Convergence Protocol) layer is responsible for Data compression and decompression, and performs uplink and downlink compressed Data transmission at the Uu interface, and the base station obtains a compression dictionary and a compression algorithm output by the local AI model, and the updated compression dictionary and/or compression algorithm are directly synchronized via the Uu interface.
It should be noted that the transmission method is adopted in both the following embodiments 2 and 3, and details thereof are not repeated.
In the present embodiment, the data transmission devices are a base station and a user terminal UE.
Step 1: and establishing connection between the user terminal UE and the base station.
It should be noted that, the interaction process between the UE and the core network is not described here, and the interaction between the UE and the core network is completed before step 2.
Step 2: the base station determines the initialized compression dictionary and compression algorithm to be used.
It should be noted that the initialized compression dictionary and compression algorithm may be determined according to preset information; or according to the compression dictionary and the compression algorithm output by the AI model when the service transmission is completed last time.
And step 3: and the base station sends the initialized compression dictionary and compression algorithm to the user terminal UE.
It should be noted that the sending method may be RRC message, MAC CE, where the PDCP header of the first packet carries the AI compression dictionary and the compression algorithm, or the PDCP subPDU carries the AI compression dictionary and the compression algorithm indication information, and the PDCP subPDU carries the AI compression dictionary and the compression algorithm.
And 4, step 4: the data sending end in the base station and the user terminal UE uses the initialized compression dictionary and compression algorithm to compress and transmit the data, and the receiving end uses the initialized compression dictionary and compression algorithm to decompress the data, and counts the compression ratio in the process.
It should be noted that, when the base station/UE is a data sending end, the initialized compression dictionary and compression algorithm are used to compress data, and the compressed data is sent to the UE/base station; and when the base station/user terminal UE is a data receiving end, receiving compressed data sent by the user terminal UE/base station, and decompressing the compressed data by using the initialized compression dictionary and compression algorithm.
It should be noted that the user terminal UE sends the compression rate of the statistics to the base station.
And 5: and the base station adjusts the training model based on the transmitted data and the data compression rate and generates a new compression dictionary and a new compression algorithm.
It should be noted that, when the base station is a data sending end, the data to be transmitted is input into an AI model, the AI model is used to perform feature extraction on the latest transmitted service data, a compression dictionary is output according to the extracted features and the correlation between the service data, the current compression dictionary is compressed by using different compression algorithms, the compression algorithm corresponding to the highest compression ratio is output, the highest compression ratio is used as feedback input, and the model parameters of the AI model are adjusted; when the base station is a data receiving end, receiving compressed data sent by User Equipment (UE), decompressing the compressed data by using the initialized compression dictionary and compression algorithm, inputting the decompressed data into an AI model, extracting the characteristics of the latest transmitted service data by using the AI model, outputting the compression dictionary according to the extracted characteristics and the correlation between the service data, compressing the current compression dictionary by using different compression algorithms, outputting the compression algorithm corresponding to the highest compression ratio, and adjusting the model parameters of the AI model by using the highest compression ratio as feedback input.
It should be noted that the process of adjusting the training model and generating a new compression dictionary and compression algorithm is continuously performed during the data transmission process.
Step 6: and when the condition that the updating condition is met is determined, the base station sends the updated compression dictionary and compression algorithm to the user terminal UE.
The update condition of step 6 may be based on a cycle or an event trigger.
It should be noted that the sending method used in step 6 is the same as that in step 3, and is not described again.
And 7: the data sending end in the base station and the user terminal UE uses the new compression dictionary and the new compression algorithm to carry out compression transmission on the data, the receiving end uses the new compression dictionary and the new compression algorithm to carry out decompression on the data, and the compression ratio is counted in the process.
It should be noted that the above embodiment is also applicable to an embodiment in which the user terminal UE obtains a compression dictionary and a compression algorithm output by the local AI model to perform data compression, and the above operation of the base station and the user terminal UE can be exchanged in the above process.
The second embodiment: and acquiring a compression dictionary and a compression algorithm which are output by using the AI model from the third-party equipment.
As shown in fig. 3, an embodiment of the present invention provides a schematic diagram of obtaining a compression dictionary and a compression algorithm output by an AI model from a third-party device for data compression.
It should be noted that, in this embodiment, the data transmission device is a base station and a user terminal UE, and the third party device is a cloud or an edge AI compression server.
Step 1: and establishing connection between the user terminal UE and the base station.
It should be noted that, for simplifying the description, the interaction process between the UE and the core network is not described here, and the interaction between the UE and the core network is completed before step 2.
Step 2a/2 b: the user terminal UE and/or the base station request the compression dictionary and the compression algorithm from the AI compression server.
It should be noted that, the above steps may be that both sides of the data transmission device request the AI compression server, or that one side of the data transmission device requests the AI compression server.
And 3, step 3: the AI compression server determines the initialized compression dictionary and compression algorithm to employ.
It should be noted that the initialized compression dictionary and compression algorithm may be determined according to preset information; or according to the compression dictionary and the compression algorithm output by the AI model when the service transmission is completed last time.
Step 4a/4 b: and the AI compression server sends the initialized compression dictionary and compression algorithm to the base station and the user terminal UE.
It should be noted that, in this step, the AI compression server may send the compression dictionary and the compression algorithm to both sides of the data transmission device, or one of the two sides may obtain the compression dictionary and the compression algorithm from the AI compression server and send them to the opposite side.
And 5: the data sending end in the base station and the user terminal UE uses the initialized compression dictionary and compression algorithm to compress and transmit the data, and the receiving end uses the initialized compression dictionary and compression algorithm to decompress the data, and counts the compression ratio in the process.
It should be noted that, when the base station/UE is a data sending end, the initialized compression dictionary and compression algorithm are used to compress data, and the compressed data is sent to the UE/base station; and when the base station/user terminal UE is a data receiving end, receiving compressed data sent by the user terminal UE/base station, and decompressing the compressed data by using the initialized compression dictionary and compression algorithm.
It should be noted that, in this process, the base station and/or the user terminal UE sends the latest transmitted service data and the compression rate of the currently transmitted service data to the AI compression server.
Step 6: and the AI compression server adjusts the training model based on the compression rate of the latest transmitted service data and the currently transmitted service data, and generates a new compression dictionary and a new compression algorithm.
It should be noted that the AI compression server receives the compression rates of the latest transmitted service data and the currently transmitted service data sent by the base station and/or the user terminal UE.
It should be noted that, the AI compression server inputs the latest transmitted service data into the AI model, performs feature extraction on the latest transmitted service data by using the AI model, outputs a compression dictionary according to the extracted features and the correlation between the service data, compresses the current compression dictionary by using different compression algorithms, outputs the compression algorithm corresponding to the highest compression ratio, and adjusts the model parameters of the AI model by using the highest compression ratio as the feedback input.
Step 7a/7 b: and when the fact that the updating condition is met is determined, the AI compression server sends the updated compression dictionary and compression algorithm to the base station and the user terminal UE.
It should be noted that the update condition of the step 7a/7b may be based on a cycle, or may be based on an event trigger.
It should be noted that, the above-mentioned sending method may be that the AI compression server sends the compression dictionary and the compression algorithm to both sides of the data transmission device, or one of the sides obtains the compression dictionary and the compression algorithm from the AI compression server and sends them to the opposite side.
And 8: the data sending end in the base station and the user terminal UE uses the new compression dictionary and the new compression algorithm to compress and transmit the data, the receiving end uses the new compression dictionary and the new compression algorithm to decompress the data, and the compression ratio is counted in the process, and the data and the compression ratio are fed back to the AI compression server.
As an optional implementation manner, on the basis of the second implementation manner, a step is added after the step 5: for the connection release of the service, a step is added after the step 6: connection is again established for this service, and the steps: and the user terminal UE and/or the base station requests the compression dictionary and the compression algorithm from the AI compression server to obtain a third implementation mode that the updating condition is determined to establish service connection. As shown in fig. 4, an embodiment of the present invention provides a schematic diagram of obtaining a compression dictionary and a compression algorithm output by using an AI model from a third-party device, and updating conditions to determine to establish a service connection for data compression.
Example 2
An embodiment of the present invention provides a flow chart of a method for data compression by a data transmission device, as shown in fig. 5, including:
step S501, in the service transmission process, determining a currently adopted compression dictionary and a compression algorithm, wherein an initialized compression dictionary and an initialized compression algorithm are adopted initially, and then when determining that an updating condition is met, updating the currently adopted compression dictionary and the currently adopted compression algorithm correspondingly by using the compression dictionary and the compression algorithm output by an AI model;
step S502, based on the currently adopted compression dictionary, the transmitted service data is compressed or decompressed by using the currently adopted compression algorithm.
Optionally, the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted feature and the service data, compress the current compression dictionary by using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust a model parameter of the AI model by using the highest compression ratio as a feedback input.
Optionally, the method further comprises:
initializing a currently adopted compression dictionary and a compression algorithm according to preset information; or
And when the service connection is established, initializing the currently adopted compression dictionary and compression algorithm according to the compression dictionary and compression algorithm output by the AI model when the service transmission is completed last time.
Optionally, determining that the update condition is satisfied includes at least one of:
when the service connection is established, the updating condition is met;
after the service connection is established, when a set updating period is reached, the updating condition is met;
and when determining that the event triggering condition is met, meeting the updating condition.
Optionally, determining that the event trigger condition is satisfied includes at least one of:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
Optionally, in the service transmission process, the method further includes:
acquiring a compression dictionary and a compression algorithm output by a local AI model; or
Acquiring a compression dictionary and a compression algorithm output by a local AI model, and sending the compression dictionary and the compression algorithm to opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from third-party equipment, wherein the third-party equipment is equipment of a functional node positioned on a cloud or edge; or
And acquiring a compression dictionary and a compression algorithm output by using the AI model from a third-party device, and sending the compression dictionary and the compression algorithm to an opposite-end data transmission device, wherein the third-party device is a device of a functional node positioned in the cloud or the edge.
Optionally, the method further comprises:
and sending the compression rate of the latest transmitted service data and the currently transmitted service data to the third-party equipment.
An embodiment of the present invention provides a flowchart of a method for data compression by a third-party device, as shown in fig. 6, including:
step S601, responding to the request of the data transmission equipment, obtaining the latest transmission service data of the current service and the compression rate of the service data which completes the transmission currently;
step S602, inputting the latest transmitted service data into an AI model, and outputting a compression dictionary and a compression algorithm by using the AI model;
step S603, sending the compression dictionary and the compression algorithm to the data transmission device.
Optionally, the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted feature and the service data, compress the current compression dictionary by using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust a model parameter of the AI model by using the highest compression ratio as a feedback input.
Optionally, the sending the compression dictionary and the compression algorithm to the data transmission device specifically includes:
and when the updating condition is determined to be met, sending the compression dictionary and the compression algorithm to the data transmission equipment.
Optionally, determining that the update condition is satisfied includes at least one of:
determining that the data transmission equipment meets the updating condition when establishing service connection;
determining that the data transmission equipment meets the updating condition when a set updating period is reached after the data transmission equipment establishes service connection;
and when determining that the event triggering condition is met, meeting the updating condition.
Optionally, determining that the event trigger condition is satisfied includes at least one of:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
The data transmission device for performing data compression provided in the embodiment of the present invention is the same inventive concept as the data transmission device in embodiment 1 of the present invention, and various embodiments of data compression performed by the data transmission device in the system provided in the embodiment may be applied to the method for performing data compression in the embodiment, and are not repeated here.
The third-party device for data compression provided in the embodiment of the present invention is the same as the third-party device in embodiment 1 of the present invention, and various implementation manners of data compression performed by the third-party device in the system provided in the embodiment may be applied to the method for data compression in the embodiment, and are not repeated here.
An embodiment of the present invention provides a schematic diagram of a data transmission device for performing data compression, as shown in fig. 7, including:
The processor 702 is responsible for managing the bus architecture and general processing, and the memory 701 may store data used by the processor 702 in performing operations. The transceiver 703 is used for receiving and transmitting data under the control of the processor 702.
The bus architecture may include any number of interconnected buses and bridges, with one or more processors, represented by processor 702, and various circuits of memory, represented by memory 701, being linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. The bus interface provides an interface. The processor 702 is responsible for managing the bus architecture and general processing, and the memory 701 may store data used by the processor 702 in performing operations.
The processes disclosed in the embodiments of the invention can be implemented in the processor 702 or implemented by the processor 702. In implementation, the steps of the signal processing flow may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 702. The processor 702 may be a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like that implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present invention. The general purpose processor may be a microprocessor or any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 701, and the processor 702 reads the information in the memory 701, and completes the steps of the signal processing flow in combination with the hardware thereof.
Specifically, the processor 702 is configured to read the program in the memory 701 and execute:
in the service transmission process, determining a currently adopted compression dictionary and a compression algorithm, wherein an initialized compression dictionary and an initialized compression algorithm are initially adopted, and then when determining that an updating condition is met, respectively and correspondingly updating the currently adopted compression dictionary and the currently adopted compression algorithm by using the compression dictionary and the compression algorithm output by the AI model;
and based on the currently adopted compression dictionary, compressing or decompressing the transmitted service data by using the currently adopted compression algorithm.
Optionally, the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted feature and the service data, compress the current compression dictionary by using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust a model parameter of the AI model by using the highest compression ratio as a feedback input.
Optionally, the processor is further configured to:
initializing a currently adopted compression dictionary and a compression algorithm according to preset information; or
And when the service connection is established, initializing the currently adopted compression dictionary and compression algorithm according to the compression dictionary and compression algorithm output by the AI model when the service transmission is completed last time.
Optionally, the processor determines that the update condition is satisfied, including at least one of:
when the service connection is established, the updating condition is met;
after the service connection is established, when a set updating period is reached, the updating condition is met;
and when determining that the event triggering condition is met, meeting the updating condition.
Optionally, the processor determines that an event trigger condition is satisfied, including at least one of:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
Optionally, in the service transmission process, the processor is further configured to:
acquiring a compression dictionary and a compression algorithm output by a local AI model; or
Acquiring a compression dictionary and a compression algorithm output by a local AI model, and sending the compression dictionary and the compression algorithm to opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from third-party equipment, wherein the third-party equipment is equipment of a functional node positioned on a cloud or edge; or
And acquiring a compression dictionary and a compression algorithm output by using the AI model from a third-party device, and sending the compression dictionary and the compression algorithm to an opposite-end data transmission device, wherein the third-party device is a device of a functional node positioned in the cloud or the edge.
Optionally, the processor is further configured to:
and sending the compression rate of the latest transmitted service data and the currently transmitted service data to the third-party equipment.
An embodiment of the present invention provides a schematic diagram of a third-party device for performing data compression, as shown in fig. 8, including:
a memory 801, a processor 802, a transceiver 803, and a bus interface 804.
The processor 802 is responsible for managing the bus architecture and general processing, and the memory 801 may store data used by the processor 802 in performing operations. The transceiver 803 is used for receiving and transmitting data under the control of the processor 802.
The bus architecture may include any number of interconnected buses and bridges, with one or more processors represented by processor 802 and various circuits of memory represented by memory 801 being linked together in particular. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. The bus interface provides an interface. The processor 802 is responsible for managing the bus architecture and general processing, and the memory 801 may store data used by the processor 802 in performing operations.
The process disclosed in the embodiments of the present invention can be applied to the processor 802, or implemented by the processor 802. In implementation, the steps of the signal processing flow may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 802. The processor 802 may be a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like that implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present invention. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 801, and the processor 802 reads the information in the memory 801 and completes the steps of the signal processing flow in combination with the hardware thereof.
Specifically, the processor 802 is configured to read the program in the memory 801 and execute:
responding to the request of the data transmission equipment, and acquiring the latest service data transmitted by the current service and the compression rate of the service data which is currently transmitted;
inputting the latest transmitted service data into an AI model, and outputting a compression dictionary and a compression algorithm by using the AI model;
and sending the compression dictionary and the compression algorithm to the data transmission equipment.
Optionally, the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted feature and the service data, compress the current compression dictionary by using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust a model parameter of the AI model by using the highest compression ratio as a feedback input.
Optionally, the compression dictionary and the compression algorithm are sent to the data transmission device, and the processor is specifically configured to:
and when the updating condition is determined to be met, sending the compression dictionary and the compression algorithm to the data transmission equipment.
Optionally, the processor determines that the update condition is satisfied, including at least one of:
determining that the data transmission equipment meets the updating condition when establishing service connection;
determining that the data transmission equipment meets the updating condition when a set updating period is reached after the data transmission equipment establishes service connection;
and when determining that the event triggering condition is met, meeting the updating condition.
Optionally, the processor determines that an event trigger condition is satisfied, including at least one of:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
The data transmission device for performing data compression provided in the embodiment of the present invention is the same as the data transmission device in embodiment 1 of the present invention, and various implementation manners for performing data compression by the data transmission device in the system provided in the embodiment may be applied to the data transmission device for performing data compression in the embodiment, and are not repeated here.
The third-party device for performing data compression provided in the embodiment of the present invention is the same as the third-party device in embodiment 1 of the present invention, and various implementation manners of performing data compression by the third-party device in the system provided in the embodiment may be applied to the third-party device for performing data compression in the embodiment, and are not repeated here.
An embodiment of the present invention provides a schematic diagram of an apparatus for performing data compression by a data transmission device, as shown in fig. 9, including:
a dictionary algorithm determining unit 901, configured to determine a currently-used compression dictionary and compression algorithm in a service transmission process, where an initialized compression dictionary and compression algorithm are initially used, and then when it is determined that an update condition is met, the currently-used compression dictionary and compression algorithm are updated correspondingly by using the compression dictionary and compression algorithm output by the AI model;
a compressing unit 902, configured to compress or decompress, based on the currently adopted compression dictionary, the service data that is transmitted by using the currently adopted compression algorithm.
Optionally, the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted feature and the service data, compress the current compression dictionary by using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust a model parameter of the AI model by using the highest compression ratio as a feedback input.
Optionally, the dictionary algorithm determining unit is further configured to:
initializing a currently adopted compression dictionary and a compression algorithm according to preset information; or
And when the service connection is established, initializing the currently adopted compression dictionary and compression algorithm according to the compression dictionary and compression algorithm output by the AI model when the service transmission is completed last time.
Optionally, the dictionary algorithm determining unit determines that the update condition is satisfied, and includes at least one of the following steps:
when the service connection is established, the updating condition is met;
after the service connection is established, when a set updating period is reached, the updating condition is met;
and when determining that the event triggering condition is met, meeting the updating condition.
Optionally, the dictionary algorithm determination unit determines that the event triggering condition is satisfied, and includes at least one of the following steps:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
Optionally, in the service transmission process, the dictionary algorithm determining unit is further configured to:
acquiring a compression dictionary and a compression algorithm output by a local AI model; or
Acquiring a compression dictionary and a compression algorithm output by a local AI model, and sending the compression dictionary and the compression algorithm to opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from third-party equipment, wherein the third-party equipment is equipment of a functional node positioned on a cloud or edge; or
And acquiring a compression dictionary and a compression algorithm output by using the AI model from a third-party device, and sending the compression dictionary and the compression algorithm to an opposite-end data transmission device, wherein the third-party device is a device of a functional node positioned in the cloud or the edge.
Optionally, the compression unit is further configured to:
and sending the compression rate of the latest transmitted service data and the currently transmitted service data to the third-party equipment.
An embodiment of the present invention provides an apparatus for data compression by a third-party device, as shown in fig. 10, including:
a data receiving unit 1001, configured to respond to a request from a data transmission device, and obtain compression ratios of service data newly transmitted by a current service and service data currently completing transmission;
a dictionary algorithm generating unit 1002, configured to input the latest transmitted service data into an AI model, and output a compression dictionary and a compression algorithm by using the AI model;
a data sending unit 1003, configured to send the compression dictionary and the compression algorithm to the data transmission device.
Optionally, the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted feature and the service data, compress the current compression dictionary by using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust a model parameter of the AI model by using the highest compression ratio as a feedback input.
Optionally, the compression dictionary and the compression algorithm are sent to the data transmission device, and the data sending unit is specifically configured to:
and when the updating condition is determined to be met, sending the compression dictionary and the compression algorithm to the data transmission equipment.
Optionally, the determining, by the data sending unit, that the update condition is satisfied includes at least one of:
determining that the data transmission equipment meets the updating condition when establishing service connection;
after the data transmission equipment establishes service connection and a set updating period is reached, updating conditions are met;
and when determining that the event triggering condition is met, meeting the updating condition.
Optionally, the data sending unit determines that the event trigger condition is satisfied, and includes at least one of the following steps:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
The apparatus for performing data compression provided in the embodiment of the present invention is the same as the data transmission device in embodiment 1 of the present invention, and may be applied to various embodiments for performing data compression by the data transmission device in the system provided in the embodiment, and will not be repeated here.
The apparatus for performing data compression provided in the embodiment of the present invention is the same as the third-party device in embodiment 1 of the present invention, and may be applied to various implementation manners of performing data compression on the third-party device in the system provided in the embodiment, and is not repeated here.
The present invention also provides a processor-readable storage medium storing a computer program for causing a processor to execute the steps of a method for data compression applied to a data transmission device provided in embodiment 1 above.
The present invention also provides a processor-readable storage medium storing a computer program for causing a processor to execute the steps of the method for data compression applied to a third party device as provided in embodiment 1 above.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical division, and in actual implementation, there may be other divisions, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present application may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may be stored in a computer readable storage medium.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.
The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The technical solutions provided by the present application are introduced in detail, and the present application applies specific examples to explain the principles and embodiments of the present application, and the descriptions of the above examples are only used to help understand the method and the core ideas of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (27)
1. A method for data compression, applied to a data transmission device, is characterized in that the method comprises:
in the service transmission process, determining a currently adopted compression dictionary and a compression algorithm, wherein an initialized compression dictionary and an initialized compression algorithm are initially adopted, and then when determining that an updating condition is met, respectively and correspondingly updating the currently adopted compression dictionary and the currently adopted compression algorithm by using the compression dictionary and the compression algorithm output by the AI model;
and based on the currently adopted compression dictionary, compressing or decompressing the transmitted service data by using the currently adopted compression algorithm.
2. The method of claim 1, wherein the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted features and the service data, and perform compression with a current compression dictionary using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust model parameters of the AI model by using the highest compression ratio as a feedback input.
3. The method of claim 1, further comprising:
initializing a currently adopted compression dictionary and a compression algorithm according to preset information; or
And when the service connection is established, initializing the currently adopted compression dictionary and compression algorithm according to the compression dictionary and compression algorithm output by the AI model when the service transmission is completed last time.
4. The method of claim 1, wherein determining that the update condition is satisfied comprises at least one of:
when the service connection is established, the updating condition is met;
after the service connection is established, when a set updating period is reached, the updating condition is met;
and when determining that the event triggering condition is met, meeting the updating condition.
5. The method of claim 4, wherein determining that an eventuality trigger condition is satisfied comprises at least one of:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
6. The method of claim 1, wherein during the service transmission process, the method further comprises:
acquiring a compression dictionary and a compression algorithm output by a local AI model; or
Acquiring a compression dictionary and a compression algorithm output by a local AI model, and sending the compression dictionary and the compression algorithm to opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from third-party equipment, wherein the third-party equipment is equipment of a functional node positioned on a cloud or edge; or
And acquiring a compression dictionary and a compression algorithm output by using the AI model from a third-party device, and sending the compression dictionary and the compression algorithm to an opposite-end data transmission device, wherein the third-party device is a device of a functional node positioned in the cloud or the edge.
7. The method of claim 1, further comprising:
and sending the compression rate of the latest transmitted service data and the currently transmitted service data to the third-party equipment.
8. A method for data compression, applied to a third-party device, includes:
responding to the request of the data transmission equipment, and acquiring the latest service data transmitted by the current service and the compression rate of the service data which is currently transmitted;
inputting the latest transmitted service data into an AI model, and outputting a compression dictionary and a compression algorithm by using the AI model;
and sending the compression dictionary and the compression algorithm to the data transmission equipment.
9. The method of claim 8, wherein the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted features and the service data, and perform compression with a current compression dictionary using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust model parameters of the AI model by using the highest compression ratio as a feedback input.
10. The method according to claim 8, wherein sending the compression dictionary and compression algorithm to the data transmission device specifically comprises:
and when the updating condition is determined to be met, sending the compression dictionary and the compression algorithm to the data transmission equipment.
11. The method of claim 10, wherein determining that the update condition is satisfied comprises at least one of:
determining that the data transmission equipment meets the updating condition when establishing service connection;
determining that the data transmission equipment meets the updating condition when a set updating period is reached after the data transmission equipment establishes service connection;
and when determining that the event triggering condition is met, meeting the updating condition.
12. The method of claim 11, wherein determining that an eventuality trigger condition is satisfied comprises at least one of:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
13. A data transmission device for performing data compression, comprising a memory, a transceiver, a processor:
a memory for storing a computer program; a transceiver for transceiving data under control of the processor; a processor for reading the computer program in the memory and performing the following operations:
in the service transmission process, determining a currently adopted compression dictionary and a compression algorithm, wherein an initialized compression dictionary and an initialized compression algorithm are adopted initially, and then when determining that an updating condition is met, updating the currently adopted compression dictionary and the currently adopted compression algorithm correspondingly by using the compression dictionary and the compression algorithm output by the AI model respectively;
and based on the currently adopted compression dictionary, compressing or decompressing the transmitted service data by using the currently adopted compression algorithm.
14. The data transmission apparatus according to claim 13, wherein the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted features and the service data, and perform compression with the current compression dictionary using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust model parameters of the AI model using the highest compression ratio as a feedback input.
15. The data transmission device of claim 13, wherein the processor is further configured to:
initializing a currently adopted compression dictionary and a compression algorithm according to preset information; or
And when the service connection is established, initializing the currently adopted compression dictionary and compression algorithm according to the compression dictionary and compression algorithm output by the AI model when the service transmission is completed last time.
16. The data transmission device of claim 13, wherein the processor determines that the update condition is satisfied, comprising at least one of:
when the service connection is established, the updating condition is met;
after the service connection is established, when a set updating period is reached, the updating condition is met;
and when determining that the event triggering condition is met, meeting the updating condition.
17. The data transmission device of claim 16, wherein the processor determines that an eventing trigger condition is satisfied, comprising at least one of:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
18. The data transmission device of claim 13, wherein during traffic transmission, the processor is further configured to:
acquiring a compression dictionary and a compression algorithm output by a local AI model; or
Acquiring a compression dictionary and a compression algorithm output by a local AI model, and sending the compression dictionary and the compression algorithm to opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from opposite-end data transmission equipment; or
Acquiring a compression dictionary and a compression algorithm output by using an AI (artificial intelligence) model from third-party equipment, wherein the third-party equipment is equipment of a functional node positioned on a cloud or edge; or
And acquiring a compression dictionary and a compression algorithm output by using the AI model from a third-party device, and sending the compression dictionary and the compression algorithm to an opposite-end data transmission device, wherein the third-party device is a device of a functional node positioned in the cloud or the edge.
19. The data transmission device of claim 13, wherein the processor is further configured to:
and sending the compression rate of the latest transmitted service data and the currently transmitted service data to the third-party equipment.
20. A third party device for data compression, comprising a memory, a transceiver, a processor:
a memory for storing a computer program; a transceiver for transceiving data under control of the processor; a processor for reading the computer program in the memory and performing the following operations:
responding to the request of the data transmission equipment, and acquiring the latest service data transmitted by the current service and the compression rate of the service data which is currently transmitted;
inputting the latest transmitted service data into an AI model, and outputting a compression dictionary and a compression algorithm by using the AI model;
and sending the compression dictionary and the compression algorithm to the data transmission equipment.
21. The third-party device of claim 20, wherein the AI model is configured to perform feature extraction on newly transmitted service data, output a compression dictionary according to a correlation between the extracted features and the service data, perform compression with a current compression dictionary using different compression algorithms, output a compression algorithm corresponding to a highest compression ratio, and adjust model parameters of the AI model by using the highest compression ratio as a feedback input.
22. The third party device of claim 20, wherein the compression dictionary and compression algorithm are sent to the data transmission device, and wherein the processor is specifically configured to:
and when the updating condition is determined to be met, sending the compression dictionary and the compression algorithm to the data transmission equipment.
23. The third party device of claim 22, wherein the processor determines that an update condition is satisfied, comprising at least one of:
determining that the data transmission equipment meets the updating condition when establishing service connection;
determining that the data transmission equipment meets the updating condition when a set updating period is reached after the data transmission equipment establishes service connection;
and when determining that the event triggering condition is met, meeting the updating condition.
24. The third party device of claim 23, wherein the processor determines that an eventuality trigger condition is met, comprising at least one of:
when the compression rate of the currently transmitted service data is lower than a preset threshold, determining that an event triggering condition is met;
and determining that the event triggering condition is met when the difference value between the compression ratio of the current transmission-finished service data and the compression ratio expected by the compression dictionary and the compression algorithm output by the current AI model is greater than a preset value.
25. An apparatus for data compression, comprising:
the dictionary algorithm determining unit is used for determining a compression dictionary and a compression algorithm which are adopted currently in the service transmission process, wherein the compression dictionary and the compression algorithm which are initialized are adopted initially, and then when the updating condition is met, the compression dictionary and the compression algorithm which are output by the AI model are utilized to respectively correspondingly update the compression dictionary and the compression algorithm which are adopted currently;
and the compression unit is used for compressing or decompressing the transmitted service data by using the currently adopted compression algorithm based on the currently adopted compression dictionary.
26. An apparatus for data compression, comprising:
the data receiving unit is used for responding to the request of the data transmission equipment and acquiring the latest transmitted service data of the current service and the compression rate of the service data which is currently transmitted;
the dictionary algorithm generating unit is used for inputting the latest transmitted service data into the AI model and outputting a compression dictionary and a compression algorithm by utilizing the AI model;
and the data sending unit is used for sending the compression dictionary and the compression algorithm to the data transmission equipment.
27. A processor-readable storage medium, characterized in that the processor-readable storage medium stores a computer program for causing a processor to perform the method of any one of claims 1 to 7 or claims 8 to 12.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011212245.2A CN114449579A (en) | 2020-11-03 | 2020-11-03 | Method, device and equipment for data compression |
PCT/CN2021/121339 WO2022095636A1 (en) | 2020-11-03 | 2021-09-28 | Method and apparatus for performing data compression and device |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011212245.2A CN114449579A (en) | 2020-11-03 | 2020-11-03 | Method, device and equipment for data compression |
Publications (1)
Publication Number | Publication Date |
---|---|
CN114449579A true CN114449579A (en) | 2022-05-06 |
Family
ID=81361567
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202011212245.2A Pending CN114449579A (en) | 2020-11-03 | 2020-11-03 | Method, device and equipment for data compression |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN114449579A (en) |
WO (1) | WO2022095636A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115347902A (en) * | 2022-10-17 | 2022-11-15 | 四川省华存智谷科技有限责任公司 | Self-adaptive compression method and system in mirror image data transmission process of storage system |
CN117076388A (en) * | 2023-10-12 | 2023-11-17 | 中科信工创新技术(北京)有限公司 | File processing method and device, storage medium and electronic equipment |
CN117076886A (en) * | 2023-08-22 | 2023-11-17 | 中通服网盈科技有限公司 | Petrochemical pipe gallery data analysis and decision system and operation method thereof |
WO2024065620A1 (en) * | 2022-09-30 | 2024-04-04 | Qualcomm Incorporated | Model selection and switching |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116506514B (en) * | 2023-06-27 | 2023-12-12 | 大唐融合通信股份有限公司 | Data compression method, device, equipment, server and sewage cloud control system |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060262812A1 (en) * | 2005-05-17 | 2006-11-23 | Nokia Corporation | Signaling compression/decompression with improved efficiency |
US20110158307A1 (en) * | 2009-12-31 | 2011-06-30 | Chik Chung Lee | Asymmetric dictionary-based compression/decompression useful for broadcast or multicast unidirectional communication channels |
CN103209436A (en) * | 2013-01-28 | 2013-07-17 | 南开大学 | Multi-parameter information fusion sparse model based on compressive sensing theory |
CN107645538A (en) * | 2017-08-14 | 2018-01-30 | 阿里巴巴集团控股有限公司 | The method and device and electronic equipment of dynamic compression dictionary are provided |
WO2019100937A1 (en) * | 2017-11-27 | 2019-05-31 | 电信科学技术研究院有限公司 | Method and apparatus for data transmission |
US10447296B1 (en) * | 2018-06-22 | 2019-10-15 | International Business Machines Coporation | Accelerated compression/decompression including predefined dictionary |
CN111178540A (en) * | 2019-12-29 | 2020-05-19 | 浪潮(北京)电子信息产业有限公司 | Training data transmission method, device, equipment and medium |
CN111193752A (en) * | 2020-02-28 | 2020-05-22 | 广州市百果园信息技术有限公司 | Data compression method and device, compression server and storage medium |
CN111510419A (en) * | 2019-01-31 | 2020-08-07 | 华为技术有限公司 | Data compression method and base station |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8698657B2 (en) * | 2012-09-10 | 2014-04-15 | Canon Kabushiki Kaisha | Methods and systems for compressing and decompressing data |
CN103281156B (en) * | 2013-05-02 | 2017-06-06 | 华为技术有限公司 | A kind of data compression, decompression method and device |
US11025271B2 (en) * | 2019-02-22 | 2021-06-01 | Qualcomm Incorporated | Compression of high dynamic ratio fields for machine learning |
CN111552669A (en) * | 2020-04-26 | 2020-08-18 | 北京达佳互联信息技术有限公司 | Data processing method and device, computing equipment and storage medium |
-
2020
- 2020-11-03 CN CN202011212245.2A patent/CN114449579A/en active Pending
-
2021
- 2021-09-28 WO PCT/CN2021/121339 patent/WO2022095636A1/en active Application Filing
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060262812A1 (en) * | 2005-05-17 | 2006-11-23 | Nokia Corporation | Signaling compression/decompression with improved efficiency |
US20110158307A1 (en) * | 2009-12-31 | 2011-06-30 | Chik Chung Lee | Asymmetric dictionary-based compression/decompression useful for broadcast or multicast unidirectional communication channels |
CN103209436A (en) * | 2013-01-28 | 2013-07-17 | 南开大学 | Multi-parameter information fusion sparse model based on compressive sensing theory |
CN107645538A (en) * | 2017-08-14 | 2018-01-30 | 阿里巴巴集团控股有限公司 | The method and device and electronic equipment of dynamic compression dictionary are provided |
WO2019100937A1 (en) * | 2017-11-27 | 2019-05-31 | 电信科学技术研究院有限公司 | Method and apparatus for data transmission |
US10447296B1 (en) * | 2018-06-22 | 2019-10-15 | International Business Machines Coporation | Accelerated compression/decompression including predefined dictionary |
CN111510419A (en) * | 2019-01-31 | 2020-08-07 | 华为技术有限公司 | Data compression method and base station |
CN111178540A (en) * | 2019-12-29 | 2020-05-19 | 浪潮(北京)电子信息产业有限公司 | Training data transmission method, device, equipment and medium |
CN111193752A (en) * | 2020-02-28 | 2020-05-22 | 广州市百果园信息技术有限公司 | Data compression method and device, compression server and storage medium |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2024065620A1 (en) * | 2022-09-30 | 2024-04-04 | Qualcomm Incorporated | Model selection and switching |
WO2024066793A1 (en) * | 2022-09-30 | 2024-04-04 | Qualcomm Incorporated | Model selection and switching |
CN115347902A (en) * | 2022-10-17 | 2022-11-15 | 四川省华存智谷科技有限责任公司 | Self-adaptive compression method and system in mirror image data transmission process of storage system |
CN115347902B (en) * | 2022-10-17 | 2023-02-28 | 四川省华存智谷科技有限责任公司 | Self-adaptive compression method and system in mirror image data transmission process of storage system |
CN117076886A (en) * | 2023-08-22 | 2023-11-17 | 中通服网盈科技有限公司 | Petrochemical pipe gallery data analysis and decision system and operation method thereof |
CN117076388A (en) * | 2023-10-12 | 2023-11-17 | 中科信工创新技术(北京)有限公司 | File processing method and device, storage medium and electronic equipment |
Also Published As
Publication number | Publication date |
---|---|
WO2022095636A1 (en) | 2022-05-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN114449579A (en) | Method, device and equipment for data compression | |
US10554791B2 (en) | Method for processing data packet and apparatus | |
CN110875909B (en) | Data transmission method and related device | |
EP3032871A1 (en) | Data transmission method, device and system | |
US11956667B2 (en) | Communication method and device | |
US20200296623A1 (en) | Transmitting device, receiving device, and methods performed therein for handling buffer reset | |
US20210185560A1 (en) | Transmitting Device, Receiving Device, and Methods Performed Therein for Handling Uplink Data Compression | |
WO2018137716A1 (en) | Method and device for keeping continuity of udc function | |
CN110798865B (en) | Data compression method, data compression device, computer equipment and computer-readable storage medium | |
CN112187400B (en) | Data transmission method and device | |
US20140036784A1 (en) | Data Transmission Method, Radio Access Network Device, Wireless Gateway and System | |
WO2021027786A1 (en) | Method and device for ue to report udc information | |
KR102629788B1 (en) | Data transmission methods and devices | |
JP7434663B2 (en) | Data compression method, device and storage medium | |
EP4277346A1 (en) | Transmission parameter adjusting method and communication device | |
WO2023098881A1 (en) | Transmission method for location auxiliary data, and terminal, location server, and storage medium | |
WO2022143149A1 (en) | Service transmission method and communication apparatus | |
WO2023173293A1 (en) | Wireless communication method, and device | |
CN112544104B (en) | Data transmission method and terminal | |
WO2024123325A1 (en) | Apparatus and method for timer-based robust header compression for real-time protocol time stamp compression | |
CN117560427A (en) | Data compression method and communication equipment | |
CN115134047A (en) | Transmission method, device, equipment and readable storage medium | |
WO2014182299A1 (en) | Multimedia messaging service (mms) file size adjustment based on network condition |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |