WO2021174839A1

WO2021174839A1 - Data compression method and apparatus, and computer-readable storage medium

Info

Publication number: WO2021174839A1
Application number: PCT/CN2020/119122
Authority: WO
Inventors: 李桃
Original assignee: 平安科技（深圳）有限公司
Priority date: 2020-03-06
Filing date: 2020-09-29
Publication date: 2021-09-10
Also published as: CN111431537A

Abstract

A data compression method and apparatus, an electronic device, and a computer-readable storage medium. The method comprises: obtaining a message data set generated by an Internet of Things-based device during transmission process, identifying a character set in the message data set, and using the character set as an initial character set (S1); calculating the frequency of each character in the initial character set, selecting a redundant data set in the message data set according to a preset mode and the frequency of the character, identifying a standard character set in the redundant data set, and encoding characters in the standard character set to obtain a character code table (S2); and using the character code table to calculate a minimum character path length in the redundant data set, and compressing the redundant data set according to the minimum character path length (S3). The method, apparatus, electronic device, and computer-readable storage medium realize the compression of redundant data, so as to reduce the waste of a storage space.

Description

Data compression method, device and computer readable storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on March 6, 2020 with the application number 202010155298.9 and the invention title "Data compression method, device and computer readable storage medium", the entire content of which is incorporated by reference In this application.

Technical field

This application relates to the field of big data technology, and in particular to a data compression method, device, and computer-readable storage medium.

Background technique

The current messaging system in the IoT business scenario puts all the message data generated by the IoT device in the transmission process into a DB database or file, and the storage medium compresses and stores the message data. Usually there is a lot of redundant device data in the messages transmitted by the Internet of Things devices, and most of these device data are unchanged or rarely changed. The inventor realizes that at present, people tend to store the redundant device data in In the disk space, due to the huge amount of data generated in the Internet of Things scenario, it is easy to cause insufficient memory in the disk space, which will bring maintenance inconvenience and cost increase in the later period.

Summary of the invention

A data compression method provided by this application includes:

Acquiring a message data set generated by the Internet of Things device during transmission, identifying a character set in the message data set, and using the character set as an initial character set;

Calculate the frequency of each character in the initial character set, filter out the redundant data set in the message data set in a preset manner according to the frequency of the character, and obtain the standard character set of the redundant data set, Encoding the characters in the standard character set to obtain a character code table;

The minimum path length of the characters in the redundant data set is calculated by using the character code table, and the redundant data set is compressed according to the minimum path length of the characters.

This application also provides an electronic device, which includes:

Memory, storing at least one instruction; and

The processor executes the instructions stored in the memory to implement the data compression method as described below:

The present application also provides a computer-readable storage medium in which at least one instruction is stored, and the at least one instruction is executed by a processor in an electronic device to implement the following data compression method:

The present application also provides a data compression device, which includes:

The recognition module is used to obtain the message data set generated by the Internet of Things device during the transmission process, identify the character set in the message data set, and use the character set as the initial character set;

The calculation and screening module is used to calculate the frequency of each character in the initial character set, filter out the redundant data set in the message data set in a preset manner according to the frequency of the character, and obtain the redundant data Set of standard character sets;

The encoding module is used to encode the characters in the standard character set to obtain a character code table;

The compression module is configured to use the character code table to calculate the minimum path length of the characters in the redundant data set, and compress the redundant data set according to the minimum path length of the characters.

Description of the drawings

FIG. 1 is a schematic flowchart of a data compression method provided by an embodiment of this application;

2 is a schematic diagram of the internal structure of a data compression device provided by an embodiment of the application;

FIG. 3 is a schematic diagram of modules of a data compression program in a data compression device provided by an embodiment of the application.

The realization, functional characteristics, and advantages of the purpose of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

Detailed ways

It should be understood that the specific embodiments described here are only used to explain the present application, and are not used to limit the present application.

This application provides a data compression method. Referring to FIG. 1, it is a schematic flowchart of a data compression method provided by an embodiment of this application. The method can be executed by a device, and the device can be implemented by software and/or hardware.

In this embodiment, the data compression method includes:

S1. Obtain a message data set generated by the Internet of Things device during transmission, identify a character set in the message data set, and use the character set as an initial character set.

In a preferred embodiment of the present application, the IoT device includes a barcode, a sensor, a scanner, and so on. In the process of using the Internet of Things device, a large amount of message data will be generated. For example, in the process of barcode scanning, a message data set including parameter configuration data of the barcode, barcode model, and barcode identification number will be generated. Preferably, this application collects the message data generated by different IoT devices in the transmission process as the message data set of this application.

In a preferred embodiment of the present application, the character set includes a word character set and a symbol character set.

Preferably, the present application recognizes the word characters in the message data set through the shortest path algorithm, so as to combine to form the word character set. Wherein, the shortest path algorithm includes: constructing a word segmentation directed acyclic graph through a custom dictionary, wherein each word in the word segmentation directed acyclic graph corresponds to a directed edge in the graph, and Assigned to the corresponding side length (weight); in all the paths from the start to the end of the word segmentation directed acyclic graph, the calculated word length values are arranged in ascending order (the values at any two different positions must be different , The same below) is the first, second,..., i,..., Nth path set as the corresponding rough score result set. If two or more paths are equal in length, then their lengths are tied for the i-th, and they must be included in the rough result set without affecting the sequence numbers of other paths. The final rough result set size is greater than or equal to N, get The character set of words contained in the path set.

Preferably, the present application recognizes the symbol characters in the message data set through a character recognition algorithm, so as to combine to form the symbol character set. Wherein, the character recognition algorithm includes: presetting a character database template, matching the character database template with the message data set using character matching technology, and extracting symbol characters in the message data set that are successfully matched, Thus, the symbol character set is obtained.

In this application, the word character set and the symbol character set are stored in the database as the initial character set.

S2. Calculate the frequency of each character in the initial character set, filter out the redundant data set in the message data set in a preset manner according to the frequency of the character, and identify the standard character set in the redundant data set, The characters in the standard character set are encoded to obtain a character code table.

In a preferred embodiment of the present application, the frequency of each character in the initial character set is calculated by the following method

Among them, f _i represents the frequency of appearance of the initial character i, n _i represents the number of characters i in the initial character set, and v represents the number of all characters in the initial character set.

Preferably, the present application filters out the redundant data in the message data set in a preset manner according to the sort of the character frequency, and identifies the standard character set in the redundant data. The preset method is to compare the character frequency with a preset threshold, and if the character frequency is less than the preset threshold, use its corresponding message data as redundant data. The threshold is set to 0.35.

In the embodiment of the present application, the method for identifying the character set in the message data set is the same as the method for identifying the character set in the message data set in step S1, and the description will not be repeated here. Further, the encoding of the standard character set in the embodiment of the present application includes: presetting the serial number character set of the standard character set, and obtaining a one-to-one correspondence between the standard character set and the serial number character set Calculating the probability of each standard character in the standard character set according to the one-to-one correspondence, forming the code number of the corresponding standard character according to the probability of each standard character, and establishing the character code according to the code number surface. Wherein, the corresponding relationship can be expressed as: v={(1,v ₁ ),(1,v ₂ ),...,(1,v _n )}, the probability of each standard character in the standard character set can be expressed for:

In the following, this application _{is explained by encoding the standard character set n 0} ={1,2,3,4} as an example: the serial number character set of _{the standard character set n 0 is preset to {01,02,03,04}} , The probability of each standard character in the standard character set n ₀ is calculated as {0.4, 0.3, 0.2, 0.1}, and the code number of each standard character in _{the standard character set n 0 is obtained as {1, 00,} 010,110}, so that the character code table of _{the standard character set n 0} _{is established as w 0} ={(1,1),(2,00),(3,010),(4,110)}.

S3. Calculate the minimum path length of the characters in the redundant data set by using the character code table, and compress the redundant data set according to the minimum path length of the characters.

In a preferred embodiment of the present application, the method for calculating the minimum path length of characters in the redundant data set includes:

Among them, MINWPL represents the minimum path length of _{a character, w k} represents the character weight in _{the k-th character code table, l k} represents the character offset in the k-th character code table, and n represents the number of characters in the character code table. Preferably, this application compresses the redundant data set according to the minimum path length of the character.

Further, after compressing the redundant data set, the embodiment of the present application further includes: performing a decoding operation on the compressed redundant data set by using a pre-created decoding rule. Wherein, the pre-created decoding rules in this application include: according to the coding diagram in the serial number characters, suppose the maximum number of layers is k, then the tree layer is cyclically created from the kth layer to the second layer. If the established tree layer is not For the kth level, take 2 nodes from the non-leaf nodes and leaf nodes of the tree layer to build a new node; if the non-leaf node of the tree layer is singular, then the last node of the non-leaf node Combine with the first node of the leaf node of the tree layer to form a new node; if the established tree layer is the kth layer, the leaf node is used to directly establish the new node, thereby forming the decoding rule.

As shown in Figure 2, it is a functional block diagram of the data compression device of the present application.

The data compression apparatus 100 described in this application can be installed in an electronic device. According to the realized functions, the data compression device may include an identification module 101, a calculation and screening module 102, an encoding module 103, and a compression module 104. The module described in the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can complete fixed functions, and are stored in the memory of the electronic device.

In this embodiment, the functions of each module/unit are as follows:

The recognition module 101 is configured to obtain a message data set generated by an Internet of Things device during transmission, identify a character set in the message data set, and use the character set as an initial character set;

The calculation and screening module 102 is configured to calculate the frequency of each character in the initial character set, filter out the redundant data set in the message data set in a preset manner according to the frequency of the character, and obtain the redundant data set. The standard character set of the remaining data set;

The encoding module 103 is configured to encode characters in the standard character set to obtain a character code table;

The compression module 104 is configured to use the character code table to calculate the minimum path length of the characters in the redundant data set, and compress the redundant data set according to the minimum path length of the characters.

In detail, the specific implementation steps of each module in the data compression device 100 are as follows:

The recognition module 101 obtains a message data set generated by the Internet of Things device during transmission, recognizes a character set in the message data set, and uses the character set as an initial character set.

The calculation and screening module 102 calculates the frequency of each character in the initial character set, filters out the redundant data set in the message data set in a preset manner according to the frequency of the character, and obtains the redundant data The standard character set of the set.

Preferably, the present application filters out the redundant data in the message data set in a preset manner according to the sorting of the character frequency, and obtains the standard character set of the redundant data, and the preset manner is The character frequency is compared with a preset threshold, and if the character frequency is less than the preset threshold, the corresponding message data is used as redundant data, wherein the preset threshold in this application is 0.35.

The encoding module 103 encodes the characters in the standard character set to obtain a character code table.

The encoding of the standard character set in the embodiment of the present application includes: presetting the serial number character set of the standard character set, obtaining the one-to-one correspondence between the standard character set and the serial number character set, and according to all According to the one-to-one correspondence, the probability of each standard character in the standard character set is calculated, the code number of the corresponding standard character is formed according to the probability of each standard character, and the character code table is established according to the code number. Wherein, the corresponding relationship can be expressed as: v={(1,v ₁ ),(1,v ₂ ),...,(1,v _n )}, the probability of each standard character in the standard character set can be expressed for:

The compression module 104 uses the character code table to calculate the minimum path length of the characters in the redundant data set, and compresses the redundant data set according to the minimum path length of the characters.

Further, after compressing the redundant data set, the embodiment of the present application further includes: performing a decoding operation on the compressed redundant data set by using a pre-created decoding rule. Wherein, the pre-created decoding rules in this application include: according to the coding diagram in the serial number characters, suppose the maximum number of layers is k, then the tree layer is cyclically created from the kth layer to the second layer. If the established tree layer is not For the kth level, take 2 nodes from the non-leaf nodes and leaf nodes of the tree layer to build a new node; if the non-leaf node of the tree layer is singular, then the last node of the non-leaf node Combine with the first node of the leaf node of the tree layer to form a new node; if the established tree layer is the kth layer, the leaf node is used directly to establish the new node, thereby forming the decoding rule.

As shown in FIG. 3, it is a schematic diagram of the structure of an electronic device implementing the data compression method of the present application.

The electronic device 1 may include a processor 10, a memory 11, and a bus, and may also include a computer program stored in the memory 11 and running on the processor 10, such as a data compression program 12.

Wherein, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, magnetic disk, CD etc. The memory 11 may be an internal storage unit of the electronic device 1 in some embodiments, for example, a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (SmartMediaCard, SMC), and a secure digital (SecureDigital, SD) equipped on the electronic device 1. Card, flash card (FlashCard), etc. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can be used not only to store application software and various data installed in the electronic device 1, such as the code of a data compression program, etc., but also to temporarily store data that has been output or will be output.

The processor 10 may be composed of integrated circuits in some embodiments, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same function or different functions, including one or more Central Processing Unit (CPU), microprocessor, digital processing chip, graphics processor and a combination of various control chips, etc. The processor 10 is the control core (ControlUnit) of the electronic device, which uses various interfaces and lines to connect the various components of the entire electronic device, and runs or executes programs or modules (such as execution data) stored in the memory 11 Compress programs, etc.), and call data stored in the memory 11 to execute various functions of the electronic device 1 and process data.

The bus may be a peripheral component interconnection standard (PCI) bus or an extended industry standard architecture (EISA) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. The bus is configured to implement connection and communication between the memory 11 and at least one processor 10 and the like.

FIG. 3 only shows an electronic device with components. Those skilled in the art can understand that the structure shown in FIG. 3 does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure. Components, or a combination of certain components, or different component arrangements.

For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, thereby controlling power The device implements functions such as charge management, discharge management, and power consumption management. The power supply may also include any components such as one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

Further, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and/or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is usually used in the electronic device 1 Establish a communication connection with other electronic devices.

Optionally, the electronic device 1 may also include a user interface. The user interface may be a display (Display) and an input unit (such as a keyboard (Keyboard)). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light emitting diode) touch device, etc. Among them, the display can also be appropriately called a display screen or a display unit, which is used to display the information processed in the electronic device 1 and to display a visualized user interface.

It should be understood that the embodiments are only for illustrative purposes, and are not limited by this structure in the scope of the patent application.

The data compression program 12 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can realize:

Calculate the frequency of each character in the initial character set, filter out the redundant data set in the message data set in a preset manner according to the frequency of the character, identify the standard character set in the redundant data set, Encode the characters in the standard character set to obtain a character code table;

Specifically, for the specific implementation method of the above-mentioned instructions by the processor 10, reference may be made to the description of the relevant steps in the embodiment corresponding to FIG. 1, which will not be repeated here.

Further, if the integrated module/unit of the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory). The computer-readable storage medium may be non-volatile or volatile.

In the several embodiments provided in this application, it should be understood that the disclosed equipment, device, and method may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional modules in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware, or may be implemented in the form of hardware plus software functional modules.

For those skilled in the art, it is obvious that the present application is not limited to the details of the foregoing exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or basic characteristics of the application.

Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of this application is defined by the appended claims rather than the above description, and therefore it is intended to fall into the claims. All changes in the meaning and scope of the equivalent elements of are included in this application. Any associated diagram marks in the claims should not be regarded as limiting the claims involved.

In addition, it is obvious that the word "including" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claims can also be implemented by one unit or device through software or hardware. The second class words are used to indicate names, and do not indicate any specific order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application and not to limit them. Although the application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the application can be Make modifications or equivalent replacements without departing from the spirit and scope of the technical solution of the present application.

Claims

A data compression method, wherein the method includes:

Acquiring a message data set generated by the Internet of Things device during transmission, identifying a character set in the message data set, and using the character set as an initial character set;

Calculate the frequency of each character in the initial character set, filter out the redundant data set in the message data set in a preset manner according to the frequency of the character, identify the standard character set in the redundant data set, Encode the characters in the standard character set to obtain a character code table;

The minimum path length of the characters in the redundant data set is calculated by using the character code table, and the redundant data set is compressed according to the minimum path length of the characters.
5. The data compression method according to claim 1, wherein said identifying a character set in said message data set comprises:

Using the shortest path algorithm to identify the word character set in the message data set;

Identify the symbol character set in the message data set through a character recognition algorithm;

The character set is obtained by combining the word character set and the symbol character set.
3. The data compression method according to claim 2, wherein said using the shortest path algorithm to identify the word character set in the message data set comprises:

Construct a word segmentation directed acyclic graph through a custom dictionary. Among all paths from the starting point to the end point of the word segmentation directed acyclic graph, calculate the path set of the word length value, and obtain the The character set of the words contained in the path set.
3. The data compression method of claim 2, wherein the identifying the symbol character set in the message data set through a character recognition algorithm comprises:

A character database template is preset, the character database template is matched with the message data set using character matching technology, and the symbol characters in the message data set that are successfully matched are extracted to obtain a symbol character set.
5. The data compression method according to claim 1, wherein said calculating the frequency of each character in said initial character set comprises:

Use the following formula to calculate the frequency of each character in the initial character set:

Among them, f i represents the frequency of appearance of the initial character i, n i represents the number of characters i in the initial character set, and v represents the number of all characters in the initial character set.
5. The data compression method according to claim 1, wherein said encoding said standard character set to obtain a character code table comprises:

Preset the serial number character set of the standard character set, obtain the one-to-one correspondence between the standard character set and the serial number character set, and calculate the value of each standard character in the standard character set according to the one-to-one correspondence With probability, the code number of the corresponding standard character is formed according to the probability of each standard character, and the character code table is established according to the code number.
7. The data compression method according to any one of claims 1 to 6, wherein the calculating the minimum path length of the characters in the redundant data set by using the character code table comprises:

Use the following formula to calculate the minimum path length of the characters in the redundant data set:

Among them, MIN WPL represents the minimum path length of characters, w k represents the character weight in the k-th character code table, l k represents the character offset in the k-th character code table, and n represents the number of characters in the character code table.
An electronic device, wherein the electronic device includes:

At least one processor; and,

A memory communicatively connected with the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the following data compression method:

Acquiring a message data set generated by the Internet of Things device during transmission, identifying a character set in the message data set, and using the character set as an initial character set;

Calculate the frequency of each character in the initial character set, filter out the redundant data set in the message data set in a preset manner according to the frequency of the character, identify the standard character set in the redundant data set, Encode the characters in the standard character set to obtain a character code table;

The minimum path length of the characters in the redundant data set is calculated by using the character code table, and the redundant data set is compressed according to the minimum path length of the characters.
8. The electronic device according to claim 8, wherein said identifying a character set in said message data set comprises:

Using the shortest path algorithm to identify the word character set in the message data set;

Identify the symbol character set in the message data set through a character recognition algorithm;

The character set is obtained by combining the word character set and the symbol character set.
9. The electronic device according to claim 9, wherein said using the shortest path algorithm to identify the word character set in the message data set comprises:

A word segmentation directed acyclic graph is constructed through a custom dictionary. Among all the paths from the start point to the end point of the word segmentation directed acyclic graph, the path set of the word length value is calculated, and the path set is obtained according to a preset method. The character set of the words contained in the path set.
9. The electronic device according to claim 9, wherein said identifying the symbol character set in the message data set through a character recognition algorithm comprises:

A character database template is preset, the character database template is matched with the message data set using character matching technology, and the symbol characters in the message data set that are successfully matched are extracted to obtain a symbol character set.
8. The electronic device according to claim 8, wherein said calculating the frequency of each character in said initial character set comprises:

Use the following formula to calculate the frequency of each character in the initial character set:

Among them, f i represents the frequency of appearance of the initial character i, n i represents the number of characters i in the initial character set, and v represents the number of all characters in the initial character set.
The electronic device according to any one of claims 8 to 12, wherein said calculating the minimum path length of the characters in the redundant data set by using the character code table comprises:

Use the following formula to calculate the minimum path length of the characters in the redundant data set:

Among them, MIN WPL represents the minimum path length of characters, w k represents the character weight in the k-th character code table, l k represents the character offset in the k-th character code table, and n represents the number of characters in the character code table.
A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the following data compression method:

Acquiring a message data set generated by the Internet of Things device during transmission, identifying a character set in the message data set, and using the character set as an initial character set;

Calculate the frequency of each character in the initial character set, filter out the redundant data set in the message data set in a preset manner according to the frequency of the character, identify the standard character set in the redundant data set, Encode the characters in the standard character set to obtain a character code table;

The minimum path length of the characters in the redundant data set is calculated by using the character code table, and the redundant data set is compressed according to the minimum path length of the characters.
15. The computer-readable storage medium of claim 14, wherein said identifying a character set in said message data set comprises:

Using the shortest path algorithm to identify the word character set in the message data set;

Identify the symbol character set in the message data set through a character recognition algorithm;

The character set is obtained by combining the word character set and the symbol character set.
15. The computer-readable storage medium according to claim 15, wherein said using the shortest path algorithm to identify the word character set in the message data set comprises:

Construct a word segmentation directed acyclic graph through a custom dictionary. Among all paths from the starting point to the end point of the word segmentation directed acyclic graph, calculate the path set of the word length value, and obtain the The character set of the words contained in the path set.
15. The computer-readable storage medium according to claim 15, wherein the identifying the symbol character set in the message data set by a character recognition algorithm comprises:

A character database template is preset, the character database template is matched with the message data set using character matching technology, and the symbol characters in the message data set that are successfully matched are extracted to obtain a symbol character set.
15. The computer-readable storage medium of claim 14, wherein the calculating the frequency of each character in the initial character set comprises:

Use the following formula to calculate the frequency of each character in the initial character set:

Among them, f i represents the frequency of appearance of the initial character i, n i represents the number of characters i in the initial character set, and v represents the number of all characters in the initial character set.
18. The computer-readable storage medium according to any one of claims 14 to 18, wherein said calculating the minimum path length of the characters in the redundant data set by using the character code table comprises:

Use the following formula to calculate the minimum path length of the characters in the redundant data set:

Among them, MIN WPL represents the minimum path length of characters, w k represents the character weight in the k-th character code table, l k represents the character offset in the k-th character code table, and n represents the number of characters in the character code table.
A data compression device, wherein the device includes:

The recognition module is used to obtain the message data set generated by the Internet of Things device during the transmission process, identify the character set in the message data set, and use the character set as the initial character set;

The calculation and screening module is used to calculate the frequency of each character in the initial character set, filter out the redundant data set in the message data set in a preset manner according to the frequency of the character, and identify the redundant data set Standard character set;

The encoding module is used to encode the characters in the standard character set to obtain a character code table;

The compression module is configured to use the character code table to calculate the minimum path length of the characters in the redundant data set, and compress the redundant data set according to the minimum path length of the characters.