JP2018006964A - Data compression method, gateway, and data transmission system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a data compression method capable of solving conventional problems in a resource limitation IoT gateway and a common compression method, which are ineffective for a short packet transmission because the transmitted IoT data amount has to be reduced for limiting usable bandwidth; i.e., inefficient compression of a dictionary which does not reflect any optimum content; token size which does not reflect actual use of the dictionary; a temporal dictionary which is insufficiently used etc.SOLUTION: The data compression method temporarily reduces the token size when encoding the token by temporarily using a virtual dictionary for approximating continuous parts only in a used original dictionary while considering dictionary efficiency when storing/updating the dictionary to thereby efficiently transmit information required for a decoder to decode the token of the reduced size.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a data transmission system, and more particularly to a sensor data transmission system using data compression. More particularly, the present invention requires that the amount of Internet of Things (IoT) / sensor data transmitted must be reduced due to available bandwidth constraints, and common data to reduce the amount of data. The present invention relates to an IoT / sensor data transmission gateway having limited computational resources and memory resources in which a compression method is used.

  Combining inexpensive and interconnected sensors and increased machine-to-machine (M2M) communications in the Internet of Things (IoT), many of the data sources that often send data to a single or several remote sites Both a change in communication needs for a concentrated amount of IoT adapters and a change in communication needs for such IoT deployments due to real-time or pseudo real-time constraints on the transmission of sensor data occur.

  For IoT deployments in remote areas, often only suboptimal Internet connectivity is available, at least limiting the effectiveness of such deployments. The utilization of available network links must be optimized not only to keep the initial and maintenance costs low, but also to avoid expensive investments in remote uplinks.

  IoT / sensor data generally includes a fixed data structure and changing values, and a single sensor often produces data that includes only a limited amount of different data structures. The data transmission includes the data of one single sensor or combines the data of multiple sensors. Thus, a single sensor data stream often includes an IoT / sensor data stream that combines a large number of iterations and data from multiple sensors. Therefore, a deduplication mechanism that reduces redundancy is effective in reducing the amount of data being transmitted.

  Dictionary compression is a deduplication mechanism in which repetitive patterns are stored in a dictionary-like data structure and only a pointer to the dictionary is transmitted over the network. When the pointer to the dictionary is smaller than the pattern pointed to by the pointer, the amount of data to be transmitted can be reduced.

  Similar dictionaries must be used on both the transmitter and receiver sides. This is generally accomplished by running a similar dictionary creation algorithm on both sides. If there are multiple dictionaries, one must be selected for use and the selection must be performed synchronously. Selection is done by using either network header information, such as IP address or port number, or any statistical attribute of the data itself, and comparing that information with the information stored with the selection source dictionary. . Synchronous selection is performed by applying the same selection mechanism on both sides, or by first selecting at the sender and signaling the selection decision to the receiver by sending any form of marker or identifier. The

  If the data to be transmitted is largely known, fixed dictionaries can be prepared on both the transmitter and receiver sides, and only the data stream patterns present in these prepared dictionaries are compressed. While high compression efficiency can be reached for static content, dynamic content causes a decrease in overall compression efficiency. Dynamic content can be dealt with by using a dynamic dictionary adapted to the input data.

  A simple variant is to start with an empty dictionary and insert new patterns continuously until a finite size of the dictionary is reached. The newly added pattern must be transmitted as a non-pointer to the receiver side. This generally reduces the efficiency of data compression until the dictionary contains a sufficient number of repetitive input patterns. To improve efficiency during this dictionary generation phase, the dictionary can be pre-initialized.

  For example, in the Lempel-Ziv-Welch (LZW) compression algorithm referenced in Non-Patent Document 1, the dictionary is pre-initialized with every single character of the input alphabet. There is also a hybrid mechanism where the dictionary is pre-compressed with the expected pattern, but also updated dynamically. The referenced patent reference 1 uses both a predetermined dictionary and identifier as well as a dynamically created dictionary and identifier, but the dictionary once created is not updated, so for example with respect to changed input data Describes a mechanism that does not allow dictionary optimization. Referenced Patent Document 2 describes a communication apparatus that determines compression / extension information of data transmitted / received to / from a predetermined communication node. Upon receiving data from a given communication node, the device determines the compression rate of the received data based on a plurality of compression / expansion information used for data sent to / received from the given communication node. Calculate compression / extension information based on the calculated data compression rate.

US 7,953,881 "Network characteristic-based compression of network traffic", Jupiter Networks, 2011 Japanese Published Patent No. 2015-043502

“Technique for High-Performance Data Compression”, Welch, T .; A, IEEE Computer, 1984

  The size of the dictionary is finite and limits the number of entries included at any one time. The size of a dictionary pointer, i.e., a token, transmitted via the network is also limited by the maximum size of the dictionary. In general, the token size continues to increase with the number of items included in the dictionary up to the maximum size of the dictionary. If the use of dictionary entries is evenly distributed throughout the dictionary, for example when using a large number of dictionaries and encoding a large amount of data, the large token size overhead is due to the large number of patterns that can be encoded with the dictionary pointer. Balanced.

  However, in IoT / sensor data transmission, in many cases, a very small amount of data, so-called short packets, must be transmitted using only a very small amount of the entire dictionary. As a result, the overhead of the dictionary pointer has a negative effect on the compression rate and causes a reduction in compression efficiency. The solution is to reduce the maximum size of the dictionary, but this makes the applicability of the dictionary very narrow, only the most recent transmission data or, in the case of transmissions from multiple sensors, only compression of the common pattern. Focus on either and reduce the overall efficiency of compression.

  Reusable dynamic dictionaries with a dictionary compression mechanism that uses multiple dynamic dictionaries are not updated after they are first created, or are updated unconditionally without considering the compression efficiency over time. It becomes an inefficient and static dictionary or a very time-specific dictionary. This reduces the efficiency of compression, especially when various patterns of input data change over time.

  The present invention aims to improve the compression efficiency of IoT / sensor data transmission.

One aspect of the present invention is a data compression method for a data transmission system that uses transfer data compression by using a session dictionary. This method
A first step for preparing a plurality of permanent dictionaries in a permanent dictionary store;
A second step for selecting one of the plurality of permanent dictionaries and duplicating the selected permanent dictionary to prepare the session dictionary in a temporary session memory;
A third step for compressing data transferred using the session dictionary;
A fourth step to evaluate the efficiency of the session dictionary;
And a fifth step for identifying based on efficiency whether the session dictionary is to be stored in the permanent dictionary store as a permanent dictionary. More preferably, the second step is performed when setting up the session, and the third step is performed during the session and includes a step for modifying the session dictionary in the temporary session memory. Then, the fourth step and the fifth step are repeatedly performed during the session.

  Another aspect of the present invention is a gateway for data transfer including an interface coupled to a processor, memory, and a network. During session setup, the processor selects one of a number of compressed dictionaries in a permanent dictionary store in memory based on network parameters for the session. The processor copies the selected compression dictionary to the session store of the memory as a session dictionary. The processor compresses session data traffic in the session dictionary while updating the session dictionary and usage counters corresponding to the session dictionary. The processor makes a decision whether to store the session dictionary in a permanent dictionary store based on the session dictionary usage counter.

  Another aspect of the present invention is a data transmission system for transferring data from a transmitter device to a receiver device by using a session dictionary and compressing the data. The transmitter device includes a processor and a memory. The memory includes a first database and a second database, wherein the first database stores a plurality of combinations of permanent dictionary and permanent dictionary efficiency parameters. The processor selects one permanent dictionary in the first database based on the data and creates a copy of the selected permanent dictionary in the second database as the session dictionary. The processor compresses the data by using the session dictionary, and the processor modifies the session dictionary and monitors and updates session dictionary efficiency parameters. The processor determines whether the permanent dictionary should be overwritten by the session dictionary based on the efficiency parameters of the permanent dictionary and the session dictionary.

  The present invention can improve compression efficiency for transmission of IoT / sensor data.

It is a component block diagram which shows the outline | summary of the environment relevant to a present Example, and the component which interacts with it. It is a block diagram which shows the functional component relevant to a present Example. It is a figure which shows the outline | summary of the component regarding the dictionary reuse operation | movement of a present Example, and its interaction. It is a figure which shows the outline | summary of the component regarding the token size shortening operation | movement of a present Example, and its interaction. It is a block diagram which shows the hardware component of a present Example. It is a sequence diagram which shows the dictionary reuse operation | movement of a present Example. It is a figure which shows the program flow of the dictionary reuse operation | movement of a present Example. Schema showing temporary session memory. FIG. 4 is a relationship diagram for a temporary session memory schema. A schema indicating a session identifier. A schema indicating a session dictionary. It is a schema which shows a session use counter. Dictionary ID A schema indicating TCP options. It is a schema which shows a permanent dictionary store. It is a relationship diagram of a permanent dictionary store schema. It is a schema which shows the stored session identifier. This is a schema indicating a stored dictionary. It is a schema which shows a stored usage counter usage counter. It is a figure which shows the program flow which shows a general dictionary encoder as a reference | standard. It is a figure which shows the program flow which shows a general dictionary decoder as a reference | standard. It is a figure which shows the program flow which shows the use counter manager which performs continuous dictionary performance evaluation regarding the dictionary reuse operation | movement of a present Example. It is a dictionary efficiency graph which shows the step which differs in the efficiency of the dictionary regarding data diversity and a dictionary pattern regarding the dictionary reuse operation | movement of a present Example. It is a figure which shows the program flow of the dictionary use examination before memorize | store or update regarding the dictionary reuse operation | movement of a present Example. It is a figure which shows the program flow of the dictionary performance examination before memorize | storing a new dictionary regarding the dictionary reuse operation | movement of a present Example. It is a figure which shows the program flow of the dictionary performance comparison before the update of the existing dictionary regarding the dictionary reuse operation | movement of a present Example. It is a sequence diagram which shows the token shortening operation | movement of a present Example. It is a figure which shows the program flow which shows the encoding in the IoT gateway by the side of a transmitter regarding the token size shortening operation | movement of a present Example. It is a figure which shows the program flow which shows calculation of a restriction | limiting dictionary token size regarding the token size shortening operation | movement of a present Example. It is a schema which shows a dictionary entry ID list. This is a schema indicating a restricted dictionary token. It is a figure which shows the program flow which shows the decoding in the receiver side IoT gateway regarding the token size shortening operation | movement of a present Example.

  The embodiment consists of two operations: a first optimization operation for dictionary reuse and a second optimization operation for token size. Both operations are used simultaneously, but the operations are described separately for ease of understanding. A short summary is followed by a more detailed explanation.

  In the first optimization operation, a dictionary compression efficiency measurement is performed, and the temporary session dictionary storage is tied to a condition relating to the dictionary compression efficiency, and the stored dictionary and reusable dictionary are continuously optimized. to enable.

  Compression efficiency is generally related to the amount of repetitive patterns available in the dictionary used. While dictionary reuse reduces the time it takes for compression to be efficient, the dictionary generally evolves with the input data being compressed, and so the efficiency of the dictionary has also changed. When this embodiment is used, the efficiency of the dictionary that can be used for reuse is optimized, and the efficiency of reuse of the general dictionary is further improved.

  In a second optimization operation, the length of the encoded token is reduced to represent only the range of dictionary entries that are actually used when encoding a finite block of input data. In general, a token may represent any entry in the dictionary that becomes less efficient as the subset of dictionary entries used is smaller. In particular, sensor data transmission consists of small bursts of data, thus allowing improved compression efficiency when limiting the range of tokens.

  In this example, encoding is performed in three steps, generating a list of dictionary entries that are used first, and in the second step, the list is used to calculate the limit range and limit token size. In a third step, a token having a restricted token size is generated using the previously generated list.

  More detailed disclosure of this embodiment follows. First, the environment of this embodiment is described using FIG. Subsequently, components of the functional blocks related to the present embodiment will be described using FIG. Subsequently, a general mode of operation of the present embodiment will be described. This description is divided into a first description of the mode of operation related to dictionary reuse using FIG. 3 and a second description related to token shortening using FIG.

  FIG. 1 shows the environment of this embodiment. The IoT devices # 1 100a, # 2 100b, and the like are connected to the transmitter-side IoT gateway 500a via the transmitter-side local area network (LAN) 300a. The transmitter-side IoT gateway 500a probably converts the data and efficiently transfers the data from the connected IoT device to one or more remote receiver-side IoT gateways 500b. The IoT gateway 500b on the receiver side probably converts the incoming data again, and transfers the incoming data to the following receivers # 1 200a, # 2 200b, etc. via the LAN 300b on the receiver side.

  The IoT gateways 500a and 500b use function blocks 600a and 600b, respectively, in this embodiment. Illustratively, a message transmission 150a between the IoT device # 1 100 and the transmitter-side IoT gateway 500a, and a message transmission 150b between the IoT device # 2 100b and the transmitter-side IoT gateway 500a are shown. The IoT device # 1 100a transmits a message # 1 400a having a header 402a and a payload 404a, and the IoT device # 2 100b transmits a message # 2 400b having a header 402b and a payload 404b. Messages between the two IoT gateways 500a, 500b are transmitted over the wide area network (WAN) 320.

  FIG. 2 shows the components of the functional block 600 of this embodiment, as well as the related data flow, control data flow, and information access flow between the different components.

  Reuse the following functional components, possibly stored data in the permanent dictionary store, select an existing temporary session memory block or create a new temporary session memory block for input data for a different TCP session (FIG. 24): And the relevant program flow shown in FIGS. 7, 23 and 26, considering storing data from the temporary session memory into the permanent dictionary store and possibly executing the session (including the subflow shown in FIG. 25) A handler 1600 and a token encoder 1700 that compresses raw input data and releases tokens in the related program flow shown in FIGS. 19 and 20 (including the subflow shown in FIG. 28), and FIGS. In the related program flow shown in FIG. A token decoder 1710 that decompresses input data and releases the decompressed raw output data, and a usage counter (Ctr.) Manager 1500 that periodically calculates dictionary efficiency in the related program flow shown in FIG. Is part of the functional block.

  Permanent dictionary store 1100, possibly including one or more permanent dictionary store blocks 1105a, each including session ID 1110a, usage counter 1130a, and dictionary 1120a, and session ID 1010a, usage counter 1030a, dictionary 1020a, respectively. And possibly one or more temporary session memory blocks 1005a including dictionary entry ID list 1040a are part of the functional block.

  The data flow, control data flow, and information access flow are described using the input network data 400c received by the session handler 1600, which then accesses the temporary session memory 1000 to receive the received network data. Probably select an existing temporary session memory block related to the header information. If the temporary session memory block is not found, the permanent dictionary store is accessed and probably selects the previously stored permanent dictionary store block for reuse when creating a new temporary session memory block. If the stored dictionary is not found, the temporary session memory is accessed again to create a new temporary session memory block.

  The received network data is then transferred to either the token encoder 1700 if it is on the transmitter side or the token decoder 1710 if it is on the receiver side. For example, related control information such as the session ID of the current session is transferred in the same manner. The token encoder 1700 accesses the temporary session memory and uses the session dictionary and dictionary entry ID list to encode the received data into a token. The token is then released and transmitted as output network data 450a and the relevant usage control information is forwarded to usage counter manager 1500. The token decoder 1710 accesses the temporary session memory and decodes the received data token into its original data while utilizing the session dictionary. The original data is then released and transmitted as output network data 450a and the relevant usage control information is forwarded to usage counter manager 1500.

  The usage counter manager 1500 accesses the usage counter in the relevant temporary session memory block and updates the relevant counter. The resulting information is transferred as control data to the session handler 1600, which then compares the temporary session memory block usage counter 1030a with the permanent dictionary store block usage counter 1130, possibly related temporary sessions. One can consider whether the memory block should be stored in a permanent dictionary store.

  3 and 4 are variations of the block diagram of FIG. 2 relating to specific embodiments showing both transmitter and receiver side components and data flow, control data flow, and information access flow. . FIG. 3 shows a dictionary reuse operation related to the first embodiment, and FIG. 4 shows a token size optimization operation related to the third embodiment. Both figures are described in more detail in their respective examples.

  For the description of the first embodiment, the following figures, that is, FIGS. 3 and 5 to 25 are used.

  FIG. 3 illustrates the components and the data flow between the components when data is received at the transmitter gateway, encoded, transmitted over the WAN, received by the receiver gateway, and decoded. It is a block chart of the dictionary reuse operation | movement of a present Example.

  FIG. 5 shows the hardware component blocks and the interconnection of the hardware component blocks to implement this embodiment.

  FIG. 6 is a sequence diagram showing a general operation of this embodiment.

  FIG. 7 is a program flow diagram showing a program flow for starting an operation related to the dictionary reusing operation of this embodiment, which is executed by the session handler 1600 every time a new session request from the network arrives.

  FIG. 8 is a data schema showing the data fields of the temporary session memory 1000 used to store data related to currently ongoing operations related to this embodiment.

  FIG. 9 is a data relationship diagram showing which data fields of other schemas are referenced by the temporary session memory schema 1000 of FIG.

  FIG. 10 is a data schema showing the data field of the session identifier used when selecting the session of the temporary session memory of FIG. 8 by comparing information from the network header of the received data packet.

  FIG. 11 is a data schema showing the data fields of the session dictionary used for data compression and decompression.

  FIG. 12 is a data schema showing session use counter information related to each temporary session memory of FIG.

  FIG. 13 shows the dictionary ID used when sending a dictionary ID between gateways to verify the existence of the selected dictionary at both gateways, for example after a stored dictionary from a permanent dictionary store is selected. It is a data schema which shows the data field of a TCP option.

  FIG. 14 is a data schema illustrating a data field permanent dictionary store 1100 that is used to store data related to a dictionary that is stored for later reuse.

  FIG. 15 is a data relationship diagram showing which data fields of other schemas are referenced by the permanent dictionary store of FIG.

  FIG. 16 illustrates a stored session used when retrieving a stored dictionary from the permanent dictionary store of FIG. 14 during the start of operations related to dictionary reuse operations, for example as shown in the program flow of FIG. It is a data schema which shows the data field of an identifier.

  FIG. 17 is a data schema showing the stored dictionary data fields used when reusing a stored dictionary at the beginning of a new session.

  FIG. 18 is a data schema showing data fields of stored usage counters including usage information relating to each permanent dictionary of FIG.

  FIG. 19 is a program flow diagram illustrating a simplified encoding operation program flow performed by the transmitter token encoder 1700 to compress input sensor data for additional transmission over the WAN. .

  FIG. 20 is a program flow diagram showing a simplified program flow of the decoding operation executed by the receiver-side token decoder 1710 to decompress the compressed data from the WAN.

  FIG. 21 is a program flow diagram showing a program flow of a usage counter management operation executed by the counter manager 1500 of both the transmitter-side gateway and the receiver-side gateway.

  FIG. 22 is a line graph illustrating an exemplary dictionary efficiency graph for a session dictionary, described over time with the related session usage counter of FIG.

  FIG. 23 illustrates a dictionary that is executed in both the transmitter and receiver session handlers 1700 to review and possibly perform storage of the session dictionary from the temporary session memory 1000 into the permanent dictionary store 1100. It is a program flow figure which shows the program flow of a reuse check operation | movement.

  FIG. 24 is a program flow diagram showing a program flow of a dictionary check operation that is executed as a subflow of the program flow of FIG. 23 in order to consider evaluating and storing a session dictionary.

  FIG. 25 compares the session dictionary in the temporary session memory 1000 considered for storage and the previously stored dictionary in the permanent dictionary store 1100 to see if the previously stored dictionary should be overwritten, or FIG. 24 is a program flow diagram showing a program flow of a dictionary comparison operation executed as a subflow of the program flow of FIG. 23 to determine whether a session dictionary should be deleted.

  FIG. 3 shows functional components and data flow of this embodiment. In this figure, the components and data flow of this embodiment are shown when input data is received, encoded, transmitted to the receiver gateway via the WAN and decoded again.

  FIG. 3 is similar to the functional block diagram of FIG. 2, but shows both the transmitter side and the receiver side, and the dictionary entry ID list 1040a is omitted because it is not relevant to this embodiment. The network message 400d including the header 402c and the payload 404c is received by the session handler 1600a on the transmitter side, and the session handler 1600a accesses the information in the permanent dictionary store 1100a and the temporary session memory 1000a and uses the information from the header 402c. , Perhaps using previously stored data from permanent dictionary store 1100a, and then sending both data and relationship control information to token encoder 1700a where the actual encoding of payload 404d occurs and relationship of temporary session memory store 1000a The information from the temporary session memory 1005b to be accessed is accessed, and an existing temporary session is selected or a new temporary session is created while releasing the token.

  The token is then released and transmitted as the payload 454a of the token message 450b, and the relevant control information is sent to the usage counter manager 1500a that accesses the session usage counter 1030b of the relevant temporary session memory 1005b. The usage counter result is then returned as control information to the session handler 1600a to allow consideration of the storage of temporary session memory.

  The token message 450b is transmitted via the WAN 320a and received by the receiver-side session handler 1600b. The receiver-side session handler 1600b accesses the information from the permanent dictionary store 1100b and the temporary session memory 1000b, and is similar to the transmitter side. An existing temporary session is selected by this method, or a new temporary session is created.

  The data and related control information are then transferred to the token decoder 1710a, which decodes the payload 454a of the token message 450b while accessing the related information from the temporary session memory 1005c. The resulting decoded data is then sent in the outgoing message 470a on the receiver side, the relevant control information is sent to the usage counter manager 1500b, and the usage counter manager 1500b accesses the information in the relevant temporary session memory 1005c. Then, control information is sent back to the session handler 1600b, and the session handler 1600b then considers storage of the temporary session memory.

  The effect of cooperation and data transmission between the different components shown in FIG. 3 is to enable the storage of a temporary session dictionary in a permanent dictionary store and a highly efficient dictionary compression of data between the two gateways. The subsequent synchronized reuse of such stored dictionaries at both the transmitter-side gateway and the receiver-side gateway.

  FIG. 5 shows the hardware component blocks and the interconnection of the hardware component blocks to implement this embodiment. The development of Example 1 may be similar to the development shown in FIG. The IoT gateway 500 includes a component 1800 shown in FIG. 5, that is, a central processing unit (CPU) 1820, a primary memory 1830, a secondary memory 1840, and a first network interface card (NIC), all connected to the messaging bus 1810. ) 1850a and a second NIC 1850b.

  The functional block 600 of this embodiment is stored in the form of program instructions in the secondary memory 1840, copied to the primary memory 1830 at initialization, and processed by the CPU 1820. Network communication between the IoT device and the IoT gateway may be performed using the first NIC 1850a, and network communication between the IoT gateway and another IoT gateway is performed using the second NIC 1850b. May be. The component 1800 represents the minimum configuration that may change without limiting the applicability of the embodiment. According to one standard embodiment, CPU 1820 executes a program stored in memory 1830 or 1840 to perform the functions described in this embodiment. It is also possible to use an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) instead of the CPU.

  FIG. 6 shows the general sequence of events for Example 1. The IoT device 100c transmits a session initialization request 7010 to the transmitter IoT gateway 500g. The IoT device 100c itself is not a part of this embodiment. The transmitter IoT gateway 500g creates an arbitrary session processing block 7020a based on the network header information of the initialization request 7010, and generates a session ID at 4500a.

  The transmitter IoT gateway 500g then searches and selects a permanently stored dictionary for use during the session at 4600a using a dictionary appropriate for the previously created session ID. The transmitter IoT gateway 500g then transmits an initialization request including the dictionary ID of the previously selected session dictionary to the receiver IoT gateway 500h at 4110a.

  The receiver IoT gateway 500 creates an arbitrary session processing block 7020b based on the network header information of the initialization request 4110a, and generates a session ID at 4500b. Subsequently, the receiver IoT gateway 500 has a dictionary ID received at 4110b and is a dictionary suitable for the session ID calculated at 4500b and a permanently stored dictionary for use during the session at 4600b. Search for and then select. If such a dictionary is found, it is used as a session dictionary.

  The receiver IoT gateway 500h then sends a session initialization request 7012 to the receiver 200c that acknowledges the request at 7014. At 7030, the receiver IoT gateway 500h returns an acknowledgment including the dictionary ID of the session dictionary to the transmitter IoT gateway 500g, and the transmitter IoT gateway 500g similarly acknowledges the session initialization request to the IoT device 100c at 7040. At this time, both the transmitter IoT gateway 500g and the receiver IoT gateway 500h have selected the same session dictionary for further use when encoding and decoding data. At 7050, the IoT device 100c transmits sensor data encoded using the session dictionary previously selected by the transmitter IoT gateway 500g at 4300a, and a token message including the encoded data is transferred at 7060. Is done.

  Receiver IoT gateway 500h decodes the data received using the session dictionary previously selected at 4400a and forwards the decoded data to receiver 200c at 7070.

  After transmitting the encoded data, the transmitter IoT gateway 500g updates its session usage counter at 6000a. Subsequently, the transmitter IoT gateway 500g starts checking the compression efficiency of the session dictionary at 4800a, and after finding a compression efficiency sufficient to permanently store the session dictionary, the current session ID generated at 4500a. After searching for a permanently stored dictionary suitable for the search, the compression efficiency of the found dictionary is compared with the session dictionary in 4900a after finding the permanently stored dictionary. It turns out that the compression efficiency of the session dictionary is excellent, and then the dictionary ID of the session dictionary is generated with the session dictionary stored 5500a and permanently, overwriting the previously found permanent dictionary.

  The receiver IoT gateway 500h updates the session use counter at 6000b after transferring the decrypted data. At this time, both the session dictionary at the transmitter IoT gateway 500g and the session dictionary at the receiver IoT gateway 500h include the same contents as a result of using the same dictionary compression mechanism. Further, the usage counters of both the transmitter IoT gateway 500g and the receiver IoT gateway 500h include the same contents because the uncompressed data amount and the compressed data amount are the same. Subsequently, the receiver IoT gateway 500h starts checking the compression efficiency of the session dictionary at 4800b and after finding a compression efficiency sufficient to permanently store the session dictionary, the current session generated at 4500b. After searching the permanently stored dictionary appropriate for the ID and finding the permanently stored dictionary, the compression efficiency of the found dictionary and the session dictionary is compared in 4900b. It turns out that the compression efficiency of the session dictionary is excellent, and then the session dictionary dictionary ID is generated with 5500b and the permanently stored session dictionary, overwriting the previously found permanent dictionary.

  Because the usage counter is similar, the results of blocks 4800b, 4900b, and 5500b performed at receiver IoT gateway 500h are similar to the results of blocks 4800a, 4900a, and 5500a performed at transmitter IoT gateway 500g. . Thus, here both the transmitter IoT gateway 500g and the receiver IoT gateway 500h updated the permanent dictionary previously stored with the current session dictionary.

  The sequence steps that start with data transmission from the IoT device at 7050 are the creation of the dictionary ID of the session dictionary and the storage of the session dictionary in the permanent dictionary store of the transmitter IoT gateway 500g in 5500a and in the receiver IoT gateway 500h in 5500b. Until, for example, all data is transmitted by the IoT device 100c, and is generally repeated many times.

  FIG. 7 shows the program flow 4000 of Example 1 relating to dictionary selection at session initialization, which is executed by the session handler 1600 after a new session request is received from the network at 4030, starting at block 4010.

  Subsequently, a session identifier is generated 4040 based on the network packet information of the received network packet including the session request, eg, IP destination address and TCP destination port. At 4050, it is checked whether the current program flow is executed at the transmitter gateway.

  If so, the generated session identifier is used at 4054 to search for a permanent dictionary store block that contains a matching session identifier for the permanent dictionary store. If a matching stored dictionary is found at 4056, a dictionary ID TCP option is created at 4100 using the dictionary ID of the previously found dictionary.

  Thereafter, a new session request including the previously created dictionary ID TCP option is sent to the WAN at 4110. An acknowledgment is received at 4120 and at 4130 the acknowledgment is checked for dictionary ID TCP options. At 4140, it is checked whether such a TCP option has been found. If such a TCP option is found, both the dictionary ID TCP option dictionary ID and the previously found dictionary ID of the stored dictionary are compared 4150. At 4160, the result of the previous comparison is checked. If both dictionary IDs are equal, the program flow jumps from 4164 outgoing label SO to 4166 incoming label SO. At 4170, a session dictionary is created from previously stored stored dictionaries.

  Thereafter, at 4180, a new session usage counter is created. Subsequently, at 4190, it is checked again whether the program flow is executed at the transmitter gateway. If yes, the program flow ends at 4290. If no matching dictionary is found at 4056, a new session dictionary is created at 4060 and a new session usage counter is created at 4070. Subsequently, at 4075, it is checked again whether the program flow is executed at the transmitter gateway.

  If yes, additional session setup is performed at 4080 without using a stored dictionary and the flow ends at 4290. If not, the flow ends immediately at 4290. If the dictionary ID TCP option is not found at 4140, the program flow jumps from the outgoing label SN at 4244 to the incoming label SN at 4246, continues at 40060 to create a new session, and the subsequent steps as described above.

  If the compared dictionary IDs are different in the 4160 check, the program flow jumps from the 4244 outgoing label SN to the 4246 incoming label SN and continues as described above. If the current program flow is executed at the receiver gateway at check 4050, then the received packet is checked at 4230 for the Dictionary ID TCP option. At 4240, it is checked whether such a TCP option is found. If yes, the found dictionary ID and generated session identifier are used at 4250 to search the permanent dictionary store block of the permanent dictionary store that includes both a dictionary with a matching dictionary ID and a matching session identifier. The

  At 4260, it is checked whether such a permanent dictionary store block has been found. If yes, program flow continues at block 4170 as described above. If not, at 4280, a session setup acknowledgment is sent to the transmitter gateway without a dictionary ID.

  The program flow then jumps from the outgoing label SN 4244 to the incoming label SN 4246 and continues as described above. If the dictionary ID TCP option is not found in the 4240 check, the program flow jumps from the outgoing label SN of 4244 to the incoming label SN of 4246 and continues as described above. If inspection of 4190 reveals that the current program flow is to be executed at the receiver-side gateway, at 4270, a dictionary ID TCP option including a dictionary is generated, and a session setup including the generated dictionary ID TCP option. An acknowledgment is sent to the transmitter gateway.

  The effect of the program flow of FIG. 7 is in the reuse of a previously stored highly efficient dictionary containing patterns targeted to efficient compression of any application data identified by network parameters, thereby If both the transmitter side and the receiver side do not have an available selected dictionary, both use the same dictionary, including the possibility of using an unused or empty dictionary.

  FIG. 8 shows a schema of the temporary session memory 1000e according to the first embodiment, explaining a data structure possible for storing the temporary session memory in the primary memory 1830 of FIG. 5, for example.

  The session ID field 1002 refers to an internal TCP session for the IoT gateway 500 and allows mapping between this session and its respective temporary session memory block. The session identifier ID field 1004 refers to the ID field 2202b of the session identifier schema 2200b. The session dictionary ID field 1006 refers to the session dictionary ID field 2360a of the session dictionary store 2300b. The usage counter ID field 1008 refers to the ID field 2102b of the session usage counter table 2100b. The used list ID field 1009 refers to the list ID field 2710 of the dictionary entry ID list store 2700.

  An exemplary temporary session memory entry 1005h is used with session ID 6 and is referenced by session identifier ID 23, session identifier referenced by session dictionary ID 223, session dictionary referenced by usage counter ID 65, and used list. The dictionary entry ID list referenced by the ID 233 is used.

  The effect of the schema of FIG. 8 is to manage the data structure related to dictionary encoding and decoding, as well as its related dictionary usage, and the efficiency calculation of multiple simultaneous sessions in memory, enabling efficient access and comparison. There is a possibility to do.

  FIG. 9 shows how the above-described temporary session memory schema of FIG. 8 refers to the different related schemas of FIGS. 10, 11, 12, and 30. The related schema will be described later. The session ID field 1002 of the temporary session memory is a unique identifier of each temporary session.

  The session identifier field 1004 refers to the ID field 2202b of the session identifier schema in FIG. The session identifier ID field 1006 refers to the session dictionary ID field 2160b of the session dictionary schema of FIG. The usage counter ID field 1008 refers to the ID field 2102b of the session usage counter schema in FIG. The used list ID field 1009 refers to the list ID field 2710 of the dictionary entry ID list schema of FIG. This field is not relevant to this example, but is relevant to Example 3 and subsequent examples, where it will be described in more detail.

  FIG. 10 shows a session identifier table 2200b according to the first embodiment, which explains a possible data structure for storing the session identifier in the primary memory 1830 of FIG. 5, for example. The session identifier table 2200b includes an ID field 2202b including a unique identifier for each session identifier entry of the session identifier table 2200b, a destination IP address field 2204b including the IP address of the WAN remote receiver 200, and the WAN remote receiver 200. In the destination TCP port field 2206b including the destination TCP port number.

  Exemplary session identifier 1010g includes ID 23, destination IP address 10.100.1, and destination TCP port number 1883. The effect of the schema of FIG. 10 resides in the storage of relevant network header data identifiers with the dictionary currently in use to allow quick selection of the relevant dictionary upon receipt of network packets.

  FIG. 11 shows the session dictionary store 2300b of the first embodiment, describing a possible data structure for storing the session dictionary in the primary memory 1830 of FIG. 5, for example. Zero or one or more session dictionaries 1020 can be stored in the session dictionary store 2300b, each clearly identified by a session dictionary ID 2360a.

  The session dictionary is uniquely identified by each session dictionary 1020 with a unique entry ID 2320b, a pattern 2330b containing a string or byte pattern, and a usage counter 2340b containing the number of usages of the dictionary entry being encoded or decoded. One or more entries. The usage counter 2340b can be used to create an ordering of dictionary entries by usage. In another embodiment, usage counter 2340b is released from session dictionary 1020.

  In another embodiment, the session dictionary 1020 includes more or other fields, allowing the ordering of dictionary entries by other information. An exemplary session dictionary 1020j is identified by session dictionary ID 223 in session dictionary ID field 2360b, has ID 27, includes the string pattern “Timestamp>”, and has at least two first entries 2350c used 12511 times. Contains one dictionary entry.

  FIG. 12 shows a session usage counter table 2100b according to the first embodiment for explaining a possible data structure for storing the session usage counter in the primary memory 1830 of FIG. 5, for example.

The session usage counter table 2100b includes a session usage counter entry 1030 and includes the following fields: an ID 2102b that uniquely identifies each session usage counter entry in the session usage counter table 2100b, and raw input data of the transmitter-side IoT gateway 500a. Or the uncompressed byte field 2104b containing the amount of processed output data of the IoT gateway 500b on the receiver side, the amount of processed output data of the IoT gateway 500a on the transmitter side, or the unprocessed of the IoT gateway 500b on the receiver side A compressed byte field 2106b containing the amount of input data, a Ratio _n-1 field 2108b containing a floating point number in the ratio of the previous compressed byte to the uncompressed byte, and the previous compressed byte to the uncompressed byte. Ratio change _n-1 field 2109b containing a floating point number that is the ratio of the ratio of the previous previous compressed byte to the uncompressed byte to the ratio of the _first .

  The first example of the session usage counter 1030h includes ID 65, receives 4223372 bytes of uncompressed input data, receives 3406483 compressed bytes, a previous compression rate of 0.76888, and a previous compression rate of 0.719096 Caused changes. This means that the compression efficiency of the sessions involved has improved last time and is now worse, but the overall trend cannot be identified.

  The effect of the schema of FIG. 12 lies in the management of a small number of usage parameters and the relational efficiency calculation of the encoding and decoding operations, and in effect the efficiency calculation of its relational dictionary in memory, thereby making access efficient. And related calculations can be performed quickly and efficiently.

  FIG. 13 illustrates the dictionary ID of Example 1 that illustrates a possible embodiment of the TCP option used to send the dictionary ID to verify that the same dictionary is available on both the sender and the receiver. TCP option 2500 is shown. The dictionary ID TCP option 2500 includes a TCP option ID field 2150, a TCP option length field 2520, and a dictionary ID field 2530. Both the TCP option ID field 2510 and the TCP option length field 2520 have the meaning described by the specification document relating to the Transmission Control Protocol (TCP) standard.

  The dictionary ID field 2530 contains the 16-byte dictionary ID of the stored dictionary of the permanent dictionary store. In another embodiment, dictionary ID field 2530 contains the truncated dictionary ID of the stored dictionary. The use of a truncated dictionary ID increases the risk of ambiguity when referring to the dictionary remotely, but this risk is mitigated when used in combination with a session identifier.

  The effect of the dictionary ID TCP option of FIG. 13 enables efficient synchronization of dictionary selection so that both the transmitter and receiver select the same dictionary, thereby compressing the receiver side transmitted from the transmitter side. The original data can be extracted from the data stream.

  FIG. 14 shows the schema of the permanent dictionary store 1100e of the first embodiment, describing a possible data structure for storing the permanent dictionary store in the primary memory 1830 of FIG. 5, for example.

  The ID field is a unique number for the store 1100e that allows a clear reference to a single entry. The session identifier field 1104 refers to the ID field 2202a of the stored session identifier table 2200a. The dictionary ID field 1106 refers to the dictionary ID field 2310 of the stored dictionary 1120 of the stored dictionary store 2300. The usage counter ID field refers to the ID field 2102a of the stored usage counter table 2100a.

  The example permanent dictionary block 1105k of the permanent dictionary store 1100e refers to a stored session identifier identified by a unique ID1 and having an ID of 19, a stored dictionary having an ID of cc7aabc37e1c7d2fe98be9185bdfe43b, and a stored usage counter having a usage counter ID of 151. .

  The effect of the permanent dictionary store schema of FIG. 14 is in the efficient management of data relating to dictionary storage and subsequent reuse of the dictionary in memory, which makes access more efficient and allows dictionary selection and possible for overwriting. The evaluation can be efficient and effective, particularly in conjunction with the relevant temporary session memory data of FIG.

  FIG. 15 illustrates how the above-described permanent dictionary store of FIG. 14 refers to the different related schemas of FIGS. 16, 17 and 18. The related schema will be described later.

  The session ID field 1102 of the permanent dictionary store is a unique identifier for each stored session. The session identifier ID field 1104 refers to the ID field 2202a of the stored session identifier schema in FIG. The dictionary ID field 1106 refers to the dictionary ID field 2310a of the stored dictionary schema in FIG. The usage counter ID field 1108 refers to the ID field 2102a of the stored session usage counter schema of FIG.

  FIG. 16 shows a stored session identifier table 2200a according to the first embodiment for explaining a possible data structure for storing the stored session identifier in the primary memory 1830 of FIG. 5, for example. In general, stored session identifier table 2200a includes fields similar to session identifier table 2200b. The session identifier table 2200b is stored in the temporary session memory, while the stored session identifier table 2200a is stored in the permanent dictionary store.

  The stored session identifier includes an ID field 2202a that includes a unique identifier for each stored session identifier entry in the stored session identifier table 2200a, a destination IP address field 2204a that includes the IP address of the remote receiver 200 of the WAN, and a WAN A destination TCP port field 2206a including the destination TCP port number at the remote receiver 200 is included. Exemplary stored session identifier 1110d includes ID 19, destination IP address 10.0.0.1, and destination TCP port number 1883.

  FIG. 17 shows the stored dictionary store 2300a of the first embodiment for explaining a possible data structure for storing the stored dictionary, for example, in the primary memory 1830 of FIG. In general, the session dictionary 1020 can be stored as a stored dictionary 1120 in the stored dictionary store 2300a, and thus both the session dictionary 1020 and the stored dictionary 1120 are very similar. When storing a dictionary, the dictionary ID is calculated and stored in the dictionary ID field 2310a of the stored dictionary 1120. This number is a 16-byte MD5 hash that clearly identifies the contents of the dictionary and is different from the session dictionary ID field 2360a of the session dictionary 1020.

  In the stored dictionary store 2300a, zero or one or a plurality of stored dictionaries 1120 can be stored. Stored dictionaries are clearly identified by a usage counter 2340a that includes a unique entry ID 2320a for each stored dictionary 1120, a pattern 2330a that includes a string or byte pattern, and the number of used dictionary entries being encoded or decoded. Contains one or more entries. The usage counter 2340a can be used to create an ordering of dictionary entries by usage.

  In other embodiments, stored dictionary 1120 includes fields similar to session dictionary 1020. The example stored dictionary 1120d has at least two dictionary entries identified by the dictionary ID cc7aabc37e1c7d2fe98be9185bdfe43b, has ID27, includes the string pattern “SetEquipmentStatus>”, and has a first entry 2350a used 51216 times. Including.

  FIG. 18 shows a stored usage counter table 2100a according to the first embodiment for explaining a possible data structure for storing the stored usage counter, for example, in the primary memory 1830 of FIG. In general, the session usage counter 1030 can be stored in the stored usage counter table 2100a as a stored usage counter 1130, so that both fields of the session usage counter 1030 and the stored usage counter 1130 are similar.

The stored usage counter table 2100a includes a stored usage counter entry 1130 and includes the following fields: an ID 2102a that uniquely identifies each stored usage counter entry in the stored usage counter table 2100a, and the IoT gateway 500a on the transmitter side. Uncompressed byte field 2104a containing either the amount of unprocessed input data or the amount of processed output data of the IoT gateway 500b on the receiver side, the amount of processed output data of the IoT gateway 500a on the transmitter side or the receiver side A compressed byte field 2106a containing either the amount of raw input data of the IoT gateway 500b, a Ratio _n-1 field 2108a containing a floating point number of the ratio of the previous compressed byte to the uncompressed byte, and the previous compressed byte A Ratio change _n-1 field 2109a containing a floating point number that is the ratio of the ratio of the previous previous compressed byte to uncompressed byte to the ratio of uncompressed byte.

  A first example of stored usage counter 1130d includes ID 46, receives 2103636 bytes of uncompressed input data, 1185188 bytes of compressed bytes, a previous compression ratio of 0.644605, and a previous compression ratio of 0.883436. Caused changes. Good compression efficiency was slightly degraded but recovered again.

  FIG. 19 shows a simplified program flow of the encoder 4300b of the first embodiment in order to better understand how the information necessary for counting the usage amount is generated. This flow is included to support references. Details of the dictionary compression or encoding mechanism are omitted because the details are not part of this embodiment.

  The flow starts at step 4310, and is executed by the token encoder 1700 of the transmitter-side IoT gateway 500a at step 4315. First, the data block is read at 4320. This data block is usually any receive buffer filled with new data received from the network interface. At step 4330, dictionary encoding is performed on the read block using the current temporary session dictionary, and tokenized output data is emitted.

  Thereafter, at step 4340, the size of the read data is added to the uncompressed byte field 2104b of the session usage counter 1030 referenced by the usage counter ID field of the current temporary session memory block, and released at step 4350. The size of the encoded data is added to the compressed byte field 2106b of the session usage counter referenced by the usage counter ID field of the current temporary session memory block. In the next step 4315, the program ends.

  FIG. 20 shows a simplified program flow for a decoder 4400b similar to the encoder program of FIG. This flow starts at step 4410 and is executed by the token decoder 1710 of the IoT gateway 500b on the receiver side as described in step 4415. First, the encoded block of data is read at 4420.

  Thereafter, dictionary decryption is performed at step 4430 using the current temporary session dictionary and the decrypted data is released. In general, input data is read from the receive buffer and output data is written to the transmit buffer. In one embodiment, both the receive buffer and the transmit buffer are part of the primary memory 1830 of FIG.

  In subsequent step 4400, the size of the input data is added to the compressed byte field 2104b of the session usage counter 1030 referenced by the current temporary session memory block usage counter ID field, after which the size of the output data is Is added to the uncompressed byte field 2106b of the session usage counter referenced by the usage counter ID field of the temporary session memory block. Finally, the program ends at step 4415.

  FIG. 21 illustrates a usage counter manager 1500 that illustrates operations related to accessing and updating the session usage counter during or after data encoding or decoding, for example, to evaluate the efficiency of the dictionary used. Fig. 7 illustrates a usage counter manager flow 6000c of Example 1 that is executed.

The usage counter manager flow 6000c starts at step 6010 and, as described in step 6015, not only the usage counter manager 1500b of the IoT gateway 500b on the receiver side, but also the usage counter manager 1500a of the IoT gateway 500a on the transmitter side. Is executed. In step 6020, time has elapsed since the previous dictionary efficiency was calculated using the current time _tnow and the previous time stamp _tprev . In step 6030, it is checked whether the elapsed time exceeds any defined limit ε. If the check is positive, the contents of the Ratio _n-1 field of the session usage counter of FIG. 12 referenced by the current temporary session memory block usage counter ID field 1008 is written to Ratio _n-2 in step 6040. .

Thereafter, in step 6050, the current compression ratio Ratio _n-1 is determined by the contents of the compressed byte field of the session usage counter referenced by the current temporary session memory block usage counter ID field 1008 and the current temporary session memory block usage. is calculated using the contents of the uncompressed byte field session usage counter referenced by the counter ID field 1008, the result is, Ratio _n session usage counter referenced by the usage counter ID field 1008 of the current temporary session memory block _{-Written in the 1} field.

Subsequently, at step 6060, the ratio change is calculated using the previously calculated Ratio _n-1 field and the previously calculated Ratio _n-2 , and the result is the current temporary session memory block usage. It is written to the ratio change _n-1 of the session use counter referenced by the counter ID field 1008. In the following step 6070, the previous time stamp t _prev is updated to the current time t _now and the program flow ends in step 6099. If the test at step 6030 is negative, program flow continues directly at end step 6099. In another embodiment, the checking of steps 6020 and 6030 and related conditional statements in program flow 6000c depend on the amount of processed data instead of the elapsed time. The effect of the flow of FIG. 21 lies in an efficient calculation of dictionary efficiency based on previously stored and updated dictionary usage parameters.

  FIG. 22 shows three graphs of the dictionary efficiency of the temporary session dictionary for the purpose of explaining the reasoning behind usage counter measurements and conditions examined in the following dictionary reuse store check flow of Example 1. Yes. The graph shows three standard changes in dictionary efficiency, as described in the following paragraphs.

  The graph shows the change in compression rate, that is, the compression efficiency over time. The lower the compression ratio, the higher the compression gain. In another embodiment, a relationship with the amount of processed data can be used instead of a relationship with time. The top graph A9510 shows the standard stage when data that hardly changes is processed, the dictionary initially started with almost no appropriate entries, so that the compression ratio improves over time. You are quickly adding the appropriate pattern. The current ratio, current ratio change, and previous ratio change all indicate an improvement in efficiency.

  In the second bluff B9520, the dictionary entry matches well with the data being processed and a slight change in the compression ratio occurs, but the overall compression ratio remains the same. The current ratio indicates efficiency, or the previous ratio change indicates an improvement in efficiency, or a comparison of the previous change and the penultimate ratio change indicates an improvement trend.

  In graph C9530, the dictionary does not match incoming data and cannot be adjusted quickly enough, not only causing momentary inefficiency, but also showing a tendency to degrade.

  FIG. 23 is a program flow diagram showing a program flow 4700 of a dictionary reuse check that is performed by both the transmitter and receiver session handlers 1700 to consider storing a session dictionary in a permanent dictionary store. is there.

  The program flow begins at step 4710a and is periodically executed by the session handler 1700 at 4710a. Subsequently, the dictionary efficiency is checked at 4800d by calling the check usage subflow of FIG. If the check is positive if the result is “sufficient” at 4720a, then at 4722a, a dictionary with a matching session identifier is searched in the permanent dictionary store.

  If a similar dictionary is found at 4725a, both the current session usage counter and the found stored usage counter are compared at 4900 by invoking the usage comparison subflow of FIG. If the result of 4730a is positive if the result is “Better Stored Usage Counter”, program flow continues at step 4735a to generate MD5 hashes for all dictionary entry IDs and dictionary patterns. By doing so, the dictionary ID of the temporary session dictionary currently used is generated. At 4740a, the previously found permanent dictionary store block dictionary is then overwritten with the current temporary session dictionary and the dictionary ID is added. At 4745a, the previously used permanent dictionary store block stored usage counter is then overwritten with the current session usage counter.

  In the following step 4799a, the program flow ends. If the test at 4730a is negative, that is, the released result of the 4900 called subflows is "Better Stored Usage Counter", then the program flow ends immediately at step 4799a.

  If the check at 4725a is negative, that is, if no matching dictionary is found in the search for the stored dictionary located at step 4722a, the dictionary ID of the currently used temporary session dictionary is the same as step 4735a described above. In addition, an MD5 hash is generated for all dictionary entry IDs and patterns in the dictionary, which is generated at 4750a.

  Subsequently, at 4755a, a new stored dictionary is generated with the contents of the current temporary session dictionary and the previously generated dictionary ID. Subsequently, at step 4760a, a new stored session identifier is generated with the contents of the current temporary session identifier. Thereafter, at step 4765a, a new stored usage counter is generated with the contents of the current temporary session usage counter.

  The data structure generated in the previous three steps 4755a, 4760a, and 4765a is permanently stored in a new permanent dictionary store block of the permanent dictionary store. The program flow then ends at step 4799a. If the test at 4720a is negative, that is, the released result of the subflow called at the previous step 4800d is “insufficient”, then the program flow ends immediately at step 4799a.

  The effect of the program flow of FIG. 23 is the repeated evaluation and review of dictionary efficiency, eg continuous, so that the dictionary reaches its maximum regardless of the actual session state or state changes. Then you can remember. This allows, for example, evaluation of dictionary efficiency changes each time a data packet is received and compressed.

  If the compression efficiency calculation and the associated evaluation are performed each time the session usage counter is updated, the evaluation is called continuous. Otherwise, the assessment is periodic if a fixed distance between each assessment-either in terms of amount of data or duration-is used. One such example is a periodic evaluation, for example every 100 Kbytes, after a fixed number of uncompressed bytes. Another example is a periodic evaluation after a fixed time, for example every hour.

  Such repeated evaluations will detail the amount of relevant information collected and combined in the session, and the actual results of such efficiency evaluations, until it is used for related considerations of dictionary efficiency evaluation and session dictionary storage. Allows fine-tuning with finer levels.

  FIG. 24 is a dictionary usage check program of embodiment 1 illustrating the operation for checking whether the efficiency of the currently used temporary session dictionary is sufficient to be permanently stored in the permanent dictionary store. A flow 4800 is shown.

The program flow begins with dictionary reuse store check flow 4700 at step 4810. In the next step 4820, the number of raw input bytes in the uncompressed byte field of the session usage counter referenced by the current temporary session memory block usage counter field 1008 is evaluated. In the next step 4825, it is checked whether the number of uncompressed bytes exceeds an arbitrary limit σ. In another embodiment, the number of processed bytes is used. If the result is positive, the program flow continues at 4830 and the current compression ratio Ratio _n is calculated using the previously read uncompressed and compressed byte field values of the same session usage counter. The

Thereafter, in step 4835, it is checked whether the calculated compression ratio Ratio _n is equal to or less than an arbitrary threshold value β. If the test is positive, program flow continues at step 4870, emitting a signal “sufficient” to inform the caller that dictionary efficiency is sufficient to store. Finally, at step 4899, the program ends.

If the check at step 4835 is negative, program flow continues at step 4840, where the ratio change Ratio change _n between the previously calculated compression ratio Ratio _n-1 and the current compression ratio Ratio _n is _equal to step 4830 using the ratio _n-1 previously computed stored in calculated ratio _n-1 field in the current ratio ratio _n and session usage counter referenced by using counter field 1008 of the current temporary session memory block Is calculated. In the next step 4845, it is checked whether the calculated ratio change Ratio change _n is below an arbitrary threshold γ.

If the test is positive, program flow continues at step 4870 as described above. If the check is negative, program flow continues at step 4850, where the change in ratio change is the previously calculated ratio change ratio of the session usage counter referenced by the usage counter field 1008 of the current temporary session memory block. Calculated using the contents of the change _n and Ratio change _n-1 fields. In the following step 4855, it is checked whether the result of the calculation is below an arbitrary threshold value δ.

  If the result is positive, the program flow continues at 4870 as described above. If the result is negative, the program flow continues at step 4860, emitting the signal “insufficient”, notifying the caller that the dictionary efficiency is not efficient for storage, and finally stepping. It ends at 4899. If the test at step 4825 is negative, program flow continues at step 4860 as described above.

  The effect of the program flow of FIG. 24 is a very efficient evaluation of whether an existing session dictionary should be stored, so that all stored dictionaries have high efficiency.

  FIG. 25 shows an embodiment involving a comparison of the currently used temporary session dictionary and the stored dictionary of the permanent dictionary store block to evaluate whether the temporary session dictionary should overwrite a previously stored dictionary. 1 shows a usage comparison program flow 4900.

The program flow begins at step 4910 and is executed as a subflow by the dictionary reuse store check flow 4700 as shown in step 4915. At the start of the current program flow, the stored dictionary is already selected. In the first program step 4920, the current compression ratio R ₁ of the current session, the session usage counter not the referenced by using counter field 1008, and compressed byte field of the same session usage counter for the current temporary session memory block Calculated using the contents of the compressed byte field.

In step 4930, the current compression ratio R ₂ of the selected memory already dictionary uncompressed byte fields to the stored usage counter referenced by using counter field 1108 of the permanent dictionary store block, and the same the stored usage counter Calculated using compressed byte field. In the next step 4940, both the current session dictionary ratio R ₁ and the selected stored dictionary ratio R ₂ are used to calculate the ratio R. In step 4950, it is checked whether the ratio R is below an arbitrary threshold λ.

  If yes, the program flow emits a signal “stored dictionary is better”, notifies the caller that the stored dictionary is more efficient at step 4960, and then ends at step 4999. If not, the program flow emits the signal “stored dictionary is worse”, notifies the caller that the stored dictionary is less efficient at step 4970 and ends at the next step 4999.

  The effect of the program flow of FIG. 25 was permanently stored for reuse by efficiently evaluating the efficiency of previously stored dictionaries and session dictionaries currently being considered for storage. It is in the continuous improvement of the dictionary efficiency of the dictionary.

  The description of the second embodiment uses the following drawings, that is, FIGS. 4 and 26 to 31. Compression of very small data blocks where the dictionary is not well utilized causes very inefficient compression due to the large token size. Example 2 examines this problem.

  FIG. 4 illustrates the components and the data flow between the components when data is received and encoded at the transmitter gateway, transmitted over the WAN, and received and decoded at the receiver gateway. It is a block chart of the token size optimization operation | movement of an Example.

  FIG. 26 is a sequence diagram showing a sequence of messages and an operation executed during the token shortening operation of the present embodiment.

  FIG. 27 is a program flow diagram showing a program flow of a token shortening operation executed by the transmission side token encoder 1700 for compressing input data such as short packet sensor data into a token of an optimized size.

  FIG. 28 is a program flow diagram showing a program flow of a restricted dictionary token size calculation operation executed as a subflow of the token shortening program flow on the transmitter side of FIG.

  FIG. 29 is a data schema showing the data fields of the dictionary entry ID list used to contain an ordered list of dictionary entry IDs used during the token shortening operation of FIG.

  FIG. 30 is emitted from the transmitter side token encoder 1700 during the token shortening operation of FIG. 27 to synchronize the necessary parameters with the receiver side token decoder 1710 shown in the token shortening operation of the receiver side of FIG. The data schema indicating the data field of the restricted dictionary token.

  FIG. 31 is a program flow showing a program flow of the token shortening operation on the receiver side executed by the receiver side token decoder 1710 in order to decompress the data compressed by the token shortening operation on the transmitter side in FIG. FIG.

  As described above, the second embodiment is a modification of the first embodiment in which a general dictionary coding mechanism is expanded by a token shortening operation so that the released token size is optimized.

  FIG. 4 illustrates the components and data flow of Example 2 when input data is received by the transmitter gateway, encoded, transmitted to the receiver gateway via the WAN, and decoded again. A block diagram is shown.

  FIG. 4 is similar to the functional block diagram of FIG. 2, but shows the functions and components on both the transmitter and receiver sides, not only the restricted dictionary tokens 460a, 460b, but also the virtual session dictionary on the transmitter side. 1025a and 1025b, and 1025c and 1025d on the receiver side are also added. Two network messages 400e, 400f are received by the transmitter session handler 1600c, and each message is processed separately in turn.

  A network message 400e including a header 402d and a payload 404d is received by the transmitter session handler 1600c, which utilizes information from the header 402d and possibly retrieves previously stored data from the permanent dictionary store 1100c. Use to access information in permanent dictionary store 1100c and temporary session memory 1000c to select an existing temporary session or create a new temporary session, then send both data and related control information to token encoder 1700b Then, actual encoding of data occurs in the token encoder 1700b, and information from the temporary session memory 1005d related to the temporary session memory store 1000c is accessed. The encoding first releases the dictionary entry ID to the newly initialized dictionary entry ID list 1040b, and the virtual session dictionary 1025a, which is a continuous sliding window on the actual session dictionary 1020d defined by the specific parameters. Allows creation of.

  These parameters are encoded and released into the restricted dictionary token 460b. Subsequently, the network data payload 404d is encoded into a token using the virtual session dictionary 1025a that emits the token, and is released as a token. Both the restricted dictionary token 406d and the encoded token are then transmitted in the token message 450c as the payload 404f, and the relevant control information accesses the session usage counter 1030d of the relevant temporary session memory 1005d and possibly the relevant session usage. Sent to usage counter manager 1500c to update the counter. The result is then sent back to the session handler 1600c as control information to allow the storage of temporary session memory to be considered.

  Token message 450c is sent via WAN 320b to access information from permanent dictionary store 1100d and temporary session memory 1000d to select an existing temporary session or create a new temporary session, similar to the transmitter side. Receiver-side session handler 1600d. The data and related control information are then transferred to a token decoder 1710b that decodes the payload 454a of the token message 450b while accessing the related information from the temporary session memory 1005c.

  First, the restricted dictionary token 460a is decrypted, allowing the creation of a virtual session dictionary 1025c using the specific parameters and actual session dictionary 1020d that are included, similar to the transmitter-side virtual session dictionary 1025a. . The remainder of payload 454b is then decrypted using virtual session dictionary 1025c. The resulting decoded data is then transmitted in a receiver-side outgoing message 470b, and the relevant control information accesses the relevant temporary session memory 1005c information and similarly considers the storage of the temporary session memory. The control information is sent back to the use counter manager 1500d back to 1600d. Network message 400f is processed similarly.

  FIG. 26 shows the general sequence of events for Example 2 related to the token size optimization mechanism. The IoT device 100d transmits IoT / sensor data 7300a to the IoT gateway 500i on the transmitter side using an arbitrary dictionary encoding mechanism in order to release an appropriate dictionary entry ID to the dictionary entry ID list at 7310a. At 7320a, the transmitter-side IoT gateway 500i calculates the offset and limit token size for use for token encoding, and at 7330a, the limit dictionary using the previously calculated offset and limit token size. Encode the token.

  At 7340a, the IoT gateway 500i on the transmitter side sends the restricted dictionary token encoded with the standard token size to the IoT gateway 500j on the receiver side, the token is decoded, and the offset and restricted token size included are Information is used to update parameters for subsequent decoding. At 7360a, the transmitter-side IoT gateway applies the previously calculated offset and then encodes all entries in the dictionary entry ID list into tokens of the previously calculated token size, and at 7310a, the receiver-side IoT gateway. The encoded token having the limited token size is transmitted to the gateway 500j. At 7420a, the receiver-side IoT gateway 500j uses the previously updated decryption parameters to decrypt the received token, and at 7390a select the pattern referenced by the decrypted entry ID from the session dictionary. In order to transmit the decoded data to the receiver 200d, an arbitrary dictionary decoding mechanism is used.

  Thereafter, at 7400a, the transmitter-side IoT gateway 500i encodes the reset restricted dictionary token, and at 7310a, transmits the token having the restricted token size to the receiver-side IoT gateway 500j, and the receiver-side IoT gateway 500j. Decrypts the token in the same manner as described above, identifies the token as a reserved entry ID, and resets the parameters subsequently used for decryption. Finally, at 7430a, the transmitter-side IoT gateway 500i updates its session usage counter, and at 7440a, the receiver-side IoT gateway 500j updates its session usage counter.

  FIG. 27 shows a program flow 5000 of Example 2 illustrating the transmitter side dictionary encoding flow when using token shortening operations to optimize the emitted token size. The program flow 5000 starts at 5010 and is executed at the token encoder 1700 of the transmitter-side IoT gateway 500a, as described in step 5015, instead of the general dictionary encoding flow. At step 5020, a block of data is received. Generally, this data is in the receive buffer and is ready to be transferred to its next destination or its final destination.

  Subsequent steps 5030 and 5035 begin at step 5025 and are executed in a loop until all data received at 5020 is completely consumed by leaving the loop through the conditional step at 5035. This loop is delimited by a loop final step 5040. Inside the loop, at step 5030, dictionary encoding is performed on the received data block, but instead of releasing the encoded dictionary entry ID token, the dictionary entry ID is added to the dictionary entry ID list. It is done.

In step 5035, it is checked whether all the data received in block 5020 has been completely consumed. If not, the loop continues at step 5040 by jumping to the beginning of the loop at step 5025. Otherwise, program flow continues at step 5045 by finding the smallest entry IDi ₁ in the dictionary entry ID list. After that, the maximum entry IDi _n of dictionary entry ID list is found in step 5050.

In step 5400a, is called restriction dictionary token size calculation subflow of FIG. 28, calculates the limit dictionaries token size using the result of the restricted token is the size t ₁ release. Thereafter, in step 5060, it limits token size _{t 1} is compared to a standard token size t. In step 5065 it is checked whether the restricted token size t ₁ is smaller than the standard token size t. If the result is positive, at step 5070, the offset is calculated by subtracting the maximum entry ID _i from the minimum entry ID _{i 1} and adding the number of reserved entry IDs. In other words, remove all dictionary entries from the dictionary except those reserved entries and those entries delimited by the minimum and maximum indexes, and then the entry ID block delimited by the minimum and maximum indexes When the offset is back-applied (ie, subtracted), the resulting virtual dictionary has a block of entry IDs adjusted for the offset, directly and seamlessly, usually significantly smaller in size than the original dictionary. It contains only the reserved entry ID block that follows.

  In the next step 5075, the restricted dictionary token uses the size of the block of unreserved dictionary entries, calculated by subtracting the maximum index from the minimum index, and the offset calculated in step 5070. Released along with dictionary token size.

In the next step 5080, the token size for encoding is set to the limited token size t ₁ previously calculated in the subflow called in step 5400a. Subsequently, subsequent steps 5090, 5095, and 5100 begin at step 5085, are called in a loop, and iterate over all entries in the dictionary entry ID list filled in step 5030 of the previous loop starting at step 5025. Inside the loop, at step 5090, the offset is subtracted from the current entry ID.

  Then, at step 5095, the offset adjusted entry ID is used to encode a token of the current token size for encoding. Thereafter, in step 5100, it is checked whether the current entry ID is the last item in the dictionary entry ID list. If not, the loop jumps back to the beginning of the loop at step 5085 and continues at step 5105 by selecting the next entry ID as the current entry ID. Otherwise, the loop is terminated by jumping from the outgoing label SE at 5110 to the incoming label SE at 5115 and clearing the dictionary entry ID list at step 5120. Subsequently, the reserved entry ID for resetting the token size is encoded in step 5125. In general, this causes a reset of the token size for encoding from the limit size to the standard size.

  In another embodiment, step 5125 is performed only if a different token size is set in step 5140. In yet another embodiment, the restricted dictionary token is sent with the offset set to zero and the size set to the full dictionary without a reserved entry ID, so when applying offset and size information Make the created virtual dictionary equal to the original dictionary.

In the next step 5130, the size of the data block received in step 5020 is added to the uncompressed byte field of the session usage counter of the current temporary session memory block, and in step 5135 all of the data released in steps 5075 and 5095 are released. The accumulated size of the token is added to the compressed byte field of the session usage counter of the current temporary session memory block. Finally, the flow ends at step 5199. If the test in step 5065 is negative, ie, the restricted dictionary token size t ₁ is larger than the standard token size t, the program flow sets the token size for encoding to the standard token size t of the original dictionary. To continue at step 5140. Program flow continues at step 5085 as described above. In another embodiment, step 5140 is omitted. The effect of the program flow of FIG. 27 is efficient optimization of the token size used when encoding short packet data with dictionary-based compression utilizing pattern locality.

FIG. 28 shows a program flow 5400 of the second embodiment related to a token size optimization operation by calculating a restricted token size. The program flow starts at step 5410 and is executed as a sub-flow by the transmitter-side shortened flow 5000 of FIG. 27 at the token encoder 1700 of the transmitter-side IoT gateway 500a as described in step 5415. Call flow 5000 provides a minimum entry IDi ₁ and the maximum entry IDi _n. In a first step 5420, limit dictionary size is calculated by subtracting the minimum entry IDi ₁ from the maximum entry IDi _n. In the next step 5430, the limit dictionary size is adjusted by adding the size of the reserved entry ID block.

  In the next step 5440, the optimal token size for the limit dictionary is calculated using the logarithm of base 2 of the adjusted limit dictionary size previously calculated. Finally, the flow ends at step 5499. The effect of the program flow of FIG. 28 is that the optimized token size not only allows for the actual application of token size optimization on both the transmitter and receiver sides, but also allows evaluation of token size optimization. Is an efficient calculation.

  FIG. 29 shows a dictionary entry ID list schema that describes a possible data structure for storing the dictionary entry ID list in the primary memory 1830 of FIG. 5, for example. The dictionary entry ID list schema includes a list of dictionary entry IDs that refer to the entry ID of the currently used session dictionary of FIG. 11, as emitted at block 5030 of the program flow of FIG. Multiple dictionary entry ID lists exist simultaneously and are uniquely identified by a list ID field 2710. The dictionary entry ID list is probably stored in the primary memory 1830 of FIG. The entry ID 2710 of the dictionary entry ID list is referred to by the used list ID field 1009 of the temporary session memory schema of FIG. Each entry ID 2720 in the dictionary entry ID list of FIG. 29 is referred to by the session dictionary ID field 1006 of the same temporary session memory schema of FIG. 8, and the entry ID 2320b of FIG. 11 of the session dictionary referenced by the session dictionary ID. Refers to a dictionary entry with

  Each released entry ID in the program flow of FIG. 27 is added to the list of entry IDs 2720. An exemplary dictionary entry ID list having a list ID of “233” of 2710a and “27” of two entry IDs 2720a and “15” of 2720b is shown as a dictionary entry ID list 1040d.

  FIG. 30 shows a restricted dictionary token schema 2800 of Example 2 of this example that illustrates possible encodings for special restricted dictionary tokens related to token size optimization operations. The restricted dictionary token includes an entry ID field 2810a that includes a reserved entry ID reserved for the restricted dictionary token. In addition, the restriction dictionary token 2800 includes an offset field 2820a that contains the offset used for the virtual restriction dictionary, and a token size field 2830a that contains the size of the subsequent tokens whose size has been restricted. An exemplary restricted dictionary token includes an entry ID field 2810b that includes reserved entry ID1, an offset field 2820b that includes offset 1423, and a token size field 2830b that includes token size 11. That is, the virtual dictionary may contain up to 2048 entries, including reserved entry IDs, and any of the decoded non-reserved entry IDs are shifted by 1423 entries that search for referenced patterns in the session dictionary. There must be.

FIG. 31 shows a program flow 5200 of Example 2 that illustrates the receiver side dictionary encoding flow when using a token shortening operation to optimize the emitted token size. Program flow 5200 begins at step 5210, and instead of a general dictionary decoding flow, such as the simplified decoding flow of FIG. 20, the token of the IoT gateway 500b on the receiver side as described in step 5215. It is executed by the encoder 1700. In step 5220, the offset is initialized to zero. Subsequently, a data block is received at step 5225. In general, this means that the receive buffer is prepared with incoming data to be read or used. In the next step 5230, the token size t for decoding, as defined by the current of the temporary session dictionary, is set to a standard token size t _d.

  The loop is then started at step 5235 and all subsequent program flow blocks are called at step 5265 until a check for a loop termination condition. In the loop, the data block previously received in step 5225 is processed until all the data is consumed. The first step inside the loop is 5240 and the token is decrypted using the token size t previously set, eg, in step 5230 described above, or one of steps 5255 or 5305 described below. . Thereafter, it is checked at 5245 whether the token is a restricted dictionary token. If the check is positive, flow continues at step 5250, where the token size field offset of the restricted dictionary token and the restricted token size are read. In the following step 5255, the token size t for decryption is set to the token size read in the previous step 5250. Subsequently, at step 5260, the offset is set to the value from the restricted dictionary token read at step 5250.

Thereafter, the loop termination condition is checked at step 5265 to see if all the data has been completely consumed. If not, the loop continues by jumping from the end of the loop at step 5290 to the beginning of the loop at step 5235, and the program flow thereafter is as described above. If the result of the 5245 check is negative if the decrypted token was a restricted dictionary token, the program flow continues at step 5300 to check if the decrypted token is a special reset restricted dictionary token. The If the result is positive, program flow continues at step 5305 and the token size t setting is the token used to decrypt the token as defined by the currently used temporary session dictionary. to set the size to a standard token size t _d. Program flow continues with the loop end condition check in step 5265 as previously described. If the check at step 5300 is negative if the decrypted token is a restricted dictionary token, then continue at step 5310 by checking if the current offset is greater than zero. If yes, continue at step 5315 by adding the current offset to the token's decoded entry ID number. Thereafter, in step 5330, the temporary dictionary is used to execute standard dictionary decoding, and the decoded pattern is added to the decoded data d and released. This data may be a temporary transmission buffer of the primary memory 1830 of FIG. Program flow continues with the loop end condition check in step 5265 as previously described. If the check at step 5310 is negative, i.e., the current offset is less than or equal to zero, program flow immediately continues at step 5330 as described above.

  If the loop end condition check at step 5265 is positive, the program flow continues by jumping from the outgoing label SD at step 5270 to the incoming label SD at step 5275. At the next step 5280, the size of the data received at step 5225 is added to the compressed byte field of the session usage counter of the current temporary session memory block.

  In the next step 5285, as filled in step 5330, for example, the accumulated size of all emitted patterns, such as the entire size of the transmit buffer, is the uncompressed byte field of the session usage counter of the current temporary session memory. To be added. Finally, at step 5399, the program flow ends.

  The effect of the program flow in FIG. 31 is the ability to decrypt the token optimized in the program flow in FIG. 27 on the receiver side.

The third embodiment is a modification of the above-described embodiment and does not use individual drawings for description.
In the third embodiment, the token size optimization operation described above in the third embodiment is used in the IoT gateway where the dictionary reuse operation in the first embodiment is not used.

  In one embodiment of Example 3, each session begins encoding and decoding a dictionary with an empty dictionary or with a dictionary that has been initialized so that the dictionary encoding and decoding mechanism can execute. In another embodiment of Example 3, a preinitialized dictionary is used, but the dictionary is not stored or updated. Therefore, Example 3 focuses on a dictionary encoding and decoding mechanism that uses only the token size optimization operation described in Example 2.

  As described above, the standard system of this embodiment continuously monitors the compression efficiency of the dictionary in use, evaluates the compression efficiency of the stored dictionary, and possibly updates with the compression efficiency of the dictionary in use. Based on the comparison with the compression efficiency of the dictionary, storage of a new dictionary or update of an existing dictionary is determined.

  Another standard system of this embodiment divides the encoding into a first dictionary entry emission stage and a second token encoding stage, and before the second token encoding stage, the first entry emission. Create a small virtual dictionary based on the original dictionary that contains an estimate of the entries utilized during the stage. This allows the same virtual dictionary to be created from the original dictionary simply by using an offset, temporarily reducing the token size to reflect only the size of the small virtual dictionary, and effectively reducing the offset and reduced token size. Enables communication to allow decoding of tokens encoded with a temporarily reduced token size.

  This embodiment includes an aspect of the present invention relating to a data compression method. In this method, the transmitter gateway sends a token to the receiver gateway based on the session dictionary. This method then further specifies the required part of the session dictionary to compress the transferred data, calculates the size of the restricted token used for token replacement, and compares the restricted token and token size And, if the size of the restricted token is smaller than the size of the token, sending the restricted token to the receiver gateway by the transmitter gateway.

  That is, this network traffic forwarding method uses a dictionary compression mechanism to compress the transmitted data, and compresses the tokens released in relation to the dictionary utilization of the dictionary compression mechanism when compressing a finite block of data. More specifically, this method of using the token compression mechanism incorporates all dictionary entry IDs used to compress a finite block of data into the subset, which uses only the offset and size, and from the dictionary Take advantage of a contiguous subset of all dictionary entry IDs in the dictionary so that the subset contains fewer entries than the dictionary.

  In the above method, the continuous subset is calculated by the dictionary compression mechanism first releasing the dictionary entry ID at the end of the list and then calculating the minimum dictionary entry ID and the maximum dictionary entry ID of the list. The dictionary compression mechanism then calculates the temporary token size by using the length of the continuous subset, and for each dictionary entry ID in the list of dictionary entry IDs released in list order, only the offset and the length of the continuous subset Rather, it emits a token of temporary token size that can be decrypted by a receiver that holds only a copy of the original dictionary.

  According to the above embodiment, the dictionary efficiency of the reused dictionary is continuously improved, and the token size overhead due to large dictionaries that are underutilized when encoding short packets is greatly reduced, and the compression efficiency Greatly improved.

100a: IoT device # 1
150a: IoT device # 1 network data 400a: message # 1
402a: Message # 1 header 404a: Message # 1 payload 100b: IoT device # 2
150b: IoT device # 2 network data 400b: message # 2
402b: Message # 2 header 404b: Message # 2 payload 300a: Transmitter side LAN
500a: IoT gateway 600a on the transmitter side: Function block 320 of the IoT gateway on the transmitter side: WAN
500b: Receiver-side IoT gateway 600b: Receiver-side IoT gateway functional block 300b: Receiver-side LAN
200a: Receiver # 1
200b: Receiver # 2

Claims (16)

A data compression method for a data transmission system with transfer data compression utilizing a session dictionary, comprising:
A first step for preparing a plurality of permanent dictionaries in a permanent dictionary store;
A second step for selecting one of the plurality of permanent dictionaries and duplicating the selected permanent dictionary to prepare the session dictionary in a temporary session memory;
A third step for compressing the transfer data by using the session dictionary;
A fourth step for evaluating the efficiency of the session dictionary;
A fifth step for identifying whether the session dictionary is to be stored in the permanent dictionary store as the permanent dictionary based on the evaluation of the efficiency;
Including
The second step is performed when setting up a session;
The third step is performed during the session, comprising modifying the session dictionary in the temporary session memory;
The fourth step and the fifth step are repeatedly performed during the session;
Data compression method. The fourth step and the fifth step are performed when the session is stopped.
The data compression method according to claim 1. A first data block including the permanent dictionary, a first session ID, and a first usage counter is stored in the permanent dictionary store;
A second data block including the session dictionary, a second session ID, and a second usage counter is stored in the temporary session memory;
When a new session request is received, a third session ID is generated based on the new session request, a permanent dictionary is searched in the permanent dictionary store, and the first session ID is combined with the third session ID. Are the same,
The first step is
When a permanent dictionary having the first session ID that is the same as the third session ID is not found, a new first data block including the third session ID as the first session ID is then created. Including
The second step is
When the permanent dictionary having the same first session ID as the third session ID is found, the second data block including the third session ID as the second session ID is prepared. Duplicating the found permanent dictionary to
The fourth step is
Obtaining an efficiency parameter based on the second usage counter;
The fifth step is
Identifying whether the session dictionary is to be stored in the permanent dictionary store by evaluating the efficiency parameter;
The data compression method according to claim 1. The fourth step and the fifth step are performed continuously during the session each time the second usage counter is changed.
The data compression method according to claim 3. The fourth step and the fifth step are performed continuously during the session whenever the second usage counter is changed and also when the session is stopped.
The data compression method according to claim 3. The second usage counter includes a counter for the number of uncompressed bytes and a counter for the number of compressed bytes;
The fourth step includes
Obtaining the uncompressed bytes and the compressed bytes;
The fifth step comprises
Checking whether the number of uncompressed bytes satisfies a first predetermined condition σ or whether the number of compressed bytes satisfies a first predetermined condition σ;
Including
If the result is negative,
The session dictionary is not stored in the permanent dictionary store;
The data compression method according to claim 3. The second usage counter includes a counter for the number of uncompressed bytes and a counter for the number of compressed bytes;
The fourth step includes
Calculating a compression ratio indicative of a ratio of the compressed bytes to the uncompressed bytes to obtain the efficiency parameter;
The fifth step comprises
Checking whether the compression ratio satisfies a second predetermined condition β;
Including
If the result is positive,
The session dictionary is stored in the permanent dictionary store;
The data compression method according to claim 5. The fourth step includes
Calculating a ratio change between a previously calculated compression ratio and a current compression ratio to obtain the efficiency parameter;
The fifth step comprises
Checking whether the ratio change satisfies a third predetermined condition γ;
Including
If the result is positive,
The session dictionary is stored in the permanent dictionary store;
The data compression method according to claim 7. The fourth step includes
Calculating a ratio change ratio between a previously calculated ratio change and a current ratio change to obtain the efficiency parameter;
The fifth step comprises
Checking whether the degree of the ratio change satisfies a fourth predetermined condition δ,
Including
If the result is positive,
The session dictionary is stored in the permanent dictionary store;
The data compression method according to claim 8. The fifth step comprises
Based on the calculation based on the first usage counter and the second usage counter belonging to the first data block and the second data block having the same first session ID and second session ID. Including comparing the results of different calculations
The data compression method according to claim 3. A data compression method implemented by a system comprising a transmitter gateway, a receiver gateway, and a network between the receiver gateway and the receiver gateway,
The first, second, third, fourth and fifth steps are performed by the transmitter gateway, wherein the transmitter gateway transmits the identifier of the selected permanent dictionary to the receiver gateway;
The receiver gateway is
A sixth step for receiving the identifier of the selected permanent dictionary;
The selected receiver for selecting one of a plurality of receiver permanent dictionaries stored in a receiver permanent dictionary store based on the identifier and preparing a receiver session dictionary in a receiver temporary session memory. A seventh step for replicating the permanent dictionary;
An eighth step for decompressing data received using the receiver session dictionary;
A ninth step for evaluating the efficiency of the receiver session dictionary;
A tenth step for identifying, based on the efficiency, whether the receiver session dictionary is to be stored in the receiver permanent dictionary store as the receiver permanent dictionary;
The data compression method according to claim 1, comprising: In the third step, a token based on the session dictionary is transmitted, and the method further comprises:
Specifying the necessary part of the session dictionary to compress the transfer data;
Calculating the size of the restricted token used in the token replacement;
Comparing the restricted token and the token size;
If the size of the restricted token is smaller than the size of the token, the transmitter gateway sends the restricted token to the receiver gateway;
The data compression method according to claim 11. A gateway for data transfer comprising an interface coupled to a processor, memory, and a network comprising:
During session setup, the processor selects one of a number of compression dictionaries in the permanent store of the memory based on network parameters for the session;
The processor copies the selected compression dictionary to the session store of the memory as a session dictionary;
The processor compresses the session traffic in the session dictionary while updating the session dictionary and a usage counter corresponding to the session dictionary;
The processor makes a decision whether to store the session dictionary in the permanent store based on the usage counter of the session dictionary;
gateway. The usage counter indicates the number of uncompressed and compressed bytes of the traffic;
The processor makes the determination based on at least one of a number of uncompressed bytes and a number of compressed bytes;
The gateway according to claim 13. The processor calculates a compression rate based on the usage counter and makes the determination based on the compression rate;
The gateway according to claim 14. A data transmission system for transferring data from a transmitter device to a receiver device and compressing the data by using a session dictionary,
A transmitter device comprising a processor and a memory, comprising:
The memory includes a first database and a second database, the first database storing a plurality of combinations of a permanent dictionary and efficiency parameters of the permanent dictionary;
The processor selects one permanent dictionary of the first database based on the data and creates a copy of the selected permanent dictionary in the second database as the session dictionary;
The processor compresses the data by using the session dictionary;
The processor modifies the session dictionary and monitors the efficiency parameters of the session dictionary;
The processor determines whether the permanent dictionary should be overwritten by the session dictionary based on the efficiency parameters of the permanent dictionary and the session dictionary;
A data transmission system comprising the transmitter device.

Images