JP7047651B2 - Information processing equipment, distributed processing systems, and distributed processing programs - Google Patents

Description

The present invention relates to an information processing apparatus, a distributed processing system, and a distributed processing program.

Conventionally, there is a system in which an edge computer collects data from an IoT (Internet of Things) device, compresses the metadata of the collected data, and then sends the data to the data center to aggregate the metadata in the data center.

As a prior art, for example, in addition to the information of the type of each attribute recorded in the general RDB (Relational Database), the number of types of attribute values of the attribute is investigated, and each attribute of the general RDB is investigated according to the type and the number of types. There is something to judge whether to apply compression by classifying. Also, for example, a technology that compresses received records for each column and registers them in the database together with column position information, compresses the search key data in the case of a match search, and compares it with the compressed column data to be searched. There is.

Japanese Unexamined Patent Publication No. 11-238073 Japanese Unexamined Patent Publication No. 8-314957

However, in the prior art, the metadata compression efficiency may decrease. For example, the smaller the amount of metadata transmitted by an edge computer at one time, the lower the efficiency of metadata compression tends to be.

In one aspect, the present invention aims to improve the compression efficiency of data.

According to one embodiment, the data having the first item common to all the coded target data is acquired, and the item data of the first item is associated with the code assigned to the item data. With reference to the first dictionary to be represented, the difference between the code assigned to the first item data of the first item included in the acquired data and the code assigned to the other item data of the first item is small. The first item data is assigned a code that does not overlap with the code assigned to the other item data of the first item, and the item data of the second item different from the first item and the item data are the same. The code assigned to the second item data of the second item and the code assigned to the first item data included in the acquired data with reference to the second dictionary corresponding to the code assigned to. A code that does not overlap with the code assigned to the other item data of the second item is assigned to the second item data so that the difference between the data and the second item data is small, and the acquired data is compressed or compressed based on the allocation result. An information processing apparatus, a distributed processing system, and a distributed processing program that synchronize the contents of the first dictionary and the second dictionary with the first apparatus to be decompressed are proposed.

According to one aspect, it is possible to improve the data compression efficiency.

FIG. 1 is an explanatory diagram showing an embodiment of a distributed processing method according to an embodiment. FIG. 2 is an explanatory diagram showing an example of the distributed processing system 200. FIG. 3 is a block diagram showing a hardware configuration example of the end processing device 201. FIG. 4 is an explanatory diagram showing a specific example of the metadata 400. FIG. 5 is an explanatory diagram showing a specific example of the coded metadata 500. FIG. 6 is an explanatory diagram showing an example of the stored contents of the version information 600. FIG. 7 is an explanatory diagram showing an example of the stored contents of the fixed value code table 700. FIG. 8 is an explanatory diagram showing an example of the stored contents of the variable value code table 800. FIG. 9 is an explanatory diagram showing an example of the stored contents of the correspondence table 900. FIG. 10 is an explanatory diagram showing an example of the stored contents of the synchronization waiting code list 1000. FIG. 11 is an explanatory diagram showing an example of the stored contents of the co-occurrence probability list 1100. FIG. 12 is an explanatory diagram showing an example of the stored contents of the cumulative difference list 1200. FIG. 13 is a block diagram showing a hardware configuration example of the central processing unit 203. FIG. 14 is a block diagram showing a functional configuration example of the distributed processing system 200. FIG. 15 is an explanatory diagram (No. 1) showing an operation example of the distributed processing system 200. FIG. 16 is an explanatory diagram (No. 2) showing an operation example of the distributed processing system 200. FIG. 17 is an explanatory diagram (No. 3) showing an operation example of the distributed processing system 200. FIG. 18 is an explanatory diagram (No. 4) showing an operation example of the distributed processing system 200. FIG. 19 is a flowchart showing an example of the registration processing procedure. FIG. 20 is a flowchart showing an example of the first payout processing procedure. FIG. 21 is a flowchart showing an example of the second payout processing procedure. FIG. 22 is a flowchart showing an example of the compression call processing procedure. FIG. 23 is a flowchart showing an example of the compression processing procedure. FIG. 24 is a flowchart showing an example of the synchronous call processing procedure. FIG. 25 is a flowchart showing an example of the synchronization processing procedure. FIG. 26 is a flowchart showing an example of the update processing procedure. FIG. 27 is a flowchart showing an example of the first code adjustment call processing procedure. FIG. 28 is a flowchart showing an example of the first code adjustment processing procedure. FIG. 29 is a flowchart showing an example of the second code adjustment call processing procedure. FIG. 30 is a flowchart showing an example of the second code adjustment processing procedure.

Hereinafter, embodiments of the information processing apparatus, the distributed processing system, and the distributed processing program according to the present invention will be described in detail with reference to the drawings.

(Example of a distributed processing method according to an embodiment)
FIG. 1 is an explanatory diagram showing an embodiment of a distributed processing method according to an embodiment. In FIG. 1, the information processing apparatus 100 is a computer used in a distributed processing system.

Distributed processing systems include, for example, IoT devices, edge computers, and data centers. In a distributed processing system, for example, an edge computer may collect data from an IoT device and send the collected data to a data center to aggregate the data in the data center and make the data available in the data center. be.

Here, not all the data generated by the IoT device is used in the data center, and the data transmitted to the data center at a communication cost may not be used in the data center. Communication costs include power consumption and bandwidth load. Therefore, it is conceivable that the edge computer aggregates the metadata in the data center by transmitting the metadata about the data collected from the IoT device to the data center.

Further, the narrower the communication band between the edge computer and the data center, the more difficult it becomes to exchange a large amount of data between the edge computer and the data center. For example, the communication band is 1.5 Mbps. Therefore, it is preferable to reduce the amount of communication between the edge computer and the data center at one time. Specifically, when the edge computer sends the metadata to the data center, it is preferable to compress the metadata. Specifically, it is preferable that the edge computer reduces the amount of metadata transmitted to the data center at one time.

However, in such a situation, it is difficult to efficiently compress the metadata, and it is difficult to reduce the amount of communication between the edge computer and the data center at one time. For example, when an edge computer uses a compression technique called Deflate, which is based on dictionary coding and Huffman coding, the smaller the amount of metadata transmitted at one time, the lower the efficiency of metadata compression. It tends to end up. In addition, dictionary information for decompression is added to the compressed metadata. Therefore, it is difficult to reduce the amount of communication between the edge computer and the data center at one time.

On the other hand, it is conceivable that the edge computer and the data center periodically synchronize the dictionary information, and the edge computer uses the synchronized dictionary information to compress the metadata. Even in this case, it is difficult to efficiently compress the metadata, and it is difficult to reduce the amount of communication between the edge computer and the data center at one time. For example, the item data is encoded for each item of the metadata, and the integer string compression is performed on the code string in which the codes for the same item are arranged, but when the difference between the adjacent codes of the code strings becomes small, However, the compression efficiency may decrease.

Therefore, in the present embodiment, in the situation where the dictionaries are synchronized between the devices, various items included in the data can be efficiently compressed by one device and transmitted to the other device. A distributed processing method that can assign a code to item data will be described.

In the example of FIG. 1, the information processing apparatus 100 can communicate with the first apparatus 110. The information processing apparatus 100 stores the first dictionary and the second dictionary in the storage unit 101. Similarly, the first device 110 stores the first dictionary and the second dictionary in the storage unit 111. The stored contents of the storage unit 101 and the stored contents of the storage unit 111 are synchronized by the information processing apparatus 100. The first device 110 is a device that compresses or decompresses data.

The first dictionary is correspondence information representing the item data of the first item and the code assigned to the item data in association with each other. The first item is an item that all coded data have in common. The first item is, for example, an item in which the identification information of the device that generates the coded data is set. The sign is, for example, an integer. Further, the second dictionary is correspondence information representing the item data of the second item different from the first item and the code assigned to the item data in association with each other. The second item is, for example, an item in which the sensor information acquired by the device that generates the coded data is set.

(1-1) The information processing apparatus 100 acquires the data 120 having the first item. For example, when the information processing apparatus 100 can communicate with the IoT device, the information processing apparatus 100 receives data 120 having the first item from the IoT device.

(1-2) The information processing apparatus 100 refers to the first dictionary and allocates the first item data 121 of the first item included in the acquired data 120 to the other item data 131 and 141 of the first item. Assign a code that does not overlap with the assigned code. The other item data 131 and 141 are, for example, item data included in the data 130 and 140 acquired in the past, and are item data to which a code has been assigned. At this time, the information processing apparatus 100 assigns a code to the first item data 121 so that the difference between the code assigned to the first item data 121 and the code assigned to the other item data 131 and 141 becomes small. ..

(1-3) The information processing apparatus 100 refers to the second dictionary and allocates the second item data 122 of the second item included in the acquired data 120 to the other item data 132 and 142 of the second item. Assign a code that does not overlap with the assigned code. The other item data 132, 142 are, for example, item data included in the data 130, 140 acquired in the past, and are item data to which a code has been assigned. At this time, the information processing apparatus 100 assigns a code to the second item data 122 so that the difference between the code assigned to the second item data 122 and the code assigned to the first item data 121 becomes small.

(1-4) The information processing apparatus 100 synchronizes the contents of the first dictionary and the second dictionary with the first apparatus 110 based on the allocation result. The information processing apparatus 100 transmits, for example, the allocation result to the first apparatus 110. The first device 110 reflects the allocation result in the storage unit 111, and updates the first dictionary and the second dictionary in the storage unit 111. Further, the information processing apparatus 100 reflects the allocation result in the storage unit 101, and updates the first dictionary and the second dictionary in the storage unit 101.

As a result, when the information processing apparatus 100 encodes a plurality of item data for the first item and performs integer string compression on the code string for the first item, the efficiency of integer string compression can be improved. As such, a code can be assigned to the item data for the first item.

For example, IoT devices tend to generate data at predetermined intervals, and item data for the first item tends to be fixed for each IoT device. In other words, it is considered that data including item data for the first item having the same contents tends to be generated from the IoT device at predetermined intervals.

Therefore, when a code having a small difference is assigned to a plurality of item data for the first item at the same time, when encoding the plurality of item data for the first item at a certain time thereafter, It is considered that it can be replaced with a code having a small difference from each other. As a result, after synchronizing the contents of the first dictionary and the second dictionary with the first device 110, the information processing apparatus 100 replaces the plurality of item data for the first item with codes having a small difference from each other. It is possible to improve the efficiency of integer sequence compression.

Further, when the information processing apparatus 100 encodes a plurality of item data for the second item and performs integer string compression on the code string for the second item, the efficiency of the integer string compression can be improved. Can be assigned a code to the item data for the second item.

For example, when a plurality of item data for the first item is replaced with a code having a small difference from each other, a plurality of item data for the second item to which a code having a small difference from the replaced code is assigned also has a difference from each other. It is thought that it can be replaced with a smaller code. As a result, after synchronizing the contents of the first dictionary and the second dictionary with the first device 110, the information processing apparatus 100 replaces a plurality of item data for the second item with a code having a small difference from each other. It is possible to improve the efficiency of integer sequence compression.

Further, the information processing device 100 can synchronize the stored contents of the storage unit 101 with the stored contents of the storage unit 111, and the first device 110 can compress the data based on the first dictionary and the second dictionary. Can be stretchable. As a result, the information processing apparatus 100 can efficiently compress the data, and the amount of communication with the first apparatus 110 can be reduced.

(Example of distributed processing system 200)
Next, an example of the distributed processing system 200 to which the information processing apparatus 100 shown in FIG. 1 is applied will be described with reference to FIG.

FIG. 2 is an explanatory diagram showing an example of the distributed processing system 200. In FIG. 2, the distributed processing system 200 includes a terminal processing device 201, an IoT device 202, and a central processing unit 203.

In the distributed processing system 200, the IoT device 202 and the terminal processing device 201 are connected by wire or wirelessly. Further, in the distributed processing system 200, the terminal processing device 201 and the central processing unit 203 are connected via a wired or wireless network 210. The network 210 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or the like.

In the example of FIG. 2, the information processing apparatus 100 shown in FIG. 1 is applied to the end processing apparatus 201. The end processing device 201 is a computer that collects data from the IoT device 202. The end processing apparatus 201 has various databases described later in FIGS. 6 to 12. The terminal processing apparatus 201 manages, for example, the fixed value code table 700 described later in FIG. 7 and the variable value code table 800 described later in FIG. 8 by the version information 600 described later in FIG.

The fixed value code table 700 is information indicating a code assigned to the data in the fixed value field of the metadata. A fixed value field is a field that all metadata have in common. The fixed value code table 700 is, for example, correspondence information in which the hash value of the data in the fixed value field of the metadata and the code assigned to the data in the fixed value field of the metadata are associated with each other. Here, the correspondence between the data and the hash value of the data is stored in the correspondence table 900 described later in FIG. Specifically, the fixed value code table 700 corresponds to the first dictionary shown in FIG.

The variable value code table 800 is information indicating a code assigned to the data in the variable value field of the metadata. A variable value field is a field other than a fixed value field. The variation value code table 800 is, for example, correspondence information in which the hash value of the data in the variation value field of the metadata and the code assigned to the data in the variation value field of the metadata are associated with each other. Here, the correspondence between the data and the hash value of the data is stored in the correspondence table 900 described later in FIG. Specifically, the variable value code table 800 corresponds to the second dictionary shown in FIG.

Further, the terminal processing apparatus 201 encodes and stores the metadata about the collected data with reference to the fixed value code table 700 and the variable value code table 800, and compresses a plurality of the encoded metadata together. Then, it is transmitted to the central processing apparatus 203. Specific examples of the metadata will be described later with reference to FIG. Specific examples of the encoded metadata will be described later with reference to FIG.

Further, the end processing apparatus 201 allocates a new code to the data in the fixed value field of the metadata and the data in the variable value field, and stores the data in the synchronization waiting code list 1000 described later in FIG. Further, the terminal processing apparatus 201 may use the co-occurrence probability list 1100 described later in FIG. 11 to reallocate the code to the data in the fixed value field of the metadata and store the code in the synchronization waiting code list 1000. Further, the terminal processing apparatus 201 adjusts the number of codes that can be assigned to the data in the variable value field by using the cumulative difference list 1200 described later in FIG.

Further, the terminal processing device 201 transmits the synchronization waiting code list 1000 to the central processing unit 203, and reflects the synchronization waiting code list 1000 in the fixed value code table 700 and the variable value code table 800. The end processing device 201 is, for example, a server, a PC (Personal Computer), or the like.

The IoT device 202 is a computer that generates data and transmits it to the end processing device 201. The IoT device 202 is provided at a predetermined location, for example, and transmits data indicating the environment around the predetermined location to the terminal processing device 201. The IoT device 202 may be attached to a predetermined object, for example, and may transmit data indicating the state of the predetermined object to the terminal processing device 201. The IoT device 202 may, for example, be possessed by a human and transmit data indicating the human condition to the terminal processing apparatus 201.

The central processing unit 203 is a computer that receives the compressed metadata and decompresses the compressed metadata. Similar to the terminal processing device 201, the central processing unit 203 has a fixed value code table 700, a variable value code table 800, and a correspondence table 900. The central processing unit 203 receives the synchronization waiting code list 1000, and reflects the synchronization waiting code list 1000 in the fixed value code table 700 and the variable value code table 800. The central processing unit 203 is, for example, a server or a PC.

Here, the case where the information processing apparatus 100 shown in FIG. 1 is applied to the terminal processing apparatus 201 has been described, but the present invention is not limited to this. For example, the information processing device 100 shown in FIG. 1 may be applied to the central processing unit 203. In this case, the central processing device 203 assigns a code to the fixed value field or the variable value field of the metadata based on the decompressed metadata to generate the synchronization waiting code list 1000, and the synchronization waiting code list 1000 is the terminal processing device. It will be transmitted to 201.

Further, for example, the information processing device 100 shown in FIG. 1 may be applied to a device different from the terminal processing device 201 and the central processing unit 203. In this case, a device different from the terminal processing device 201 or the central processing unit 203 collects metadata from the terminal processing device 201 or the central processing unit 203. Then, a device different from the terminal processing device 201 and the central processing device 203 assigns a code to the fixed value field or the variable value field of the metadata to generate the synchronization waiting code list 1000. Then, a device different from the terminal processing device 201 and the central processing unit 203 transmits the synchronization waiting code list 1000 to the terminal processing device 201 and the central processing unit 203.

Here, the case where the end treatment device 201 is one device has been described, but the present invention is not limited to this. For example, the end processing device 201 may be a system including a plurality of cooperating devices. Further, the case where the central processing unit 203 is one device has been described, but the present invention is not limited to this. For example, the central processing unit 203 may be a system including a plurality of cooperating devices.

(Hardware configuration example of end processing device 201)
Next, a hardware configuration example of the end processing apparatus 201 will be described with reference to FIG.

FIG. 3 is a block diagram showing a hardware configuration example of the end processing device 201. In FIG. 3, the terminal processing device 201 includes a CPU (Central Processing Unit) 301, a memory 302, a network I / F (Interface) 303, a recording medium I / F 304, and a recording medium 305. Further, each component is connected by a bus 300.

Here, the CPU 301 controls the entire terminal processing device 201. The memory 302 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash ROM, and the like. Specifically, for example, a flash ROM or ROM stores various programs, and RAM is used as a work area of CPU 301. The program stored in the memory 302 is loaded into the CPU 301 to cause the CPU 301 to execute the coded process.

The network I / F 303 is connected to the network 210 through a communication line, and is connected to another computer via the network 210. The network I / F 303 controls an internal interface with the network 210 and controls the input / output of data from another computer. The network I / F 303 is, for example, a modem, a LAN adapter, or the like.

The recording medium I / F 304 controls read / write of data to the recording medium 305 according to the control of the CPU 301. The recording medium I / F 304 is, for example, a disk drive, an SSD (Solid State Drive), a USB (Universal Serial Bus) port, or the like. The recording medium 305 is a non-volatile memory that stores data written under the control of the recording medium I / F 304. The recording medium 305 is, for example, a disk, a semiconductor memory, a USB memory, or the like. The recording medium 305 may be detachable from the end processing device 201.

The end processing device 201 may include, for example, a keyboard, a mouse, a display, a printer, a scanner, a microphone, a speaker, and the like, in addition to the above-mentioned components. Further, the terminal processing device 201 may have a plurality of recording media I / F 304 and recording media 305. Further, the end processing device 201 does not have to have the recording medium I / F 304 or the recording medium 305.

(Specific example of metadata 400)
Next, a specific example of the metadata 400 will be described with reference to FIG. The metadata 400 is stored in, for example, a storage area such as a memory 302 or a recording medium 305 of the terminal processing apparatus 201 shown in FIG.

FIG. 4 is an explanatory diagram showing a specific example of the metadata 400. As shown in FIG. 4, the metadata 400 has fields of source_id, location, and one or more keys. Data that identifies the IoT device 202 that generated the metadata 400 is set in the source_id field. In the location field, data indicating the location of the IoT device 202 that generated the metadata 400 is set. The data acquired by the IoT device 202 that generated the metadata 400 is set in the key field.

The metadata 400 may not have a key field. In the following description, for example, the fields of source_id and location are collectively treated as fixed value fields. Further, for example, a key field is treated as a variable value field.

(Specific example of encoded metadata 500)
Next, a specific example of the coded metadata 500 will be described with reference to FIG. The encoded metadata 500 is stored in, for example, a storage area such as a memory 302 or a recording medium 305 of the terminal processing apparatus 201 shown in FIG.

FIG. 5 is an explanatory diagram showing a specific example of the coded metadata 500. As shown in FIG. 5, the encoded metadata 500 has a field of a fixed value, a variable value, and a character string. The fixed value field is set with a code replaced with the data in the fixed value field. The variable value field is set with a code replaced with the data in the variable value field. The data that was not replaced with the code is set in the field of the character string.

(Stored contents of version information 600)
Next, an example of the stored contents of the version information 600 will be described with reference to FIG. The version information 600 is realized, for example, by a storage area such as a memory 302 or a recording medium 305 of the terminal processing apparatus 201 shown in FIG.

FIG. 6 is an explanatory diagram showing an example of the stored contents of the version information 600. As shown in FIG. 6, the version information 600 has a field of Ver of the fixed value code table 700 and Ver of the variable value code table 800. In the Ver field of the fixed value code table 700, a version number indicating how many times the fixed value code table 700 has been updated is set. In the Ver field of the variable value code table 800, a version number indicating how many times the variable value code table 800 has been updated is set.

(Stored contents of fixed value code table 700)
Next, an example of the stored contents of the fixed value code table 700 will be described with reference to FIG. 7. The fixed value code table 700 is realized, for example, by a storage area such as a memory 302 or a recording medium 305 of the terminal processing apparatus 201 shown in FIG.

FIG. 7 is an explanatory diagram showing an example of the stored contents of the fixed value code table 700. As shown in FIG. 7, the fixed value code table 700 has a field of hash values and codes. In the fixed value code table 700, the fixed value code information is stored as a record by setting information in each field for each hash value of the data in the fixed value field.

The hash value of the data in the fixed value field is set in the hash value field. The code assigned to the data in the fixed value field is set in the code field.

(Stored contents of fluctuation value code table 800)
Next, an example of the storage contents of the variable value code table 800 will be described with reference to FIG. The variable value code table 800 is realized, for example, by a storage area such as a memory 302 or a recording medium 305 of the terminal processing apparatus 201 shown in FIG.

FIG. 8 is an explanatory diagram showing an example of the stored contents of the variable value code table 800. As shown in FIG. 8, the variable value code table 800 has fields for a hash value, a code, and an allocation number. In the variable value code table 800, the variable value code information is stored as a record by setting information in each field for each hash value of the data in the variable value field.

The hash value of the data in the variable value field is set in the hash value field. The code assigned to the data in the variable value field is set in the code field. A plurality of codes may be set in the code field. In the allocation number field, the number of codes that can be assigned to the data in the variable value field is set.

(Stored contents of correspondence table 900)
Next, an example of the stored contents of the correspondence table 900 will be described with reference to FIG. 9. The correspondence table 900 is realized, for example, by a storage area such as a memory 302 or a recording medium 305 of the terminal processing apparatus 201 shown in FIG.

FIG. 9 is an explanatory diagram showing an example of the stored contents of the correspondence table 900. As shown in FIG. 9, the correspondence table 900 has fields for hash values and data. In the correspondence table 900, the correspondence information is stored as a record by setting information in each field for each data.

In the hash value field, a hash value calculated from the data of the fixed value field or the data of the variable value field included in the metadata is set. The data field is set to the data of the fixed value field or the data of the variable value field included in the metadata.

(Stored contents of synchronization waiting code list 1000)
Next, an example of the stored contents of the synchronization waiting code list 1000 will be described with reference to FIG. The synchronization waiting code list 1000 is realized, for example, by a storage area such as a memory 302 or a recording medium 305 of the terminal processing apparatus 201 shown in FIG.

FIG. 10 is an explanatory diagram showing an example of the stored contents of the synchronization waiting code list 1000. As shown in FIG. 10, the synchronization wait code list 1000 has fields for data / hash value, fixed value / variable value, code, and addition / deletion. The synchronization wait code list 1000 stores the synchronization wait code information as a record by setting information in each field for each data or hash value.

A data or hash value is set in the data / hash value field. Data is set in the data / hash value field, for example, when a combination of data and code is to be added to the fixed value code table 700 or the variable value code table 800. In the data / hash value field, for example, a hash value is set when a combination of a hash value and a code is to be deleted from the fixed value code table 700 or the variable value code table 800.

In the fixed value / variable value field, flag information indicating whether the fixed value code table 700 or the variable value code table 800 is to be processed is set. If the flag information is a fixed value, it indicates that the fixed value code table 700 is to be processed, and if the flag information is a variable value, it indicates that the variable value code table 800 is to be processed. In the code field, a code to be added or deleted with respect to the processing target is set. In the add / delete field, add or delete is set as the processing type.

(Memory content of co-occurrence probability list 1100)
Next, an example of the stored contents of the co-occurrence probability list 1100 will be described with reference to FIG. The co-occurrence probability list 1100 is realized, for example, by a storage area such as a memory 302 or a recording medium 305 of the terminal processing device 201 shown in FIG.

FIG. 11 is an explanatory diagram showing an example of the stored contents of the co-occurrence probability list 1100. As shown in FIG. 11, the co-occurrence probability list 1100 has a field of a hash value pair and a co-occurrence probability. In the co-occurrence probability list 1100, the co-occurrence probability information is stored as a record by setting information in each field for each hash value pair.

In the field of the hash value pair, a hash value pair calculated from each data of two data different from each other for the fixed value field is set. In the field of co-occurrence probability, the co-occurrence probability, which is the probability that two different data about the fixed value field are acquired per unit time, is set.

(Stored contents of cumulative difference list 1200)
Next, an example of the stored contents of the cumulative difference list 1200 will be described with reference to FIG. The cumulative difference list 1200 is realized, for example, by a storage area such as a memory 302 or a recording medium 305 of the terminal processing device 201 shown in FIG.

FIG. 12 is an explanatory diagram showing an example of the stored contents of the cumulative difference list 1200. As shown in FIG. 12, the cumulative difference list 1200 has fields for hash values, cumulative differences, and quotas. In the cumulative difference list 1200, the cumulative difference information is stored as a record by setting information in each field for each hash value.

The hash value calculated from the data for the variable value field is set in the hash value field. The cumulative difference field is the result of accumulating the difference between the code replaced with the data for the variable value field contained in the metadata and the code replaced with the data for the fixed value field contained in the same metadata. Set. The assigned number field is set to the number of codes that can be assigned to the data for the variable value field.

(Hardware configuration example of IoT device 202)
The hardware configuration example of the IoT device 202 is the same as the hardware configuration example of the end processing device 201 shown in FIG. 3, but the IoT device 202 does not have a recording medium I / F or a recording medium. good. Further, the IoT device 202 may have, for example, a sensor device.

The sensor device detects, for example, the state of the IoT device 202. The state of the IoT device 202 is, for example, at least one of the position, movement, and orientation of the IoT device 202. Specifically, the sensor device may include any of an acceleration sensor, a geomagnetic sensor, a vibration sensor, a GPS receiver, and the like.

Further, the sensor device may detect the environment around the IoT device 202, for example. The environment is, for example, sound or light. Specifically, the sensor device may include either an optical sensor, a sound sensor, or the like. Further, the sensor device may detect biometric information, for example. The biometric information is, for example, pulse, heart rate, body temperature, or amount of exercise.

(Hardware configuration example of central processing unit 203)
Next, a hardware configuration example of the central processing unit 203 will be described with reference to FIG.

FIG. 13 is a block diagram showing a hardware configuration example of the central processing unit 203. In FIG. 13, the central processing unit 203 includes a CPU (Central Processing Unit) 1301, a memory 1302, a network I / F (Interface) 1303, a recording medium I / F 1304, and a recording medium 1305. Further, each component is connected by a bus 1300.

Here, the CPU 1301 controls the entire control of the central processing unit 203. The memory 1302 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash ROM, and the like. Specifically, for example, a flash ROM or ROM stores various programs, and RAM is used as a work area of CPU 1301. The program stored in the memory 1302 is loaded into the CPU 1301 to cause the CPU 1301 to execute the coded process. The memory 1302 stores a fixed value code table 700, a variable value code table 800, a correspondence table 900, and the like.

The network I / F 1303 is connected to the network 210 through a communication line, and is connected to another computer via the network 210. Then, the network I / F 1303 controls the internal interface with the network 210 and controls the input / output of data from another computer. The network I / F 1303 is, for example, a modem, a LAN adapter, or the like.

The recording medium I / F 1304 controls data read / write to the recording medium 1305 according to the control of the CPU 1301. The recording medium I / F 1304 is, for example, a disk drive, an SSD (Solid State Drive), a USB (Universal Serial Bus) port, or the like. The recording medium 1305 is a non-volatile memory that stores data written under the control of the recording medium I / F 1304. The recording medium 1305 is, for example, a disk, a semiconductor memory, a USB memory, or the like. The recording medium 1305 may be detachable from the central processing unit 203.

The central processing unit 203 may include, for example, a keyboard, a mouse, a display, a printer, a scanner, a microphone, a speaker, and the like, in addition to the above-mentioned components. Further, the central processing unit 203 may have a plurality of recording media I / F 1304 and recording media 1305. Further, the central processing unit 203 does not have to have the recording medium I / F 1304 or the recording medium 1305.

(Example of functional configuration of distributed processing system 200)
Next, a functional configuration example of the distributed processing system 200 will be described with reference to FIG. Specifically, a functional configuration example of the information processing device 100 included in the distributed processing system 200 and a functional configuration example of the first device 110 capable of communicating with the information processing device 100 will be described.

FIG. 14 is a block diagram showing a functional configuration example of the distributed processing system 200. In the example of FIG. 14, a case where the terminal processing device 201 shown in FIG. 2 corresponds to the information processing device 100 and the central processing unit 203 shown in FIG. 2 corresponds to the first device 110 will be described.

In FIG. 14, the information processing apparatus 100 includes a storage unit 1400, an acquisition unit 1401, an allocation unit 1402, a synchronization unit 1403, a compression unit 1404, and an output unit 1405. Further, the first device 110 includes a storage unit 1410, an update unit 1411, and an extension unit 1412.

The storage unit 1400 is realized by, for example, a storage area such as the memory 302 or the recording medium 305 shown in FIG. Hereinafter, the case where the storage unit 1400 is included in the information processing apparatus 100 will be described, but the present invention is not limited to this. For example, the storage unit 1400 may be included in a device different from the information processing device 100, and the stored contents of the storage unit 1400 may be visible from the information processing device 100.

The acquisition unit 1401 to the output unit 1405 function as an example of the control unit. Specifically, the acquisition unit 1401 to the output unit 1405 may cause the CPU 301 to execute a program stored in a storage area such as the memory 302 or the recording medium 305 shown in FIG. 3, or the network I / F 303. To realize the function. The processing result of each functional unit of the information processing apparatus 100 is stored in a storage area such as the memory 302 or the recording medium 305 shown in FIG. 3, for example.

The storage unit 1400 stores various information referred to or updated in the processing of each functional unit. The storage unit 1400 stores, for example, a first dictionary and a second dictionary. The first dictionary is correspondence information representing the item data of the first item and the code assigned to the item data in association with each other. The first item is an item that all coded data have in common. The first item may include a plurality of items. The first item is, for example, a fixed value field. The sign is, for example, an integer. Further, the second dictionary is correspondence information representing the item data of the second item different from the first item and the code assigned to the item data in association with each other. The second item is, for example, a variable value field. Specifically, the storage unit 1400 stores various databases shown in FIGS. 6 to 12.

The acquisition unit 1401 acquires various information used for processing of each functional unit. The acquisition unit 1401 stores various acquired information in the storage unit 1400 or outputs it to each function unit. Further, the acquisition unit 1401 may output various information stored in the storage unit 1400 to each function unit. The acquisition unit 1401 acquires various information based on, for example, the operation input of the user. The acquisition unit 1401 may receive various information from a device different from the information processing device 100, for example.

Specifically, the acquisition unit 1401 acquires data having the first item. The data has one or more items and includes item data of each item of one or more items. The item data may include an item name. The data is, for example, metadata. The data does not have to be metadata. More specifically, the acquisition unit 1401 receives metadata from the IoT device 202. As a result, the acquisition unit 1401 can make the metadata referenceable to each functional unit.

The allocation unit 1402 refers to the first dictionary and assigns a code that does not overlap with the code assigned to the other item data of the first item to the first item data of the first item included in the acquired data. The other item data is, for example, item data included in the data acquired in the past, and is item data to which a code has been assigned. At this time, the allocation unit 1402 allocates a code to the first item data so that the difference between the code assigned to the first item data and the code assigned to the other item data becomes small. To make the difference smaller is, for example, to bring the difference closer to zero. Making the difference smaller may be, for example, making the difference smaller than a certain amount.

For example, the allocation unit 1402 refers to the first dictionary, and if there is no code assigned to the item data of the first item included in the acquired data, the allocation unit 1402 encodes the item data of the first item included in the acquired data. To assign. Further, the allocation unit 1402 refers to, for example, the first dictionary, and if there is a code assigned to the item data of the first item included in the acquired data, the item data of the first item included in the acquired data. It is not necessary to assign a code to.

Specifically, the allocation unit 1402 calculates the hash value of the data of the fixed value field included in the metadata. Next, the allocation unit 1402 searches the fixed value code table 700 using the hash value as a key, and determines whether or not there is a code corresponding to the hash value. Then, if there is no code, the allocation unit 1402 associates the hash value with the new code and stores it in the synchronization waiting code list 1000. Here, the allocation unit 1402 may associate the hash value with a new code and add it to the fixed value code table 700 without storing it in the synchronization waiting code list 1000.

As a result, the allocation unit 1402 encodes a plurality of item data for the first item, and when the integer string compression is performed on the code string for the first item, the efficiency of the integer string compression can be improved. Can be assigned a code to the item data for the first item.

The allocation unit 1402 refers to the second dictionary and assigns a code that does not overlap with the code assigned to the other item data of the second item to the second item data of the second item included in the acquired data. The other item data is, for example, item data included in the data acquired in the past, and is item data to which a code has been assigned. At this time, the allocation unit 1402 assigns a code to the second item data so that the difference between the code assigned to the second item data and the code assigned to the first item data becomes small.

For example, the allocation unit 1402 refers to the second dictionary, and if there is no code assigned to the item data of the second item included in the acquired data, the allocation unit 1402 encodes the item data of the second item included in the acquired data. To assign. Further, the allocation unit 1402 refers to, for example, the second dictionary, and if there is a code assigned to the item data of the second item included in the acquired data, the item data of the second item included in the acquired data. It is not necessary to assign a code to.

Specifically, the allocation unit 1402 calculates the hash value of the data of the variable value field included in the metadata. Next, the allocation unit 1402 searches the variable value code table 800 using the hash value as a key, and determines whether or not there is a code corresponding to the hash value. Then, if there is no code, the allocation unit 1402 associates the hash value with the new code and stores it in the synchronization waiting code list 1000. Here, the allocation unit 1402 may associate the hash value with a new code and add it to the variable value code table 800 without storing it in the synchronization waiting code list 1000.

Further, the allocation unit 1402 refers to, for example, the second dictionary, and the number of codes assigned to the item data of the second item included in the acquired data is the item data of the second item included in the acquired data. It may be determined whether or not it is less than the number of codes that can be assigned to. Then, the allocation unit 1402 allocates a code to the item data of the second item included in the acquired data if the number is less than the number of assignable codes. Further, the allocation unit 1402 does not have to allocate the code to the item data of the second item included in the acquired data as long as it is equal to or more than the number of assignable codes.

Specifically, the allocation unit 1402 calculates the hash value of the data of the variable value field included in the metadata. Next, the allocation unit 1402 searches the variable value code table 800 using the hash value as a key, and determines whether or not there is a code corresponding to the hash value. Then, if there is no code, the allocation unit 1402 associates the hash value with the new code and stores it in the synchronization waiting code list 1000. Here, the allocation unit 1402 may associate the hash value with a new code and add it to the variable value code table 800 without storing it in the synchronization waiting code list 1000.

Further, if there is a code corresponding to the hash value, the allocation unit 1402 determines whether or not the number of codes corresponding to the hash value is less than the number of assignable codes. Then, if the number is less than the number of assignable codes, the allocation unit 1402 associates the hash value with the new code and stores it in the synchronization waiting code list 1000. Here, the allocation unit 1402 may associate the hash value with a new code and add it to the variable value code table 800 without storing it in the synchronization waiting code list 1000.

As a result, the allocation unit 1402 encodes a plurality of item data for the second item, and when the integer string compression is performed on the code string for the second item, the efficiency of the integer string compression can be improved. Can be assigned a code to the item data for the second item.

The allocation unit 1402 calculates the probability that the data including one item data and the data including the other item data will be acquired per unit time among the two different item data for the first item. .. The probability is the co-occurrence probability. Then, the allocation unit 1402 allocates a code to each item data so that the higher the calculated probability is, the smaller the difference between the codes assigned to the respective item data of the two item data is.

Specifically, the allocation unit 1402 calculates the co-occurrence probability for the two types of data based on the number of times that the two types of data appear simultaneously in the fixed value field group in the metadata group for each unit time. Next, the allocation unit 1402 associates the calculated co-occurrence probability with a hash value pair that combines the hash values of the two types of data, and adds the calculated co-occurrence probability to the co-occurrence probability list 1100. Then, the allocation unit 1402 allocates a code to the hash value pair so that the difference of the code corresponding to the hash value pair becomes smaller in descending order of the co-occurrence probability.

The allocation unit 1402 stores the allocation result in the synchronization waiting code list 1000. Here, the allocation unit 1402 may not store the allocation result in the synchronization waiting code list 1000, but may reflect it in the fixed value code table 700. As a result, the allocation unit 1402 can reassign the code to the item data for the first item so that the efficiency of integer sequence compression can be further improved.

Each time the allocation unit 1402 acquires the data having the first item, it is replaced with the code replaced with the item data of the first item included in the acquired data and the item data of the second item included in the acquired data. Calculate the difference from the sign. Then, the allocation unit 1402 increases the number of codes that can be assigned to any of the item data if the magnitude of the total value of the differences calculated for any of the item data of the second item is equal to or more than a certain value.

Specifically, the allocation unit 1402 calculates the difference between the code of the fixed value field and the code of the variable value field included in the encoded metadata each time the metadata is encoded. Next, the allocation unit 1402 updates the cumulative difference associated with the hash value corresponding to the code of the variable value field in the cumulative difference list 1200 based on the calculated difference. The cumulative difference may be normalized based on the number of signs associated with the hash value corresponding to the sign of the variable value field.

Then, if the cumulative difference is a certain value or more, the allocation unit 1402 increases the number of assignable codes associated with the hash values corresponding to the codes of the fluctuation value fields in the variation value code table 800. As a result, the allocation unit 1402 can newly assign a code to the item data for the second item so that the efficiency of integer sequence compression can be further improved.

For example, the allocation unit 1402 says that reducing the difference between the code replaced with the item data of the first item and the code replaced with the item data of the second item contributes to the efficiency of integer sequence compression. Utilizing the nature. Therefore, when the cumulative difference is large, it is considered that the code currently replaced with the item data of the second item is unlikely to contribute to the efficiency of integer sequence compression. On the other hand, the allocation unit 1402 can newly assign a code to the item data for the second item, and can further improve the efficiency of integer sequence compression.

If the magnitude of the total value of the differences calculated for any of the item data of the second item is less than a certain value, the allocation unit 1402 deletes the code assigned to any of the item data from the second dictionary. Specifically, the allocation unit 1402 calculates the difference between the code of the fixed value field and the code of the variable value field included in the encoded metadata each time the metadata is encoded. Next, the allocation unit 1402 updates the cumulative difference associated with the hash value corresponding to the code of the variable value field in the cumulative difference list 1200 based on the calculated difference.

Then, if the cumulative difference is less than a certain value, the allocation unit 1402 deletes the code associated with the hash value corresponding to the code of the variable value field in the variable value code table 800. The allocation unit 1402 saves the deletion result in the synchronization waiting code list 1000. Here, the allocation unit 1402 may not save the deletion result in the synchronization waiting code list 1000, but may reflect it in the variable value code table 800. Further, the allocation unit 1402 reduces the number of assignable codes associated with the hash values corresponding to the codes of the variation value fields in the variation value code table 800. As a result, the allocation unit 1402 can delete the code having a relatively small number of uses and increase the number of available codes.

The synchronization unit 1403 synchronizes the contents of the first dictionary and the second dictionary with the first device 110 based on the allocation result. The synchronization unit 1403 transmits, for example, the allocation result to the first device 110. Further, the synchronization unit 1403 updates the contents of the first dictionary and the second dictionary stored in the storage unit 1400 based on the allocation result. Specifically, the synchronization unit 1403 transmits the synchronization waiting code list 1000 to the central processing unit 203. Further, the synchronization unit 1403 updates the fixed value code table 700 and the variable value code table 800 based on the synchronization waiting code list 1000.

As a result, the synchronization unit 1403 makes the contents of the first dictionary and the second dictionary stored in the first device 110 the same as the contents of the first dictionary and the second dictionary stored in the information processing device 100. The device 110 can be updated. Therefore, the synchronization unit 1403 can make the data compressible or decompressable by the first apparatus 110 based on the first dictionary and the second dictionary. As a result, the synchronization unit 1403 can efficiently compress the data, and can reduce the amount of communication with the first device 110.

The synchronization unit 1403 may accumulate the allocation result according to the assignment of the code to the item data of the first item or the assignment of the code to the item data of the second item. Then, the synchronization unit 1403 synchronizes the contents of the first dictionary and the second dictionary with the first device 110 based on the accumulated allocation result.

The synchronization unit 1403 synchronizes the contents of the first dictionary and the second dictionary with the first device 110, for example, based on the allocation results accumulated until a certain number or more of data are acquired. The synchronization unit 1403 synchronizes the contents of the first dictionary and the second dictionary with the first device 110 based on the allocation result accumulated within a certain period of time. As a result, the synchronization unit 1403 can reduce the amount of communication required for synchronization.

Each time the compression unit 1404 acquires the data having the first item, the item data of the first item included in the acquired data and the item data of the first item are included in the acquired data based on the first dictionary and the second dictionary synchronized with the first apparatus 110. The item data of the second item is replaced with a code, and the data after the replacement is accumulated.

For example, if the compression unit 1404 has a code assigned to the item data of the first item included in the acquired data, the item data of the first item included in the acquired data is replaced with the code. Further, for example, if the compression unit 1404 does not have a code assigned to the item data of the first item included in the acquired data, the compression unit 1404 replaces the item data of the first item included in the acquired data with a predetermined code.

For example, if the compression unit 1404 has a code assigned to the item data of the second item included in the acquired data, the item data of the second item included in the acquired data is replaced with the code. Further, for example, if the compression unit 1404 does not have a code assigned to the item data of the second item included in the acquired data, the compression unit 1404 replaces the item data of the second item included in the acquired data with a predetermined code.

Then, the compression unit 1404 collectively compresses the accumulated data after replacement. The compression unit 1404 compresses, for example, a code string in which the replacement code and the item data of the first item included in the replacement data of the accumulated plurality of replacement data are arranged. Further, the compression unit 1404 compresses, for example, a code string in which the replacement code is arranged with the item data of the second item included in the replacement data of each of the plurality of replacement data accumulated. As a result, the compression unit 1404 can efficiently compress the data.

The output unit 1405 transmits the compression result to the first device 110. Specifically, the output unit 1405 centrally processes the plurality of compressed metadata together with the version number of the fixed value code table 700 used for compression and the version number of the variable value code table 800 used for compression. It is transmitted to the device 203. As a result, the output unit 1405 can reduce the amount of communication with the first device 110.

The storage unit 1410 is realized by, for example, a storage area such as the memory 1302 or the recording medium 1305 shown in FIG. Hereinafter, the case where the storage unit 1410 is included in the first device 110 will be described, but the present invention is not limited to this. For example, the storage unit 1410 may be included in a device different from the first device 110, and the stored contents of the storage unit 1410 may be visible from the first device 110.

Specifically, the update unit 1411 and the extension unit 1412 perform their functions by causing the CPU 1301 to execute a program stored in a storage area such as a memory 1302 or a recording medium 1305, or by using a network I / F 1303. Realize. The processing result of each functional unit of the first device 110 is stored in a storage area such as the memory 1302 or the recording medium 1305 shown in FIG. 13, for example.

The storage unit 1410 stores the first dictionary and the second dictionary in the same manner as the storage unit 1400. The stored contents of the storage unit 1400 and the stored contents of the storage unit 1410 are synchronized with each other. For example, the first dictionary and the second dictionary stored in the storage unit 1400 and the first dictionary and the second dictionary stored in the storage unit 1410 are synchronized so as to have the same contents. Specifically, the storage unit 1410 stores the fixed value code table 700 and the variable value code table 800. Further, the storage unit 1410 may store the version information 600.

The update unit 1411 updates the first dictionary and the second dictionary stored in the storage unit 1410 based on the allocation result of the allocation unit 1402. Specifically, the updating unit 1411 receives the synchronization waiting code list 1000, and updates the fixed value code table 700 and the variable value code table 800 stored in the storage unit 1410 based on the synchronization waiting code list 1000. As a result, the update unit 1411 can synchronize the information processing device 100 and the first device 110 so that they have the first dictionary and the second dictionary having the same contents.

The decompression unit 1412 decompresses the compressed data based on the first dictionary and the second dictionary. Specifically, the decompression unit 1412 receives a plurality of compressed metadata together with the version number of the fixed value code table 700 used for compression and the version number of the variable value code table 800 used for compression. Then, the decompression unit 1412 decompresses a plurality of compressed metadata based on the fixed value code table 700, the variable value code table 800, and the version information 600. As a result, the decompressed portion 1412 can make the compressed data available.

Here, in the distributed processing system 200 shown in FIG. 2, the case where the terminal processing device 201 corresponds to the information processing device 100 and the central processing unit 203 corresponds to the first device 110 has been described, but the present invention is not limited to this. For example, the terminal processing device 201 may correspond to the first device 110, and the central processing unit 203 may correspond to the information processing device 100.

For example, in this case, the acquisition unit 1401 may receive the compressed metadata from the end processing device 201 and decompress the compressed metadata. Further, for example, the synchronization unit 1403 may transmit the synchronization waiting code list 1000 to the terminal processing device 201. Further, for example, the compression unit 1404 and the output unit 1405 of the information processing apparatus 100 may be interchanged with the expansion unit 1412 of the first apparatus 110.

(Operation example of distributed processing system 200)
Next, an operation example of the distributed processing system 200 shown in FIG. 2 will be described with reference to FIGS. 15 to 18.

15 to 18 are explanatory views showing an operation example of the distributed processing system 200. In FIG. 15, the end processing device 201 receives the metadata 1501 from the IoT device 202. Here, in the metadata 1501, the fixed value field is a combination of the source_id field and the location field. Further, the variable value field is a field of hoge and a field of foo, respectively.

The terminal processing apparatus 201 determines whether or not the fixed value code table 700 has a code corresponding to the data in the fixed value field of the metadata 1501, and if there is a code, the data in the fixed value field of the metadata 1501 is used. Encode. If there is no code, the end processing device 201 assigns a code to the data in the fixed value field of the metadata 1501 and stores it in the synchronization waiting code list 1000.

In the example of FIG. 15, the terminal processing apparatus 201 has a hash value “abcddefg” of the data ““ source_id ”:“ 001 ”,“ location ”: {“lat ”: xx,“ ng ”: yy}” in the fixed value field. Is calculated. The terminal processing apparatus 201 determines that the fixed value code table 700 has the code "1" corresponding to the hash value "abcdeffg". Therefore, the terminal processing apparatus 201 replaces the data "" source_id ":" 001 "," location ": {" lat ": xx," lng ": yy}" in the fixed value field with the code "1".

Here, when the terminal processing apparatus 201 assigns a code to the data, the codes assigned to the two different data for the fixed value field may not be continuous and may be separated by at least a certain distance. As a result, the terminal processing apparatus 201 may provide an extra range around the code assigned to the data in the fixed value field, and make it easy to assign a code having a small difference from the code to the data in the variable value field. can.

Further, the end processing apparatus 201 determines whether or not the variable value code table 800 has a code corresponding to the data in the variable value field of the metadata 1501, and if there is a code, the variable value field of the metadata 1501. Encode the data. If there is no code, the end processing device 201 assigns a code to the data in the variable value field of the metadata 1501 and stores it in the synchronization waiting code list 1000. Further, the end processing apparatus 201 assigns a code to the data in the variable value field of the metadata 1501 and waits for synchronization if the number of assigned codes is less than the number of assignable codes even if there is a code. It may be stored in the code list 1000.

In the example of FIG. 15, the terminal processing apparatus 201 calculates the hash value “nopqrst” of the data ““ hoge”: “fuga” “of the first variable value field. Since the terminal processing apparatus 201 has the code "1" corresponding to the hash value "nopqrst" in the variation value code table 800, the data in the first variation value field "" hoge ":" fuga "" is coded ". Replace with "1". The code to be replaced with the data in the fixed value field and the code to be replaced with the data in the variable value field may have the same value.

Here, the terminal processing device 201 does not have to replace the data in the second and subsequent variable value fields with a code. Therefore, the end processing device 201 leaves the data ““foo”: “bar” ”of the second variable value field as it is. Further, the end processing device 201 may also replace the data in the second and subsequent variable value fields with a code. The end processing device 201 stores the encoded metadata 1502 in the transmission waiting buffer. The transmit wait buffer is, for example, a FIFO queue.

Further, the end processing apparatus 201 calculates the difference between the code replaced with the data in the fixed value field and the code replaced with the data in the variable value field, and updates the cumulative difference list 1200. The terminal processing apparatus 201 calculates, for example, the quotient +1 obtained by dividing the code replaced with the data in the variable value field by the maximum number of codes that can be assigned to the data in the variable value field. The maximum number is, for example, 3.

Further, the terminal processing apparatus 201 calculates a quotient obtained by dividing a code replaced with the data in the fixed value field by the maximum number of codes that can be assigned to the data in the variable value field. Then, the terminal processing apparatus 201 calculates a difference obtained by subtracting the quotient calculated for the fixed value field from the quotient +1 calculated for the variable value field, and adds the difference to the cumulative difference in the cumulative difference list 1200.

As a result, the terminal processing apparatus 201 increases the cumulative difference as the contribution to the efficiency of integer sequence compression of the code decreases, and the cumulative difference decreases as the number of times the code is used decreases. Can be memorized. Then, the terminal processing apparatus 201 can make the assignment of the code in the variable value code table 800 adjustable so that the efficiency of integer sequence compression can be improved based on the cumulative difference. Next, the description proceeds to FIG.

In FIG. 16, the terminal processing apparatus 201 collects a plurality of the accumulated encoded metadata when the amount of encoded metadata stored in the transmission waiting buffer becomes a certain amount or more, or when a certain time elapses. And compress.

In the example of FIG. 16, the terminal processing apparatus 201 sorts the encoded metadata in the order of the codes in the fixed value field, and obtains the version number of the fixed value code table 700 and the version number of the variable value code table 800. Fill in as an entry for the given file 1600.

Next, the terminal processing apparatus 201 performs padding in order to match the number of codes for each entry. Specifically, the terminal processing apparatus 201 performs padding on the entry “5, {“piyo ”:“ bas ”}”, and makes the entry “5, 0, {“ piyo ”:“ baz ”}”.

Then, the terminal processing apparatus 201 arranges a code or a character string for each column of the entry and writes it in the file 1601. Specifically, the terminal processing apparatus 201 has "1,3,5,7" in which the codes in the fixed value field are arranged, and "1,3,0,7" in which the codes in the variable value field are arranged. Is entered in file 1601. Further, the terminal processing device 201 arranges the character strings in the character string field as “[{“foo ”:“ bar ”}, {“ hoge ”:“ fuga ”}, {“ piyo ”:“ baz ”}, nil] ”is entered in file 1601. nil indicates that there is no character string.

Then, the terminal processing apparatus 201 performs integer sequence compression in column units for the file 1601. Specifically, the terminal processing apparatus 201 compresses the code string “1,3,5,7” in which the codes in the fixed value field are arranged as an integer sequence, and the code string “1” in which the codes in the variable value field are arranged. , 3, 0, 7 ”is compressed into an integer sequence. Further, the terminal processing device 201 performs compression other than integer sequence compression on the character string group in which the character strings in the character string field are arranged.

Then, the terminal processing device 201 transmits the binary data 1602, which is the compression result, to the central processing unit 203. The central processing unit 203 executes the reverse processing of the terminal processing device 201, and decompresses the file 1600 from the binary data 1602. As a result, the end processing device 201 can efficiently compress the encoded metadata, and the amount of communication with the central processing unit 203 can be reduced.

Here, the end processing device 201 sets the co-occurrence probability list for the hash value pair corresponding to the code pair replaced with the data in the fixed value field based on the encoded metadata stored in the transmission waiting buffer. The co-occurrence probability of 1100 may be updated. Then, the end processing device 201 refers to the co-occurrence probability list 1100 so that the higher the co-occurrence probability of the hash value pair, the smaller the difference in the sign assigned to each hash value of the hash value pair. Reassign the code to each hash value. The end processing apparatus 201 associates the hash value, the code reassigned to the hash value, and the addition as the processing type, and adds the hash value to the synchronization waiting code list 1000.

For example, in the end processing device 201, reducing the difference between the codes in the code string in which the codes of the fixed value fields included in the coded metadata are arranged contributes to the efficiency of integer string compression for the code string. It utilizes the property of. Further, the high co-occurrence probability of the hash value pair indicates that the data pair of the fixed value field corresponding to the hash value pair is likely to be encoded at the same timing.

Therefore, if the pair of data in the fixed value field corresponding to the pair of hash values with high co-occurrence probability is replaced with a code with a small difference, the code string to be compressed by the integer string contains a code with a small difference. It is thought that the possibility of being affected can be improved. On the other hand, the end processing device 201 can further improve the efficiency of integer sequence compression by reassigning a code to each hash value of the hash value pair having a high co-occurrence probability.

Here, the case where the terminal processing apparatus 201 once assigns a code to the data and then reassigns the code to the data based on the co-occurrence probability has been described, but the present invention is not limited to this. For example, the terminal processing apparatus 201 may assign a code to the data based on the co-occurrence probability without assigning the code to the data once. Next, the description shifts to FIG.

In FIG. 17, the terminal processing device 201 transmits a synchronization request for the fixed value code table 700 and the variable value code table 800 to the central processing unit 203 at regular time intervals. The synchronization request is provided with, for example, a synchronization waiting code list 1000 showing how to update the fixed value code table 700 and the variable value code table 800. Further, the synchronization request is given a version number of the fixed value code table 700 and a version number of the variable value code table 800 based on the version information 600.

The central processing unit 203 receives the synchronization request. The central processing unit 203 reflects the synchronization waiting code list 1000 assigned to the synchronization request in the fixed value code table 700 and the variable value code table 800 stored in the own device. The central processing unit 203, for example, adds or adds a correspondence between a hash value and a code to the fixed value code table 700 and the variable value code table 800 in the processing type of the field for adding / deleting the synchronization waiting code list 1000. delete. The central processing unit 203 overwrites the version information 1700 stored in the own device with the version number of the fixed value code table 700 and the version number of the variable value code table 800 assigned to the synchronization waiting code list 1000.

Here, when the central processing unit 203 reflects the synchronization waiting code list 1000 in the fixed value code table 700 stored by the own device, the central processing unit 203 may leave the fixed value code table 700 before the reflection. Similarly, when the central processing unit 203 reflects the synchronization waiting code list 1000 in the variable value code table 800 stored by the own device, the central processing unit 203 may leave the variable value code table 800 before the reflection. The central processing unit 203 reflects the synchronization waiting code list 1000 assigned to the synchronization request in the fixed value code table 700 and the variable value code table 800 stored in the own device, and then sends an update completion response to the terminal processing unit 201. Send.

Upon receiving the update completion response, the terminal processing device 201 reflects the synchronization waiting code list 1000 in the fixed value code table 700 and the variable value code table 800 stored in the own device. Then, the terminal processing apparatus 201 initializes the synchronization waiting code list 1000. Thereby, the terminal processing device 201 can have the fixed value code table 700 and the variable value code table 800 having the same contents as those of the central processing unit 203. Therefore, the terminal processing device 201 can allow the central processing unit 203 to decompress the compressed metadata. Next, the description proceeds to FIG.

In FIG. 18, the end processing apparatus 201 refers to the cumulative difference list 1200 and increases or decreases the number of codes that can be assigned to the data in the variable value field based on the magnitude of the cumulative difference. Here, the cumulative difference indicates that the larger the value, the smaller the contribution to the efficiency of the integer sequence compression of the code, and the smaller the value, the smaller the number of times the code is used.

Therefore, the terminal processing apparatus 201 generates a distribution 1800 of cumulative differences for each hash value. The end processing device 201 increases the number of assignable codes for the data corresponding to the hash value whose cumulative difference exceeds the threshold value 1801. The threshold value 1801 is, for example, 30. Further, the end processing device 201 deletes the code assigned to the data corresponding to the hash value whose cumulative difference is less than the threshold value 1802. The threshold value 1802 is, for example, 3. Then, the end processing device 201 initializes the cumulative difference list 1200.

In the example of FIG. 18, the terminal processing apparatus 201 assigns the number of codes that can be assigned to the data corresponding to the hash value “nopqrst”, which is the cumulative difference “40”, in the variable value code table 800 and the cumulative difference list 1200 to “1”. To "2". Further, the terminal processing apparatus 201 initializes the cumulative difference corresponding to the hash value “nopqrst”, which is the cumulative difference “40”, to “0”. Further, the terminal processing apparatus 201 deletes the record corresponding to the hash value "cdeffhi" which is the cumulative difference "1" from the variable value code table 800 and the cumulative difference list 1200.

For example, the end processing device 201 can reduce the difference between the code in the fixed value field and the code in the variable value field for the same metadata, thereby compressing the integer sequence for the code string in which the codes in the variable value field are arranged. It utilizes the property of contributing to the efficiency of. Therefore, when the cumulative difference is large, it is considered that the code that is currently replaced with the data in the variable value field is unlikely to contribute to the efficiency of integer sequence compression. On the other hand, the terminal processing device 201 can newly assign a code to the data and can further improve the efficiency of integer sequence compression.

Further, it is considered that the larger the number of times the code is used in coding, the more efficiently the dictionary-type compression using the variable value code table 800 can be contributed. In addition, a code that does not contribute to the efficiency of compression assigned to any of the data at present contributes to the efficiency of lexicographic compression or the efficiency of integer sequence compression when assigned to other data. May be possible. On the other hand, the terminal processing device 201 can delete the code assigned to the data and make the code available so that the efficiency of lexicographic compression or the efficiency of integer sequence compression can be improved. be able to.

(Registration process procedure)
Next, an example of the registration processing procedure executed by the terminal processing apparatus 201 will be described with reference to FIG. The registration process is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as a memory 302 or a recording medium 305, and a network I / F 303.

FIG. 19 is a flowchart showing an example of the registration processing procedure. In FIG. 19, the end processing device 201 receives data from the IoT device 202 and stores it (step S1901). Then, the end processing device 201 saves the metadata about the received data in association with the storage location of the received data (step S1902).

Next, the terminal processing apparatus 201 executes the first payout process described later in FIG. 20 and encodes the metadata (step S1903). Then, the terminal processing apparatus 201 executes the second payout process described later in FIG. 21 to encode the metadata (step S1904).

Next, the terminal processing apparatus 201 stores the encoded metadata in the transmission waiting buffer (step S1905). Then, the terminal processing device 201 ends the registration process.

(1st payout processing procedure)
Next, an example of the first payout processing procedure executed by the terminal processing apparatus 201 will be described with reference to FIG. The first payout process is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as a memory 302 or a recording medium 305, and a network I / F 303.

FIG. 20 is a flowchart showing an example of the first payout processing procedure. In FIG. 20, the end processing apparatus 201 calculates the hash value of the entire data of the fixed value field included in the metadata (step S2001).

Next, the terminal processing apparatus 201 refers to the fixed value code table 700 and determines whether or not a code corresponding to the calculated hash value exists (step S2002). Here, when the code corresponding to the hash value does not exist (step S2002: No), the terminal processing apparatus 201 shifts to the processing of step S2004. On the other hand, when the code corresponding to the hash value exists (step S2002: Yes), the terminal processing apparatus 201 shifts to the processing of step S2003.

In step S2003, the end processing apparatus 201 refers to the fixed value code table 700 and replaces the data of the fixed value field included in the metadata with the code corresponding to the calculated hash value (step S2003). Then, the end processing device 201 ends the first payout process.

In step S2004, the end processing apparatus 201 assigns a new code to the data in the fixed value field included in the metadata. The terminal processing apparatus 201 associates the data in the fixed value field with the new code and the addition as the processing type, and adds the data to the synchronization waiting code list 1000 (step S2004).

Next, the end processing device 201 replaces the data of the fixed value field included in the metadata with 0, and moves the data of the fixed value field included in the metadata to the character string field included in the metadata (step). S2005). Then, the end processing device 201 ends the first payout process.

(Second payout processing procedure)
Next, an example of the second payout processing procedure executed by the terminal processing apparatus 201 will be described with reference to FIG. 21. The second payout process is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as a memory 302 or a recording medium 305, and a network I / F 303.

FIG. 21 is a flowchart showing an example of the second payout processing procedure. In FIG. 21, the end processing apparatus 201 calculates the hash value of the entire data of the variable value field included in the metadata (step S2101).

Next, the terminal processing apparatus 201 refers to the variable value code table 800 and determines whether or not a code corresponding to the calculated hash value exists (step S2102). Here, when the code corresponding to the hash value does not exist (step S2102: No), the terminal processing apparatus 201 shifts to the processing of step S2109. On the other hand, when the code corresponding to the hash value exists (step S2102: Yes), the terminal processing apparatus 201 shifts to the processing of step S2103.

In step S2103, the terminal processing apparatus 201 refers to the variable value code table 800, and whether or not the number of codes corresponding to the calculated hash value is less than the number of assignable codes corresponding to the calculated hash value. Is determined (step S2103). Here, if the number is less than the number of assignable codes (step S2103: Yes), the terminal processing apparatus 201 shifts to the processing of step S2106. On the other hand, when the number is equal to or greater than the number of assignable codes (step S2103: No), the terminal processing apparatus 201 shifts to the processing of step S2104.

In step S2104, the end processing apparatus 201 refers to the variation value code table 800 and replaces the data of the variation value field included in the metadata with the code corresponding to the calculated hash value (step S2104). Next, the end processing device 201 calculates the difference between the code in the variable value field and the code in the fixed value field, and updates the cumulative difference corresponding to the calculated hash value in the cumulative difference list 1200 ( Step S2105). Then, the terminal processing apparatus 201 shifts to the processing of step S2112.

In step S2106, the end processing apparatus 201 determines whether or not there is a code that can be assigned to the data in the variable value field in the extra range corresponding to the code in the fixed value field (step S2106). Here, if there is no assignable code (step S2106: No), the terminal processing apparatus 201 shifts to the processing of step S2108. On the other hand, when the assignable code exists (step S2106: Yes), the terminal processing apparatus 201 shifts to the processing of step S2107.

In step S2107, the end processing apparatus 201 assigns a new code to the data in the variable value field included in the metadata. The terminal processing apparatus 201 associates the data in the variable value field with the new code and the addition as the processing type, and adds the data to the synchronization waiting code list 1000 (step S2107). Then, the terminal processing apparatus 201 shifts to the processing of step S2108.

In step S2108, the end processing apparatus 201 replaces the data in the variable value field included in the metadata with a code corresponding to the calculated hash value (step S2108). Then, the terminal processing apparatus 201 shifts to the processing of step S2112.

In step S2109, the end processing apparatus 201 determines whether or not there is a code that can be assigned to the data in the variable value field in the extra range corresponding to the code in the fixed value field (step S2109). Here, if there is no assignable code (step S2109: No), the terminal processing apparatus 201 shifts to the processing of step S2111. On the other hand, when the assignable code exists (step S2109: Yes), the terminal processing apparatus 201 shifts to the processing of step S2110.

In step S2110, the end processing apparatus 201 assigns a new code to the data in the variable value field included in the metadata. The terminal processing apparatus 201 associates the data in the variable value field with the new code and the addition as the processing type, and adds the data to the synchronization waiting code list 1000 (step S2110). Then, the terminal processing apparatus 201 shifts to the processing of step S2111.

In step S2111, the end processing device 201 replaces the data of the variable value field included in the metadata with 0, and moves the data of the variable value field included in the metadata to the character string field included in the metadata (. Step S2111). Then, the terminal processing apparatus 201 shifts to the processing of step S2112.

In step S2112, the end processing apparatus 201 determines whether or not the data of all the variable value fields included in the metadata has been processed (step S2112). Here, if there is data in the unprocessed variable value field (step S2112: No), the terminal processing apparatus 201 returns to the processing in step S2101. On the other hand, when processing the data of all the variable value fields (step S2112: Yes), the terminal processing apparatus 201 ends the second payout process.

(Compression call processing procedure)
Next, an example of the compression call processing procedure executed by the terminal processing apparatus 201 will be described with reference to FIG. 22. The compression call process is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as a memory 302 or a recording medium 305, and a network I / F 303.

FIG. 22 is a flowchart showing an example of the compression call processing procedure. In FIG. 22, the terminal processing apparatus 201 determines whether or not a start instruction issued periodically from the daemon has been received (step S2201). Here, when the start instruction is received (step S2201: Yes), the terminal processing apparatus 201 shifts to the processing of step S2203. On the other hand, when the start instruction is not received (step S2201: No), the terminal processing apparatus 201 shifts to the processing of step S2202.

In step S2202, the terminal processing apparatus 201 determines whether or not the amount of metadata stored in the transmission waiting buffer is equal to or greater than a certain amount (step S2202). Here, when the amount of metadata is less than a certain amount (step S2202: No), the terminal processing apparatus 201 returns to the processing of step S2201. On the other hand, when the amount of metadata is a certain amount or more (step S2202: Yes), the terminal processing apparatus 201 shifts to the processing of step S2203.

In step S2203, the end processing apparatus 201 executes the compression process described later in FIG. 23 (step S2203). Then, the terminal processing device 201 ends the compression call processing.

(Compression processing procedure)
Next, an example of the compression processing procedure executed by the terminal processing apparatus 201 will be described with reference to FIG. 23. The compression process is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as a memory 302 or a recording medium 305, and a network I / F 303.

FIG. 23 is a flowchart showing an example of the compression processing procedure. In FIG. 23, the end processing device 201 indicates the co-occurrence probability for each pair of hash values corresponding to the pair of codes in the fixed value field included in the metadata stored in the transmission waiting buffer in the co-occurrence probability list 1100. Is updated (step S2301).

Next, the terminal processing apparatus 201 sorts the metadata stored in the transmission waiting buffer with reference to the code in the fixed value field (step S2302). Then, the end processing device 201 performs integer sequence compression on the code string in which the codes in the fixed value field included in the sorted metadata are arranged (step S2303).

Next, the terminal processing apparatus 201 calculates n of the metadata stored in the transmission waiting buffer so that the number of metadata including n or more variable value fields exceeds th_v (step S2304). .. Then, the end processing apparatus 201 performs integer sequence compression on the code string in which the codes in each of the variable value fields up to the nth variable value field included in the sorted metadata are arranged (step S2305).

Next, the end processing device 201 performs character string compression on the data in the character string field included in the sorted metadata while maintaining the order (step S2306). Then, the terminal processing device 201 transmits the compressed metadata to the central processing unit 203 (step S2307). Then, the end processing device 201 ends the compression processing.

(Synchronous call processing procedure)
Next, an example of the synchronous call processing procedure executed by the terminal processing apparatus 201 will be described with reference to FIG. 24. The synchronous call process is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as a memory 302 or a recording medium 305, and a network I / F 303.

FIG. 24 is a flowchart showing an example of the synchronous call processing procedure. In FIG. 24, the terminal processing apparatus 201 determines whether or not a start instruction issued periodically from the daemon has been received (step S2401). Here, when the start instruction is not received (step S2401: No), the terminal processing apparatus 201 returns to the processing of step S2401. On the other hand, when the start instruction is received (step S2401: Yes), the terminal processing apparatus 201 shifts to the processing of step S2402.

In step S2402, the end processing apparatus 201 executes the synchronization process described later in FIG. 25 (step S2402). Then, the terminal processing device 201 ends the synchronous call processing.

(Synchronous processing procedure)
Next, an example of a synchronous processing procedure executed by the terminal processing apparatus 201 will be described with reference to FIG. 25. The synchronization process is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as a memory 302 or a recording medium 305, and a network I / F 303.

FIG. 25 is a flowchart showing an example of the synchronization processing procedure. In FIG. 25, the terminal processing device 201 increments the version number and transmits the synchronization waiting code list 1000 associated with the version number to the central processing unit 203 (step S2501).

Next, the terminal processing unit 201 determines whether or not an update completion response has been received from the central processing unit 203 (step S2502). Here, if the update completion response has not been received (step S2502: No), the terminal processing apparatus 201 returns to the processing of step S2502. On the other hand, when the update completion response is received (step S2502: Yes), the terminal processing apparatus 201 shifts to the processing of step S2503.

In step S2503, the terminal processing apparatus 201 reflects the synchronization waiting code list 1000 in the fixed value code table 700 and the variable value code table 800, and then initializes the synchronization waiting code list 1000 (step S2503). Then, the end processing device 201 ends the synchronization processing.

(Update processing procedure)
Next, an example of the update processing procedure executed by the central processing unit 203 will be described with reference to FIG. 26. The update process is realized, for example, by the CPU 1301 shown in FIG. 13, a storage area such as a memory 1302 or a recording medium 1305, and a network I / F 1303.

FIG. 26 is a flowchart showing an example of the update processing procedure. In FIG. 26, the central processing unit 203 receives the synchronization waiting code list 1000 associated with the version number (step S2601).

Next, the central processing unit 203 records the received version number, and reflects the received synchronization waiting code list 1000 in the fixed value code table 700 and the variable value code table 800 corresponding to the end processing device 201 of the transmission source. (Step S2602). Then, the central processing unit 203 transmits the update completion response to the end processing device 201 of the transmission source (step S2603). Then, the central processing unit 203 ends the update process.

(Procedure for processing the first code adjustment call)
Next, an example of the first code adjustment call processing procedure executed by the terminal processing apparatus 201 will be described with reference to FIG. 27. The first code adjustment call process is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as a memory 302 or a recording medium 305, and a network I / F 303.

FIG. 27 is a flowchart showing an example of the first code adjustment call processing procedure. In FIG. 27, the terminal processing apparatus 201 determines whether or not a start instruction issued periodically from the daemon has been received (step S2701). Here, when the start instruction is not received (step S2701: No), the terminal processing apparatus 201 returns to the processing of step S2701. On the other hand, when the start instruction is received (step S2701: Yes), the terminal processing apparatus 201 shifts to the processing of step S2702.

In step S2702, the terminal processing apparatus 201 executes the first code adjustment process described later in FIG. 28 (step S2702). Then, the terminal processing device 201 ends the first code adjustment call processing.

(Procedure for adjusting the first code)
Next, an example of the first code adjustment processing procedure executed by the terminal processing apparatus 201 will be described with reference to FIG. 28. The first code adjustment process is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as a memory 302 or a recording medium 305, and a network I / F 303.

FIG. 28 is a flowchart showing an example of the first code adjustment processing procedure. In FIG. 28, the end processing apparatus 201 refers to the co-occurrence probability list 1100. The end processing device 201 reassigns a code to each hash value so that the higher the co-occurrence probability of the hash value pair, the smaller the difference in the code assigned to each hash value of the hash value pair (step). S2801).

Next, the terminal processing apparatus 201 associates the hash value, the code reassigned to the hash value, and the addition as the processing type, and adds the hash value to the synchronization waiting code list 1000 (step S2802). Then, the end processing apparatus 201 ends the first code adjustment process.

(Procedure for processing the second code adjustment call)
Next, an example of the second code adjustment call processing procedure executed by the terminal processing apparatus 201 will be described with reference to FIG. 29. The second code adjustment call process is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as a memory 302 or a recording medium 305, and a network I / F 303.

FIG. 29 is a flowchart showing an example of the second code adjustment call processing procedure. In FIG. 29, the terminal processing apparatus 201 determines whether or not a start instruction issued periodically from the daemon has been received (step S2901). Here, when the start instruction is not received (step S2901: No), the terminal processing apparatus 201 returns to the processing of step S2901. On the other hand, when the start instruction is received (step S2901: Yes), the terminal processing apparatus 201 shifts to the processing of step S2902.

In step S2902, the end processing apparatus 201 executes the second code adjustment process described later in FIG. 30 (step S2902). Then, the terminal processing device 201 ends the second code adjustment call process.

(Second code adjustment processing procedure)
Next, an example of the second code adjustment processing procedure executed by the terminal processing apparatus 201 will be described with reference to FIG. 30. The second code adjustment process is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as a memory 302 or a recording medium 305, and a network I / F 303.

FIG. 30 is a flowchart showing an example of the second code adjustment processing procedure. In FIG. 30, the end processing apparatus 201 refers to the cumulative difference list 1200 and is normalized by the number of codes assigned to each hash value in the variation code table 800 for each hash value in the variation code table 800. Generate a distribution of the cumulative difference of (step S3001).

Next, the terminal processing apparatus 201 deletes the code assigned to the hash value in which the cumulative difference is equal to or less than a certain value in the variable value code table 800 based on the distribution (step S3002). Then, the terminal processing apparatus 201 associates the hash value whose cumulative difference is equal to or less than a certain value, the deleted code, and the deletion as the processing type, and adds the hash value to the synchronization waiting code list 1000 (step S3003).

Next, the terminal processing apparatus 201 reduces the number of codes that can be assigned to the hash values whose cumulative difference is equal to or less than a certain number (step S3004). Then, the terminal processing apparatus 201 increases the number of codes that can be assigned to the hash values whose cumulative difference is equal to or greater than a certain value in the variable value code table 800 based on the distribution (step S3005).

Next, the terminal processing apparatus 201 initializes the cumulative difference list 1200 (step S3006). Then, the end processing device 201 ends the second code adjustment process.

Here, in each of the flowcharts of FIGS. 19 to 30, the order of processing of some steps may be changed. Further, in each of the flowcharts of FIGS. 19 to 30, the processing of some steps may be omitted.

As described above, according to the information processing apparatus 100, it is possible to acquire the data having the first item common to all the coded data. According to the information processing apparatus 100, the code assigned to the first item data of the first item included in the acquired data and the code assigned to the other item data of the first item by referring to the first dictionary. A code can be assigned to the first item data so that the difference between the two is small. According to the information processing apparatus 100, with reference to the second dictionary, the difference between the code assigned to the second item data of the second item included in the acquired data and the code assigned to the first item data becomes smaller. As described above, a code can be assigned to the second item data. According to the information processing apparatus 100, the contents of the first dictionary and the second dictionary can be synchronized with the first apparatus 110 that compresses or decompresses the acquired data based on the allocation result.

As a result, when the information processing apparatus 100 encodes a plurality of item data for the first item and performs integer string compression on the code string for the first item, the efficiency of integer string compression can be improved. As such, a code can be assigned to the item data for the first item. Further, when the information processing apparatus 100 encodes a plurality of item data for the second item and performs integer string compression on the code string for the second item, the efficiency of the integer string compression can be improved. Can be assigned a code to the item data for the second item. Then, in the information processing apparatus 100, the first apparatus 110 can compress or decompress the data based on the first dictionary and the second dictionary. As a result, the information processing apparatus 100 can efficiently compress the data, and the amount of communication with the first apparatus 110 can be reduced.

According to the information processing apparatus 100, each time the data having the first item is acquired, the first item included in the acquired data is based on the first dictionary and the second dictionary synchronized with the first apparatus 110. The item data and the item data of the second item can be replaced with a code. According to the information processing apparatus 100, the replaced data can be accumulated, and the accumulated plurality of replaced data can be collectively compressed and transmitted to the first apparatus 110. As a result, the information processing apparatus 100 can efficiently compress the data.

According to the information processing apparatus 100, referring to the first dictionary synchronized with the first apparatus 110, if there is a code assigned to the item data of the first item included in the acquired data, the acquired data. The item data of the first item included in the above can be replaced with the reference numeral. According to the information processing apparatus 100, if there is no code assigned to the item data of the first item included in the acquired data, the item data of the first item included in the acquired data can be replaced with a predetermined code. .. As a result, the information processing apparatus 100 encodes a plurality of item data for the first item even if there is no code assigned to the item data for the first item, and is an integer string for the code string for the first item. Compression can be made feasible.

According to the information processing apparatus 100, referring to the second dictionary synchronized with the first apparatus 110, if there is a code assigned to the item data of the second item included in the acquired data, the acquired data. The item data of the second item included in the above can be replaced with the reference numeral. According to the information processing apparatus 100, if there is no code assigned to the item data of the second item included in the acquired data, the item data of the second item included in the acquired data can be replaced with a predetermined code. .. As a result, the information processing apparatus 100 encodes a plurality of item data for the second item even if there is no code assigned to the item data for the second item, and is an integer string for the code string for the second item. Compression can be made feasible.

According to the information processing apparatus 100, it is possible to compress a code string in which the item data of the first item and the replaced code included in the replaced data of the plurality of stored data after replacement are arranged. According to the information processing apparatus 100, it is possible to compress a code string in which the replacement code and the item data of the second item included in the respective replacement data of the accumulated plurality of replacement data are arranged. As a result, the information processing apparatus 100 can efficiently compress the data.

According to the information processing apparatus 100, of the two different item data for the first item, the probability that both the data including one item data and the data including the other item data are acquired per unit time. Can be calculated. According to the information processing apparatus 100, a code can be assigned to each item data so that the higher the calculated probability, the smaller the difference between the codes assigned to the respective item data of the two item data. According to the information processing apparatus 100, the contents of the first dictionary can be synchronized with the first apparatus 110 based on the result of assigning a code to each item data. As a result, the information processing apparatus 100 can reassign the code to the item data for the first item so that the efficiency of integer sequence compression can be further improved.

According to the information processing apparatus 100, if there is no code assigned to the item data of the first item included in the acquired data by referring to the first dictionary, the item data of the first item included in the acquired data A code can be assigned. According to the information processing apparatus 100, referring to the second dictionary, the number of codes assigned to the item data of the second item included in the acquired data is the item data of the second item included in the acquired data. It can be determined whether or not the number of codes can be assigned. According to the information processing apparatus 100, if the number is less than the number of assignable codes, the codes can be assigned to the item data of the second item included in the acquired data. As a result, the information processing apparatus 100 can suppress the number of codes assigned to the same item data.

According to the information processing apparatus 100, each time the data having the first item is acquired, the code replaced with the item data of the first item included in the acquired data and the item data of the second item included in the acquired data. The difference between the code and the replaced code can be calculated. According to the information processing apparatus 100, if the magnitude of the total value of the differences calculated for any of the item data of the second item is equal to or more than a certain value, the number of codes that can be assigned to any of the item data is increased. Can be done. As a result, the information processing apparatus 100 can newly assign a code to the item data for the second item so that the efficiency of integer sequence compression can be further improved.

According to the information processing apparatus 100, if the magnitude of the total value of the differences calculated for any item data of the second item is less than a certain value, the code assigned to any item data from the second dictionary is used. Can be deleted. As a result, the information processing apparatus 100 can delete the codes that are used relatively few times and increase the number of available codes.

According to the information processing apparatus 100, the allocation result can be accumulated according to the assignment of the code to the item data of the first item or the assignment of the code to the item data of the second item. According to the information processing apparatus 100, the contents of the first dictionary and the second dictionary can be synchronized with the first apparatus 110 based on the accumulated allocation result. As a result, the information processing apparatus 100 can reduce the amount of communication required for synchronization.

According to the information processing apparatus 100, the contents of the first dictionary and the second dictionary can be synchronized with the first apparatus 110 based on the allocation result accumulated until a certain number or more of data are acquired. .. As a result, the information processing apparatus 100 can reduce the amount of communication required for synchronization.

According to the information processing apparatus 100, the contents of the first dictionary and the second dictionary can be synchronized with the first apparatus 110 based on the allocation result accumulated within a certain period of time. As a result, the information processing apparatus 100 can reduce the amount of communication required for synchronization.

According to the information processing apparatus 100, the first item can include a plurality of items. As a result, the information processing apparatus 100 can collectively encode the item data of a plurality of items.

According to the information processing apparatus 100, metadata can be used as the data to be acquired. Thereby, the information processing apparatus 100 can be applied to the communication of metadata.

According to the information processing apparatus 100, an integer can be used as a code. As a result, the information processing apparatus 100 can perform integer sequence compression.

The distributed processing system described in this embodiment can be realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. The distributed processing program described in this embodiment is recorded on a computer-readable recording medium such as a hard disk, flexible disk, CD-ROM, MO, or DVD, and is executed by being read from the recording medium by the computer. Further, the distributed processing program described in this embodiment may be distributed via a network such as the Internet.

The following additional notes are further disclosed with respect to the above-described embodiment.

(Appendix 1) Acquire the data having the first item common to all the coded data, and obtain the data.
It is assigned to the first item data of the first item included in the acquired data by referring to the first dictionary representing the item data of the first item and the code assigned to the item data in association with each other. The first item data does not overlap with the code assigned to the other item data of the first item so that the difference between the code and the code assigned to the other item data of the first item becomes small. Assign a sign,
The second item of the second item included in the acquired data with reference to the second dictionary representing the item data of the second item different from the first item and the code assigned to the item data in association with each other. The code assigned to the other item data of the second item overlaps with the code assigned to the other item data of the second item so that the difference between the code assigned to the two item data and the code assigned to the first item data becomes small. Assign a code that does not
Synchronize the contents of the first dictionary and the second dictionary with the first device that compresses or decompresses the acquired data based on the allocation result.
An information processing device characterized by having a control unit.

(Appendix 2) Each time the control unit acquires the data having the first item, the control unit includes the acquired data based on the first dictionary and the second dictionary synchronized with the first apparatus. The item data of the first item and the item data of the second item are replaced with codes, and the data after the replacement is accumulated.
The information processing apparatus according to Appendix 1, wherein a plurality of accumulated data after the replacement are collectively compressed and transmitted to the first apparatus.

(Appendix 3) Each time the control unit acquires the data having the first item, the control unit refers to the first dictionary synchronized with the first apparatus and includes the first data included in the acquired data. If there is a code assigned to the item data of the item, the item data of the first item included in the acquired data is replaced with the code and assigned to the item data of the first item included in the acquired data. The information processing apparatus according to Appendix 2, wherein if there is no designated code, the item data of the first item included in the acquired data is replaced with a predetermined code.

(Appendix 4) Each time the control unit acquires the data having the first item, the control unit refers to the second dictionary synchronized with the first apparatus and includes the second data included in the acquired data. If there is a code assigned to the item data of the item, the item data of the second item included in the acquired data is replaced with the code and assigned to the item data of the second item included in the acquired data. The information processing apparatus according to Appendix 2 or 3, wherein if there is no designated code, the item data of the second item included in the acquired data is replaced with a predetermined code.

(Appendix 5) The control unit compresses a code string in which the code replaced with the item data of the first item included in each of the accumulated data after the replacement is arranged. Addendum 2 to 4, characterized in that a code string in which the code replaced with the item data of the second item included in each of the accumulated data after the replacement is arranged is compressed. The information processing apparatus according to any one of the above.

(Appendix 6) Of the two different item data for the first item, the data including one item data and the data including the other item data are both acquired per unit time. The higher the probability, the smaller the difference between the codes assigned to each item data of the two item data.
Described in any one of Supplementary Provisions 2 to 5, wherein the contents of the first dictionary are synchronized with the first apparatus based on the result of assigning a code to each of the item data. Information processing equipment.

(Appendix 7) Each time the control unit acquires the data having the first item, the reference numeral is assigned to the item data of the first item included in the acquired data by referring to the first dictionary. If not, a code is assigned to the item data of the first item included in the acquired data.
Each time the data having the first item is acquired, the number of codes assigned to the item data of the second item included in the acquired data is included in the acquired data by referring to the second dictionary. If the number is less than the number of codes that can be assigned to the item data of the second item, the code is assigned to the item data of the second item included in the acquired data. The information processing device described in one.

(Appendix 8) Each time the control unit acquires the data having the first item, the code replaced with the item data of the first item included in the acquired data and the first item included in the acquired data. Calculate the difference between the item data of the two items and the replaced code,
If the magnitude of the total value of the differences calculated for any of the item data of the second item is a certain value or more, the number of codes that can be assigned to the item data of any of the above items is increased. The information processing apparatus according to 7.

(Appendix 9) If the magnitude of the total value of the differences calculated for the item data of any of the second items is less than a certain value, the control unit allocates the data to any of the items from the second dictionary. The information processing apparatus according to Appendix 8, wherein the added reference numeral is deleted.

(Appendix 10) The control unit accumulates the allocation result according to the assignment of the code to the item data of the first item or the assignment of the code to the item data of the second item.
The information according to any one of Supplementary note 7 to 9, wherein the contents of the first dictionary and the second dictionary are synchronized with the first apparatus based on the accumulated allocation result. Processing equipment.

(Appendix 11) The control unit synchronizes the contents of the first dictionary and the second dictionary with the first device based on the allocation results accumulated until a certain number or more of data are acquired. , The information processing apparatus according to Appendix 10.

(Appendix 12) The control unit is characterized in that it synchronizes the contents of the first dictionary and the second dictionary with the first device based on the allocation results accumulated within a certain period of time. The information processing apparatus according to Appendix 10 or 11.

(Supplementary Note 13) The information processing apparatus according to any one of Supplementary note 1 to 12, wherein the first item includes a plurality of items.

(Supplementary Note 14) The information processing apparatus according to any one of Supplementary note 1 to 13, wherein the data is metadata.

(Supplementary Note 15) The information processing apparatus according to any one of Supplementary note 1 to 14, wherein the reference numeral is an integer.

(Appendix 16) A distributed processing system including an information processing device and a first device capable of communicating with the information processing device.
The information processing device is
Acquire the data having the first item that all the coded data have in common,
It is assigned to the first item data of the first item included in the acquired data by referring to the first dictionary representing the item data of the first item and the code assigned to the item data in association with each other. The first item data does not overlap with the code assigned to the other item data of the first item so that the difference between the code and the code assigned to the other item data of the first item becomes small. Assign a sign,
The second item of the second item included in the acquired data with reference to the second dictionary representing the item data of the second item different from the first item and the code assigned to the item data in association with each other. The code assigned to the other item data of the second item overlaps with the code assigned to the other item data of the second item so that the difference between the code assigned to the two item data and the code assigned to the first item data becomes small. Assign a code that does not
Based on the allocation result, the contents of the first dictionary and the second dictionary are synchronized with the first device.
The first device is
Stores the contents of the first dictionary and the second dictionary synchronized with the information processing device.
A distributed processing system characterized by that.

(Appendix 17) To the computer
Acquire the data having the first item that all the coded data have in common,
It is assigned to the first item data of the first item included in the acquired data by referring to the first dictionary representing the item data of the first item and the code assigned to the item data in association with each other. The first item data does not overlap with the code assigned to the other item data of the first item so that the difference between the code and the code assigned to the other item data of the first item becomes small. Assign a sign,
The second item of the second item included in the acquired data with reference to the second dictionary representing the item data of the second item different from the first item and the code assigned to the item data in association with each other. The code assigned to the other item data of the second item overlaps with the code assigned to the other item data of the second item so that the difference between the code assigned to the two item data and the code assigned to the first item data becomes small. Assign a code that does not
Synchronize the contents of the first dictionary and the second dictionary with the first device that compresses or decompresses the acquired data based on the allocation result.
A distributed processing program characterized by executing processing.

100 Information processing unit 101, 111, 1400, 1410 Storage unit 110 First device 120, 130, 140 Data 121 First item data 122 Second item data 131, 132, 141, 142 Item data 200 Distributed processing system 201 Terminal processing device 202 IoT device 203 Central processing unit 210 Network 300,1300 Bus 301,1301 CPU
302,1302 Memory 303,1303 Network I / F
304, 1304 Recording medium I / F
305,1305 Recording medium 400,500,1501,1502 Metadata 600 Version information 700 Fixed value code table 800 Variable value code table 900 Correspondence relationship table 1000 Synchronized waiting code list 1100 Co-occurrence probability list 1200 Cumulative difference list 1401 Acquisition unit 1402 Allocation Part 1403 Synchronization part 1404 Compression part 1405 Output part 1411 Update part 1412 Extension part 1600,1601 File 1602 Binary data 1800 Distribution 1801,1802 Threshold

Claims (8)

Acquire the data having the first item that all the coded data have in common,
It is assigned to the first item data of the first item included in the acquired data by referring to the first dictionary representing the item data of the first item and the code assigned to the item data in association with each other. The first item data does not overlap with the code assigned to the other item data of the first item so that the difference between the code and the code assigned to the other item data of the first item becomes small. Assign a sign,
The second item of the second item included in the acquired data with reference to the second dictionary representing the item data of the second item different from the first item and the code assigned to the item data in association with each other. The code assigned to the other item data of the second item overlaps with the code assigned to the other item data of the second item so that the difference between the code assigned to the two item data and the code assigned to the first item data becomes small. Assign a code that does not
Synchronize the contents of the first dictionary and the second dictionary with the first device that compresses or decompresses the acquired data based on the allocation result.
An information processing device characterized by having a control unit. The control unit includes the first data included in the acquired data based on the first dictionary and the second dictionary synchronized with the first apparatus each time the data having the first item is acquired. The item data of the item and the item data of the second item are replaced with a code, and the data after the replacement is accumulated.
The information processing apparatus according to claim 1, wherein a plurality of accumulated data after the replacement are collectively compressed and transmitted to the first apparatus. Of the two different item data for the first item, the control unit has a high probability that both the data including one item data and the data including the other item data are acquired per unit time. The code is assigned to each item data so that the difference between the codes assigned to each item data of the two item data becomes smaller.
The information processing apparatus according to claim 2, wherein the contents of the first dictionary are synchronized with the first apparatus based on the result of assigning a code to each of the item data. Each time the control unit acquires the data having the first item, the control unit refers to the first dictionary and acquires the data if there is no code assigned to the item data of the first item included in the acquired data. A code is assigned to the item data of the first item included in the data.
Each time the data having the first item is acquired, the number of codes assigned to the item data of the second item included in the acquired data is included in the acquired data by referring to the second dictionary. 2. The information processing device described. Each time the control unit acquires the data having the first item, the code replaced with the item data of the first item included in the acquired data and the item of the second item included in the acquired data. Calculate the difference between the data and the replaced code,
A claim characterized in that the number of codes that can be assigned to any of the item data is increased if the magnitude of the total value of the differences calculated for the item data of any of the second items is a certain value or more. Item 4. The information processing apparatus according to item 4. If the magnitude of the total value of the differences calculated for the item data of any of the second items is less than a certain value, the control unit uses the code assigned to the item data of any of the items from the second dictionary. The information processing apparatus according to claim 5, wherein the information processing device is deleted. A distributed processing system including an information processing device and a first device capable of communicating with the information processing device.
The information processing device is
Acquire the data having the first item that all the coded data have in common,
It is assigned to the first item data of the first item included in the acquired data by referring to the first dictionary representing the item data of the first item and the code assigned to the item data in association with each other. The first item data does not overlap with the code assigned to the other item data of the first item so that the difference between the code and the code assigned to the other item data of the first item becomes small. Assign a sign,
The second item of the second item included in the acquired data with reference to the second dictionary representing the item data of the second item different from the first item and the code assigned to the item data in association with each other. The code assigned to the other item data of the second item overlaps with the code assigned to the other item data of the second item so that the difference between the code assigned to the two item data and the code assigned to the first item data becomes small. Assign a code that does not
Based on the allocation result, the contents of the first dictionary and the second dictionary are synchronized with the first device.
The first device is
Stores the contents of the first dictionary and the second dictionary synchronized with the information processing device.
A distributed processing system characterized by that. On the computer
Acquire the data having the first item that all the coded data have in common,
It is assigned to the first item data of the first item included in the acquired data by referring to the first dictionary representing the item data of the first item and the code assigned to the item data in association with each other. The first item data does not overlap with the code assigned to the other item data of the first item so that the difference between the code and the code assigned to the other item data of the first item becomes small. Assign a sign,
The second item of the second item included in the acquired data with reference to the second dictionary representing the item data of the second item different from the first item and the code assigned to the item data in association with each other. The code assigned to the other item data of the second item overlaps with the code assigned to the other item data of the second item so that the difference between the code assigned to the two item data and the code assigned to the first item data becomes small. Assign a code that does not
Synchronize the contents of the first dictionary and the second dictionary with the first device that compresses or decompresses the acquired data based on the allocation result.
A distributed processing program characterized by executing processing.

Images