JP6460961B2 - Data compression collection system and method - Google Patents

Description

  The present invention relates to a data compression and collection system and method for compressing data collected via a network connection device using data compression information associated with the network connection device.

  Conventionally, a method using a compression technique using dictionary data has been proposed as a data compression method. This method encodes data by comparing data transmitted from each data generation source with dictionary data stored in advance by a data compression server, and stores the encoded data in a data management server. (For example, see Non-Patent Document 1).

“Advanced Compression with Oracle Database 11g”, ORACLE (registered trademark), An Oracle White Paper January 2012

  In the above conventional technique, the processing in the data compression server increases in proportion to the number of data of the data generation source. For this reason, if the data transmitted from each data generation source is collectively compressed by one data compression server, the processing load of the data compression server becomes very high. Therefore, in order to reduce the processing load of the data compression server, a plurality of data compression servers are provided, and the data can be distributed by dividing the data in units of sensors and inputting the data to the data compression servers. A system has been proposed.

  However, in the conventional system, it is necessary to store all necessary dictionary data in each data compression server so that any data can be handled. Thus, there is a problem that even the dictionary data cannot be divided.

  The present invention has been made paying attention to the above circumstances, and an object thereof is to provide a technique for reducing the required storage capacity of the database and further reducing the processing load in the data compression apparatus.

  In order to solve the above-described problem, a first aspect of the present invention includes a plurality of network connection devices in which a plurality of data generation sources divided into a plurality of groups are associated with each other, and the plurality of network connection devices. A plurality of data compression devices capable of communicating with each other via a network, a load balancer disposed between the plurality of network connection devices and the plurality of data compression devices, and the plurality of data compression devices, And a data management device capable of communicating via the network. Each of the plurality of network connection devices receives data transmitted from the group of data generation sources associated with the own device, and uses the received information to identify identification information for identifying the network connection device. After the assignment, the data is transmitted to the data compression apparatus. The data management device stores, for each group of the data generation sources, dictionary data necessary for compressing data generated from the data generation sources belonging to the group in a data compression device corresponding to the group. . The load balancer transfers data transmitted from the plurality of network connection devices to a data compression device associated with the transmission source network connection device based on identification information assigned to the data. Each of the plurality of data compression devices receives the data transferred from the load balancer, compares the received data with the stored dictionary data, compresses the overlapped portion, and the data after the compression processing Is transmitted to the data management apparatus and stored.

  According to a second aspect of the present invention, when the dictionary data is set, data before compression processing is repeatedly collected over a preset period from all the data generation sources, and each collected data is scanned. Thus, dictionary data necessary for the compression process is created, and the created dictionary data is divided into groups of the data generation sources and transmitted to the corresponding data compression apparatus for storage.

  According to a third aspect of the present invention, when the dictionary data is set, data before compression processing is collected over a preset period for each group of the data generation sources, and each collected data is scanned. In this way, dictionary data necessary for the compression processing is created, and the dictionary data created for each group is transmitted to and stored in the data compression apparatus corresponding to the group.

  According to the first aspect of the present invention, each data compression apparatus needs to compress only the data transmitted from the corresponding group of data generation sources, thereby reducing the number of data to be compressed. It becomes possible to reduce the processing load of the data compression apparatus. In addition, the data compression apparatus stores only dictionary data necessary for compression processing of data transmitted from the corresponding group of data generation sources. For this reason, the storage capacity required to store the dictionary data of the data compression apparatus is reduced, and the number of dictionary data to be compared when compressing one data is also reduced. The processing load can be further reduced by reducing the amount.

According to the second aspect of the present invention, dictionary data corresponding to all data transmitted from the data generation source is created in the data management device, and the created dictionary data is divided for each group of data generation sources. To the corresponding data compression device.
According to the third aspect of the present invention, the data management device creates dictionary data for each group of data generation sources and sets the dictionary data in the corresponding data compression device.
Therefore, dictionary data required by each data compression device can be automatically created and set in the data compression device.

  According to the present invention, it is possible to provide a data compression collection system and method capable of reducing the required storage capacity of the database and further reducing the processing load in the data compression apparatus.

The figure which shows the whole structure of the data compression collection system which concerns on 1st Embodiment of this invention. The block diagram which shows the function structure of the data compression collection system shown in FIG. The figure which shows the operation | movement sequence of the system shown in FIG.

Embodiments according to the present invention will be described below with reference to the drawings.
[First Embodiment]
(Constitution)
FIG. 1 is a diagram showing an overall configuration of a data compression / collection system according to a first embodiment of the present invention, and S1 to Sn denote sensors as examples of data generation sources. The sensors S1 to Sn are installed or possessed by equipment or people, and examples include an infrared camera, a visible camera, a temperature sensor, an electric energy sensor, a heart rate sensor, a step sensor, a position sensor, a three-dimensional acceleration sensor, and the like. .

  The data compression and collection system according to the first embodiment includes a plurality of gateway terminals (GW terminals) 1A to 1X as network connection devices that receive data transmitted from the sensors S1 to Sn, respectively, and the GW terminal 1A. Data compression servers 3A-3X for compressing data transmitted from 1X, a load balancer 2 disposed between the GW terminals 1A-1X and the data compression servers 3A-3X, and a data management server 4 ing. These GW terminals 1A to 1X, load balancer 2, data compression servers 3A to 3X, and data management server 4 are capable of data communication with each other via a network (not shown).

  The sensors S1 to Sn are grouped, for example, for each region, for each organization, or for each type of sensor, and are associated with the GW terminals 1A to 1X for each group. For example, in FIG. 1, the sensors S1 and S2 are associated with the GW terminal 1A, and the sensors Sn-1 and Sn are associated with the GW terminal 1X. Therefore, each of the GW terminals 1A to 1X receives data only from the sensors S1 to Sn associated with the own terminal.

FIG. 2 is a block diagram showing a functional configuration of the data compression / collection system.
Each of the GW terminals 1 </ b> A to 1 </ b> X has a GW information adding unit 11 as a new function necessary for realizing the first embodiment. When the data transmitted from the associated sensors S1 to Sn is received, the GW information adding unit 11 is identification information (GW-ID) for uniquely identifying the GW terminal with respect to the received data. Is granted.

  When data is transmitted from the GW terminals 1A to 1X, the load balancer 2 uses the data compression server 3A associated in advance with the GW terminals 1A to 1X based on the GW-ID assigned to the data. Process for transferring the data to ˜3X.

  Each of the data compression servers 3 </ b> A to 3 </ b> X includes a dictionary data database (dictionary data DB) 31 and a data compression unit 32. The dictionary data DB 31 stores, for each data compression server 3A to 3X, dictionary data necessary and sufficient for compressing data handled by the data compression servers 3A to 3X. The data compression unit 32 compresses the data transferred from the load balancer 2 using the dictionary data stored in the dictionary data DB 31.

  The data management server 4 includes a dictionary data DB 43 and a dictionary data management unit 44 in addition to the existing collection data database (collection data DB) 41 and the data management unit 42 as functions necessary for realizing the first embodiment. Has a new.

  The collected data DB 41 is used for storing data compressed by the data compression servers 3A to 3X. The data management unit 42 performs processing for storing the compressed data sent from the data compression servers 3A to 3X in the collected data DB 41 in association with, for example, the reception date and time, sensor identification information, and GW-ID.

  The dictionary data DB 43 stores dictionary data corresponding to data of all the sensors S1 to Sn handled by the system of the first embodiment. The dictionary data management unit 44 classifies the dictionary data stored in the dictionary data DB 43 corresponding to the group of the sensors S1 to Sn, and transmits the classified dictionary data to the corresponding data compression servers 3A to 3X. Processing to be stored in the dictionary DB 31 is performed.

(Operation)
Next, the data compression and collection operation in the system configured as described above will be described. FIG. 3 shows the operation sequence.
(1) Updating dictionary data First, in the dictionary data updating phase P1, the data management server 4 compresses the data of all the sensors S1 to Sn handled by the system in step ST11 under the control of the dictionary data management unit 44. Dictionary data necessary for the creation is created, and the created dictionary data is stored in the dictionary data DB 43.

  Then, the created dictionary data is classified corresponding to the group of sensors, and the classified dictionary data is transmitted to the corresponding data compression servers 3A to 3X and stored in the dictionary DB 31.

The dictionary data classification methods include the following methods.
(1) A system operator manually inputs dictionary data necessary for compressing data of all the sensors S1 to Sn handled by the system from a maintenance terminal (not shown) to the data management server 4. The input dictionary data is stored in the dictionary data DB 43 in association with the GW-IDs of the GW terminals 1A to 1X associated with the groups of the sensors S1 to Sn. The dictionary data management unit 44 of the data management server 4 reads the dictionary data stored in the dictionary data DB 43 for each GW-ID, transmits it to the corresponding data compression servers 3A to 3X, and stores them in the dictionary data DB 31. .
Instead of the manual input, dictionary data for each group of the sensors S1 to Sn may be created separately, and the dictionary data may be input to the data management server 4 using a storage medium.

  (2) In the dictionary data management unit 44 of the data management server 4, first, data before compression is repeatedly collected from all the sensors S1 to Sn over a preset period, and each collected data is scanned and compressed. Create dictionary data required for processing. Then, the created dictionary data is stored in the dictionary data DB 43 in a state associated with the GW-IDs of the GW terminals 1A to 1X associated with the groups of the sensors S1 to Sn. Next, the dictionary data stored in the dictionary data DB 43 is read for each GW-ID, transmitted to the corresponding data compression servers 3A to 3X, and stored in the dictionary data DB 31.

  (3) The data management server 4 first collects the data before compression from the sensors S1 to Sn for each of the GW terminals 1A to 1X over a preset period. Then, the collected data is scanned for each of the GW terminals 1A to 1X, and dictionary data necessary for the GW terminals 1A to 1X to perform coding compression is created. Next, the created dictionary data is stored in the dictionary data DB 43 in a state associated with the GW-IDs of the GW terminals 1A to 1X. Then, the dictionary data stored in the dictionary data DB 43 is read for each GW-ID, transmitted to the corresponding data compression servers 3A to 3X, and stored in the dictionary data DB 31.

  Through the above processing, only dictionary data corresponding to the type of data handled by the GW terminals 1A to 1X is stored in the dictionary data DB 31 of the GW terminals 1A to 1X. The dictionary data update process described above is periodically executed.

(2) Data compression and collection When the dictionary data update process is completed, the system can subsequently perform data compression and collection processes.
When the data is received from the sensors S1 to Sn, the GW terminals 1A to 1X are self-regarded with respect to the received data in step ST12 under the control of the GW information adding unit 11 in the data compression and collection phase P2 shown in FIG. GW-ID is assigned.

The following method can be considered as a method for assigning a GW-ID.
(1) The data of the sensors S1 to Sn is managed as a resource tree on the data management server 4, and the tree corresponding to the data handled by the GW terminals 1A to 1X in the resource tree is notified to the GW terminals 1A to 1X. The GW terminals 1A to 1X assign GW-IDs to the data received from the corresponding sensors S1 to Sn with reference to the tree. Note that the resource tree may be created and managed by the amount corresponding to the sensor data handled by the GW terminals 1A to 1X.
(2) A GW-ID attribute is newly added as a data attribute, and is given as an attribute value of the GW-ID.

  Next, the GW terminals 1A to 1X transmit the data provided with the GW-ID to the data compression servers 3A to 3X. When the load balancer 2 receives the data on the data transmission path, the load balancer 2 determines the transfer destination of the data based on the GW-ID assigned to the data. Then, the data with the GW-ID is transferred to the data compression servers 3A to 3X previously associated with the GW terminals 1A to 1X.

  When the data compression servers 3A to 3X receive the data, the data compression servers 3A to 3X determine whether or not the data should be handled by the server based on the GW-ID assigned to the received data. If it should be handled by its own server, the data is compressed in step ST13 under the control of the data compression unit 32. This compression process is performed by compressing the data using the dictionary data stored in the dictionary data DB 31.

  Subsequently, the data compression servers 3 </ b> A to 3 </ b> X transmit the data compressed as described above to the data management server 4. Upon receiving the compressed data, the data management server 4 collects the received compressed data in association with the reception date / time, sensor identification information, and the GW-ID under the control of the data management unit 42. Store in the data DB 41.

(effect)
As described above in detail, in the first embodiment, the sensors S1 to Sn are grouped and associated with the GW terminals 1A to 1X for each group, and the GW information providing unit 11 is provided in the GW terminals 1A to 1X, and the sensor S1 By giving the GW-ID of the own terminal to the data received from ~ Sn, the data is transmitted to the data compression servers 3A to 3X associated with the GW terminals 1A to 1X. On the other hand, the data management server 4 is provided with a dictionary data management unit 44 to create dictionary data corresponding to each group of the sensors and store them in the data compression servers 3A to 3X.

  Therefore, the data compression servers 3A to 3X need only compress the data transmitted from the corresponding sensor group, thereby reducing the number of data to be compressed and reducing the processing load on the data compression servers 3A to 3X. It becomes possible to reduce. In addition, the data compression servers 3A to 3X store only dictionary data necessary for compression processing of data transmitted from the corresponding sensor group. For this reason, the required storage capacity of the dictionary database is reduced, and the number of dictionary data to be compared when compressing one piece of data is reduced. Therefore, the processing load can be further reduced by reducing the amount of calculation required for the compression processing. Can do.

[Other Embodiments]
In the first embodiment, the case where the data compression servers 3A to 3X and the GW terminals 1A to 1X are separately provided has been described as an example. However, the data compression unit may be provided in the GW terminal. With such a configuration, the compressed data is transmitted from the GW terminal to the data management server, and the amount of communication on the network can be reduced.

  Although the case where data as target data of the first embodiment is applied has been described as an example, the present invention can also be applied to a system in which a cloud server collects and manages various management data and statistical data in addition to data.

  In the first embodiment, the GW terminals 1A to 1X assign GW-IDs for identifying the gateway terminals to the data. However, group identification information is determined for each group of sensors S1 to Sn. The group identification information may be given.

  In addition, the type of network connection device, the method of data compression processing performed by the data compression server, the dictionary data setting method for each data compression device, the method of assigning identification information in the network connection device, etc. Various modifications can be made without departing from the scope.

  That is, the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, you may combine suitably the component covering different embodiment.

  S1 to Sn: sensor, 1A to 1X ... GW terminal, 2 ... load balancer, 3A to 3X ... data compression server, 4 ... data management server, 11 ... GW information adding unit, 31 ... dictionary data DB, 32 ... data compression unit 41 ... Collected data DB, 42 ... Data management unit, 43 ... Dictionary data DB, 44 ... Dictionary data management unit.

Claims (2)

A plurality of network connection devices in which a plurality of data generation sources divided into a plurality of groups are associated with each group;
A plurality of data compression devices capable of communicating with the plurality of network connection devices via a network;
A load balancer disposed between the plurality of network connection devices and the plurality of data compression devices;
A data management device capable of communicating with the plurality of data compression devices via the network;
Each of the plurality of network connection devices includes:
Means for receiving data transmitted from the group of data generation sources associated with the device;
Identification information giving means for giving identification information for identifying the network connection device to the received data;
Means for transmitting the data provided with the identification information to the data compression device,
The data management device stores, for each group of the data generation sources, dictionary data necessary for compressing data generated from the data generation sources belonging to the group in a data compression device corresponding to the group. With dictionary data management means,
The load balancer is configured to transfer data transmitted from the plurality of network connection devices to a data compression device associated with the transmission source network connection device based on identification information attached to the data. With
Each of the plurality of data compression devices includes:
Means for receiving data transferred from the load balancer;
Compression processing means for comparing the received data with the stored dictionary data and compressing overlapping portions;
Means for transmitting and storing the data after the compression processing to the data management device. A plurality of data compression units capable of communicating via a network between a plurality of network connection devices in which a plurality of data generation sources divided into a plurality of groups are associated with each group, and the plurality of network connection devices An apparatus, a load balancer disposed between the plurality of network connection devices and the plurality of data compression devices, and a data management device capable of communicating with the plurality of data compression devices via the network A data compression and collection method executed by a system comprising:
For each group of the data generation source, the data management device stores dictionary data necessary for compressing data generated from the data generation source belonging to the group in the data compression device corresponding to the group. Dictionary data setting process,
Each of the plurality of network connection devices receiving data transmitted from the group of data generation sources associated with the device;
A step in which each of the plurality of network connection devices assigns identification information for identifying the network connection device to the received data;
A process in which each of the plurality of network connection devices transmits the data to which the identification information is attached to the data compression device;
A process in which the load balancer transfers data transmitted from the plurality of network connection devices to a data compression device associated with the transmission source network connection device based on the identification information given to the data. When,
Each of the plurality of data compression devices receiving data transferred from the load balancer;
Each of the plurality of data compression devices compressing the overlapping portion by comparing the received data with the stored dictionary data;
Each of the plurality of data compression devices transmits the compressed data to the data management device;
A method for collecting and compressing data, wherein the data management device comprises a step of receiving and storing data after compression processing transmitted from the plurality of data compression devices.