WO2018170773A1

WO2018170773A1 - Internet of things gateway data management method and internet of things gateway

Info

Publication number: WO2018170773A1
Application number: PCT/CN2017/077628
Authority: WO
Inventors: 刘强; 李成功; 黄龑
Original assignee: 深圳中兴力维技术有限公司
Priority date: 2017-03-22
Filing date: 2017-03-22
Publication date: 2018-09-27

Abstract

Disclosed are an Internet of Things gateway data management method and an Internet of Things gateway. The Internet of Things gateway data management method comprises the steps of: parsing data that is transmitted by means of a heterogeneous protocol by a sensing device accessing the Internet of Things gateway to obtain data information about the sensing device; carrying out data fragmentation on the data information according to a pre-set fragmentation rule in the Internet of Things gateway to obtain fragmented data, wherein the data information includes the attribute of the data; packing the fragmented data and converting same into Internet protocol data; and sending the Internet protocol data. The present invention has the effect of improving data transmission and processing efficiency.

Description

IoT gateway data management method and IoT gateway

[0001] The present invention relates to the field of Internet of Things technologies, and in particular, to an Internet of Things gateway data management method and an Internet of Things gateway.

[0002] In the combination of the Internet of Things and big data platforms, the IoT gateway is the transportation center between the Internet of Things and the big data platform. The big data platform connects to many sensing devices through the IoT gateway, resulting in continuous and large amounts of information. The big data platform performs the approach calculation and trend judgment by receiving a large amount of collected data.

[0003] However, the data transmission methods adopted by the existing IoT gateways are not well suited for mass information transfer, and are not conducive to fast storage and data analysis of big data platforms.

technical problem

The main object of the present invention is to provide an Internet of Things gateway data management method and an Internet of Things gateway, which are intended to improve data transmission and processing efficiency.

Problem solution

Technical solution

[0005] In order to achieve the above object, the present invention provides an Internet of Things gateway data management method, and the data management method includes the following steps:

[0006] The sensing device accessing the Internet of Things gateway parses the data transmitted by the heterogeneous protocol to obtain the data information of the sensing device;

[0007] performing data fragmentation on the data information according to a preset fragmentation rule in the Internet of Things gateway to obtain fragmentation data; and the data information includes attributes of the data;

[0008] converting the fragment data into internet protocol data;

[0009] transmitting the internet protocol data.

[0010] Preferably, the step of converting the fragment data into internet protocol data comprises:

[0011] setting a column family of data according to a data attribute of the slice data; [0012] dividing the fragment data according to the column family to obtain column family data corresponding to the fragment data [0013] packetizing the column family data into internet protocol data.

[0014] Preferably, the step of converting the column family data into internet protocol data comprises:

[0015] performing block processing on the block data of the column family data according to a preset rule, and obtaining partition column family data in which the block data has a logical partition;

[0016] packaging the partitioned column family data into internet protocol data.

[0017] Preferably, the step of converting the package into internet protocol data comprises:

[0018] cleaning the data according to a preset cleaning rule, and compressing the cleaned data to obtain compressed data;

[0019] converting the compressed data into internet protocol data according to a preset network rule.

[0020] Preferably, before the step of converting the compressed data into internet protocol data according to a preset network rule, the method further includes:

[0021] The compressed data is backed up and stored.

[0022] The present invention also provides an Internet of Things gateway, where the Internet of Things gateway includes:

[0023] a parsing module, configured to parse data sent by the heterogeneous protocol for the sensing device accessing the Internet of Things gateway to obtain data information of the sensing device;

[0024] a first data processing module, configured to perform data fragmentation on the data information according to a preset fragmentation rule, and obtain score piece data; the data information includes attributes of the data;

[0025] a second data processing module, configured to package the fragment data into Internet Protocol data;

[0026] a data sending module, configured to send the internet protocol data.

[0027] Preferably, the second data processing module is specifically configured to:

[0028] setting a column family of data according to a data attribute of the slice data;

And [0030] obtaining the column family data corresponding to the fragment data according to the column family, and converting the column family data block into internet protocol data.

[0031] Preferably, the step of converting, by the second data processing module, the column family data block into internet protocol data comprises: [0032] performing block processing on the block data of the column family data according to a preset rule, and obtaining partition column family data in which the block data has a logical partition;

[0033] packaging the partitioned column family data into internet protocol data.

[0034] Preferably, the step of converting the packet of the second data processing module into Internet protocol data comprises: [0035] cleaning data according to a preset cleaning rule, and compressing the cleaned data to obtain compressed data;

[0036] converting the compressed data into internet protocol data according to a preset network rule.

[0037] Preferably, the Internet of Things gateway further includes:

And a backup module, configured to perform backup storage of the compressed data before the step of converting, by the second data processing module, the compressed data into Internet protocol data according to a preset network rule.

Advantageous effects of the invention

Beneficial effect

[0039] The IoT gateway data management method and the Internet of Things gateway provided by the present invention convert the real data stream into batch data by performing the fragmentation operation on the data information, which can increase each processing in the subsequent processing process. The amount of data reduces the number of processing, thereby improving the efficiency of data transmission and processing, and facilitating the rapid storage and data analysis of the big data platform that subsequently receives data.

Brief description of the drawing

DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings to be used in the embodiments or the prior art description will be briefly described below, and obviously, in the following description The drawings are only some of the embodiments of the present invention, and those skilled in the art can obtain other drawings according to the structures shown in the drawings without any creative work.

1 is a flowchart of a first embodiment of a method for managing an Internet of Things gateway data according to the present invention;

2 is a flowchart of a second embodiment of a method for managing an Internet of Things gateway data according to the present invention;

3 is a flowchart of a third embodiment of a method for managing an Internet of Things gateway data according to the present invention;

4 is a flowchart of a fourth embodiment of an Internet of Things gateway data management method according to the present invention;

5 is a flowchart of a fifth embodiment of an Internet of Things gateway data management method according to the present invention;

6 is a schematic diagram showing a data hierarchical structure of the data processing method of the Internet of Things gateway shown in FIG. 5; 7 is a schematic diagram of a logical architecture of an existing big data platform;

8 is a structural diagram of an existing IoT gateway system;

9 is a schematic block diagram of an embodiment of an Internet of Things gateway according to the present invention.

[0050] The implementation, functional features, and advantages of the present invention will be further described with reference to the accompanying drawings.

Embodiments of the invention

The specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Referring to FIG. 1, a first embodiment of a network switching method according to the present invention, the method for managing an Internet of Things gateway includes the following steps:

[0053] Step S100: The sensing device accessing the Internet of Things gateway performs data parsing through the heterogeneous protocol to obtain data information of the sensing device. Among them, there are multiple sensing devices connected to the IoT gateway, and usually include multiple types. Each type of device will transmit data to the IoT gateway through a heterogeneous protocol. The data transmitted through the heterogeneous protocol includes at least two parts, one of which is related data obtained by the sensing device during the monitoring process, and the other is the type of protocol used by the sensing device to transmit data. After receiving the data information, the Internet of Things gateway can parse the related data according to the protocol type in the data information.

[0054] Step S101, performing data fragmentation on the data information according to a preset fragmentation rule in the IoT gateway, and obtaining fragment data; the data information includes attributes of the data, for example: a sensing device The type of protocol used and other attributes. For the specific fragmentation method, the fixed inter-frame interval may be a slice unit, or the size of the buffer data may be a slice unit.

[0055] Step S102: Packetizing the fragment data into Internet Protocol data. The Internet Protocol data is used to interconnect the front-end aware device with the network.

[0056] Step S103, sending the internet protocol data.

[0057] In this embodiment, by performing a fragmentation operation on the data information in the Internet of Things gateway, the real data stream is converted into batch data, which can increase the amount of data processed in each subsequent processing process, and reduce the number of processing times. Therefore, the effect of improving data transmission and processing efficiency is achieved, which facilitates rapid storage and data analysis of a big data platform that subsequently receives data. Referring to FIG. 2, a second embodiment of the network switching method of the present invention is based on the first embodiment.

Step S101 is described in detail. details as follows:

[0059] Step S200 is the same as step S100 in the first embodiment, and details are not described herein again.

[0060] Step S201 is the same as step S101 in the first embodiment, and details are not described herein again.

[0061] Step S202: Set a column family of data according to the data attribute of the fragment data.

[0062] Step S203, the fragment data is divided into blocks according to the column family, and column family data corresponding to the fragment data is obtained. Referring to FIG. 6, for example, data segmentation is performed according to a fixed inter-turn interval 30s; data is included in each slice data: temperature, humidity, current, voltage, position, and device ID; then this embodiment will be established The column family of data including temperature, humidity, current, voltage, position, and device ID; further put together the data of the same column family of the same piece of data, that is, the columnar storage logic of the sliced data is realized.

[0063] Step S204: The column family data is packaged and converted into Internet Protocol data.

[0064] Step S205 is the same as step S103 in the first embodiment, and details are not described herein again.

[0065] In this embodiment, by dividing the fragment data into columns according to the column family, the effect that the data of the same attribute is placed together can be obtained. Further, because of the similarity between the same attribute data, the same data attribute is more conducive to improving the data compression capability, thereby reducing the network transmission bandwidth. At the same time, it is also convenient to do column storage, data processing, etc. in the back-end big data platform.

Referring to FIG. 3, a third embodiment of the network switching method of the present invention is based on the second embodiment.

Step S204 is described in detail. details as follows:

[0067] Step S300 is the same as step S200 in the second embodiment, and details are not described herein again.

[0068] Step S301 is the same as step S201 in the second embodiment, and details are not described herein again.

[0069] Step S302 is the same as step S202 in the second embodiment, and details are not described herein again.

[0070] Step S303 is the same as step S203 in the second embodiment, and details are not described herein again.

[0071] Step S304, the block data of the column family data is partitioned according to a preset rule, and the partition column family data whose block data has a logical partition is obtained. Among them, the method of partitioning the column family data can hash the column family data by using hash Hash or user designation. Please refer to Figure 6. In this example, the storage mode of the voltage column family is exemplified, and the voltage column family is divided into N partitions.

[0072] Step S305, packetizing the partition column family data into Internet Protocol data. [0073] Step S306 is the same as step S205 in the second embodiment, and details are not described herein again.

[0074] In this embodiment, by performing hash processing on the block data, the block data can be hashed into different partitions, so that the block data can be distributed and stored, and the data tilt problem can be solved, and the data reading capability can be improved. In the big data platform, distributed storage is the basis of distributed computing, and the logical architecture diagram of the platform is shown in Figure 7. If the column family data is not partitioned, it will be concentrated on a certain disk in the storage process, which will form a read/write bottleneck during the calculation process, which will affect the distributed computing performance. By partitioning the column family data, the column group data can be allocated to multiple disks in a partitioned manner, and multiple disk 1/0 can be used to improve data reading capability, which is beneficial to the back-end big data platform to perform distributed computing on data. .

Referring to FIG. 4, a fourth embodiment of the network switching method of the present invention is provided. The present embodiment is based on the third embodiment, and step S305 is described in detail. details as follows:

[0076] Step S400 is the same as step S300 in the third embodiment, and details are not described herein again.

[0077] Step S401 is the same as step S301 in the third embodiment:

[0078] Step S402 is the same as step S302 in the third embodiment:

[0079] Step S403 is the same as step S303 in the third embodiment:

[0080] Step S404 is the same as step S304 in the third embodiment:

[0081] Step S405: Clean the partition column family data according to a preset cleanup rule, and compress the cleaned data to obtain compressed data. The cleaning rule may be to perform data cleaning on the duplicate and abnormal data under the same device in the partition information; the compression method may be compressed by a compression algorithm such as GZIP, LZO, and Snappy. It can also be encrypted in compression or encrypted after compression, such as MD5, RSA and other encryption methods.

[0082] Step S406: Convert the compressed data into Internet Protocol data according to a preset network rule.

[0083] Step S407 is the same as step S306 in the third embodiment, and details are not described herein again.

[0084] In this embodiment, by cleaning and re-compressing data, it is advantageous to maximize compression and reduce network transmission bandwidth.

Referring to FIG. 5, a fifth embodiment of the network switching method of the present invention is based on the fourth embodiment, and a new step is added. details as follows:

[0086] Step S500 is the same as step S400 in the third embodiment, and details are not described herein again.

[0087] Step S501 is the same as step S401 in the third embodiment, and details are not described herein again. [0088] Step S502 is the same as step S402 in the third embodiment, and details are not described herein again.

[0089] Step S503 is the same as step S403 in the third embodiment, and details are not described herein again.

[0090] Step S504 is the same as step S404 in the third embodiment, and details are not described herein again.

[0091] Step S505 is the same as step S405 in the third embodiment, and details are not described herein again.

[0092] Step S506: The compressed data is backed up and stored.

[0093] Step S507 is the same as step S406 in the third embodiment, and details are not described herein again.

[0094] Step S508 is the same as step S407 in the third embodiment, and details are not described herein again.

[0095] In this embodiment, the compressed data is backed up, so that data retransmission, search, and the like are performed in the event of a network failure. The maximum backup period of data can be calculated and confirmed according to the amount of data processed by the IoT gateway every day and the capacity of the storage module. Users can set it themselves, but it cannot exceed the maximum backup period. The recommended backup period is 1 to 3 months. .

The present invention also provides an Internet of Things gateway. Referring to FIG. 8, in the IoT gateway application example, the sensing device 1, the sensing device 2, the sensing device 3, and the sensing device 4 of the IoT gateway are accessed. Each sensing device is used to detect a battery pack that is used to form a battery matrix. The Internet of Things Gateway connects to the network, with clients and big data platforms connected to the other ends of the network.

[0097] Referring to FIG. 9, the Internet of Things gateway includes:

[0098] The parsing module 1000 is configured to obtain, by the sensing device accessing the Internet of Things gateway, the data sent by the heterogeneous protocol, and obtain the data information of the sensing device; wherein, the sensing device accessing the Internet of Things gateway has Multiple, and usually include multiple types. Each type of device will transmit data to the IoT gateway through heterogeneous protocols. The data transmitted through the heterogeneous protocol includes at least two parts, one of which is related data obtained by the sensing device during the monitoring process, and the other of which is the type of protocol used by the sensing device to transmit data. After receiving the data information, the Internet of Things gateway can parse the related data according to the protocol type in the data information.

[0099] The first data processing module 1100 is configured to perform data fragmentation on the data information according to a preset fragmentation rule to obtain fragmentation data; the data information includes attributes of the data, for example: used by the sensing device Protocol type and other attributes. For the specific fragmentation method, the fixed inter-frame interval may be a slice unit, or the size of the cache data may be a slice unit.

[0100] The second data processing module 1200 is configured to package the fragment data into Internet Protocol data. [0101] The data sending module 1300 is configured to send the internet protocol data. The Internet Protocol data is used to implement the interconnection between the front-end sensing device and the network.

[0102] In this embodiment, by performing a fragmentation operation on the data information in the Internet of Things gateway, the real data stream is converted into batch data, which can increase the amount of data processed in each subsequent processing process, and reduce the number of processing times. Therefore, the effect of improving data transmission and processing efficiency is achieved, which facilitates rapid storage and data analysis of a big data platform that subsequently receives data.

[0103] Preferably, the second data processing module 1200 is specifically configured to:

[0104] The column family of data is set according to the data attribute of the slice data.

[0105] The fragment data is divided according to the column family, and column family data corresponding to the fragment data is obtained. Referring to FIG. 6, for example, data segmentation is performed according to a fixed inter-turn interval 30s; data is included in each slice data: temperature, humidity, current, voltage, position, and device ID; then this embodiment will be established The column family of data including temperature, humidity, current, voltage, position, and device ID; further put together the data of the same column family of the same piece of data, that is, the columnar storage logic of the sliced data is realized.

[0106] Converting the column family data block into internet protocol data.

[0107] In this embodiment, by dividing the slice data into columns by the column family, the effect that the data of the same attribute is placed together can be obtained. Further, because of the similarity between the same attribute data, the same data attribute is more conducive to improving the data compression capability, thereby reducing the network transmission bandwidth. At the same time, it is also convenient to do column storage, data processing, etc. in the back-end big data platform.

[0108] Preferably, the step of converting, by the second data processing module 1200, the column family data block into the internet protocol data comprises:

[0109] partitioning the block data of the column family data according to a preset rule to obtain partition column family data of the block data having a logical partition; wherein the method of partitioning the column family data may be performed by using a hash hash or User-specified methods and other methods hash the column family data. Please refer to Figure 6. In this example, the storage mode of the voltage family is exemplified, and the voltage column family is divided into N partitions.

[0110] packaging the partitioned column family data into internet protocol data.

[0111] In this embodiment, by performing hash processing on the block data, the block data can be hashed into different partitions, so that the block data can be distributed and stored, and the data tilt problem can be solved, and the data reading capability can be improved. In the big data platform, distributed storage is the basis of distributed computing, and the logical architecture diagram of its platform is as Figure 7 shows. If the column family data is not partitioned, it will be concentrated on a certain disk in the storage process, which will form a read-write bottleneck during the calculation process, which affects the distributed computing performance. By partitioning the column family data, the column group data can be allocated to multiple disks in a partitioned manner, and multiple disk I/Os can be used to improve data reading capability, which is beneficial to the back-end big data platform to perform distributed computing on data. .

[0112] Preferably, the step of converting the packet of the second data processing module 1200 into internet protocol data comprises:

[0113] the data is cleaned according to the preset cleaning rule, and the cleaned data is compressed to obtain compressed data; wherein, the cleaning rule may be data cleaning of duplicate and abnormal data under the same device in the partition information; Compression can be performed using compression algorithms such as GZIP, LZO, and Snappy. It can also be encrypted in compression or encrypted after compression, such as MD5, RSA and other encryption methods.

[0114] converting the compressed data into internet protocol data according to a preset network rule.

[0115] In this embodiment, by cleaning and re-compressing data, it is advantageous to maximize compression and reduce network transmission bandwidth.

[0116] Preferably, the Internet of Things gateway further includes:

[0117] The backup module 1400 is configured to perform backup storage on the compressed data before the step of converting the compressed data into Internet protocol data according to the preset network rule by the second data processing module 1200.

[0118] In this embodiment, the compressed data is backed up, so as to perform data retransmission, search, and the like in the event of a network failure. The maximum backup period of data can be calculated and confirmed according to the amount of data processed by the IoT gateway every day and the capacity of the storage module. Users can set it themselves, but it cannot exceed the maximum backup period. The recommended backup period is 1 to 3 months. .

[0119] It is to be noted that the terms "comprising", "including", or any other variants thereof are intended to encompass a non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes Those elements, but also other elements not explicitly listed, or elements that are inherent to such a process, method, item or device. An element that is defined by the phrase "comprising a ..." does not exclude the presence of additional elements in the process, method, article, or device that comprises the element. [0120] The foregoing serial numbers of the embodiments of the present invention are merely for the description, and do not represent the advantages and disadvantages of the embodiments.

[0121] Through the description of the above embodiments, those skilled in the art can clearly understand that the foregoing embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be through hardware, but in many cases, the former It is a better implementation. Based on such understanding, the technical solution of the present invention, which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk, The optical disc includes a number of instructions for causing a terminal device (which may be a mobile terminal, a computer, a server, an air conditioner, or a network device, etc.) to perform the methods described in various embodiments of the present invention.

The above are only the preferred embodiments of the present invention, and are not intended to limit the scope of the present invention, and the equivalent structure or equivalent process transformations made by the description of the present invention and the contents of the drawings may be directly or indirectly applied to other related The technical field is equally included in the scope of patent protection of the present invention.

Industrial applicability

[0123] The IoT gateway data management method and the Internet of Things gateway provided by the present invention convert the real data stream into batch data by performing the fragmentation operation on the data information, which can increase each processing in the subsequent processing process. The amount of data reduces the number of processing, thereby improving the efficiency of data transmission and processing, and facilitating the rapid storage and data analysis of the big data platform that subsequently receives data.

Claims

Claim

An IoT gateway data management method, the data management method includes the steps of: obtaining, by the sensing device accessing the Internet of Things gateway, the data transmitted by the heterogeneous protocol, and obtaining the data information of the sensing device;

Performing data fragmentation on the data information according to a preset fragmentation rule in the Internet of Things gateway to obtain fragmentation data; the data information includes attributes of the data;

Converting the fragmented data into internet protocol data;

The internet protocol data is sent.

The method of managing the Internet of Things gateway data according to claim 1, wherein the step of converting the fragmented data into Internet Protocol data comprises:

Setting a column family of data according to a data attribute of the slice data;

And dividing the fragment data according to the column family to obtain column family data corresponding to the fragment data;

The column family data is packaged and converted into internet protocol data.

The IoT gateway data management method according to claim 2, wherein the step of converting the column family data into Internet Protocol data comprises:

The block data of the column family data is partitioned according to a preset rule, and the partition column family data of the block data having the logical partition is obtained;

The partitioned column family data is packaged and converted into internet protocol data.

The method for managing an Internet of Things gateway data according to claim 1, wherein the step of converting the package into Internet Protocol data comprises:

The data is cleaned according to a preset cleaning rule, and the cleaned data is compressed to obtain compressed data;

Converting the compressed data into Internet Protocol data according to a preset network rule.

The method for managing the Internet of Things gateway data according to claim 4, wherein the step of converting the compressed data into the Internet Protocol data according to the preset network rule further comprises: backing up the compressed data.

An IoT gateway, the IoT gateway includes: a parsing module, configured to parse the data sent by the heterogeneous protocol for the sensing device accessing the Internet of Things gateway to obtain the data information of the sensing device;

a first data processing module, configured to perform data fragmentation on the data information according to a preset fragmentation rule, to obtain fragmentation data; where the data information includes attributes of the data;

And a second data processing module, configured to package the fragment data into an internet protocol data data sending module, configured to send the internet protocol data.

The IoT gateway according to claim 6, wherein the second data processing module is specifically configured to:

The column family data block is packaged and converted into internet protocol data.

The IoT gateway according to claim 7, wherein the step of converting, by the second data processing module, the column family data block into internet protocol data comprises:

The IoT gateway according to claim 6, wherein the step of converting the packet of the second data processing module into Internet Protocol data comprises:

The IoT gateway of claim 9, wherein the IoT gateway further comprises: a backup module, configured to: in the second data processing module, convert the compressed data into an internet according to a preset network rule The compressed data is backed up and stored before the step of the protocol data.