CN111131219B - Efficient data transmission method for Internet of things based on FlatBuffers - Google Patents
Efficient data transmission method for Internet of things based on FlatBuffers Download PDFInfo
- Publication number
- CN111131219B CN111131219B CN201911317809.6A CN201911317809A CN111131219B CN 111131219 B CN111131219 B CN 111131219B CN 201911317809 A CN201911317809 A CN 201911317809A CN 111131219 B CN111131219 B CN 111131219B
- Authority
- CN
- China
- Prior art keywords
- flatbuffers
- message
- serialized
- internet
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 21
- 230000005540 biological transmission Effects 0.000 title claims abstract description 15
- 238000012545 processing Methods 0.000 claims abstract description 27
- 238000004891 communication Methods 0.000 description 4
- 238000013500 data storage Methods 0.000 description 3
- 235000006719 Cassia obtusifolia Nutrition 0.000 description 2
- 235000014552 Cassia tora Nutrition 0.000 description 2
- 244000201986 Cassia tora Species 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
- H04L67/61—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources taking into account QoS or priority requirements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/08—Protocols for interworking; Protocol conversion
Abstract
The invention relates to the technical field of Internet of things, and discloses a high-efficiency data transmission method of the Internet of things based on FlatBuffers, which comprises the following steps: s1) the publisher acquires the message, and serializes the message by using FlatBuffers; s2) transmitting the serialized messages; s3) the MQTT server acquires the information after the serialization processing, and carries out the deserialization processing on the information after the serialization processing by using FlatBuffers; s4) the Subscriber acquires the serialized message and deserializes the serialized message using the FlatBuffers. The invention utilizes the FlatBuffers data protocol to carry out serialization processing and deserialization, can reduce the length of the carried data in the message and simultaneously reduce the serialization cost and the deserialization cost of the data.
Description
Technical Field
The invention relates to the technical field of Internet of things, in particular to a high-efficiency data transmission method of the Internet of things based on FlatBuffers.
Background
MQTT (Message Queuing Telemetry Transport) is a "lightweight" communication protocol based on publish/subscribe (publish/subscribe) mode, which is built on TCP/IP protocol, published by IBM in 1999, and has three identities in MQTT protocol: publisher (pushlife), Broker (Broker) (server), subscriber (Subscribe). The message publisher and the message subscriber are clients, the message broker is a server, and the message publisher can be a subscriber at the same time. The MQTT has the greatest advantage that a real-time reliable message service can be provided for connecting remote devices with few codes and limited bandwidth. As an instant messaging protocol with low cost and low bandwidth occupation, the method has wide application in the aspects of Internet of things, small-sized equipment, mobile application and the like.
The MQTT communication protocol is a protocol designed for remote sensor or control communication of a large number of networks with limited computing power and low bandwidth and unreliable, and has the following characteristics: 1) using publish/subscribe messaging schema, providing one-to-many message publishing, decoupling applications: 2) message transmission for load content shielding; 3) providing a network connection using TCP/IP; 4) there are three types of quality of service (Qos) for message distribution:
at most once: message distribution relies entirely on the underlying TCP/IP network. Message loss or duplication can occur. This level can be used in situations where environmental sensor data is lost, so it is not said that a read record is lost, since a second transmission will occur shortly.
At least once: message arrival is assured but message duplication may occur.
Only once: ensuring that the message arrives once. This level can be used in situations where message duplication or loss can lead to incorrect results in a billing system. The highest quality message publishing service can also be used for pushing an instant messaging APP to ensure that a user receives the APP once.
In addition to the first message distribution quality of service, the message content needs to be temporarily stored. Meanwhile, in an actual project, the Broker side can collect the published information and store the published information in various databases for data analysis, AI learning, strategy judgment and the like. Currently, the mainstream technology basically adopts json (javascript Object notification) format to store Payload data of a message in a message. Although simple to use and easy to understand, it has problems in the size of data storage and the speed of data processing. Especially for the data distribution occasions with the connection of the devices with the maximum level of 100 ten thousand, and more than 1 ten thousand times per second, the data unloading pressure of the Broker server is very large. In the environments of Internet of things and the like facing large-scale MQTT connection and high event release throughput, the json-based Payload data storage scheme has the problems of large data size and low data processing speed.
Disclosure of Invention
The invention aims to provide a high-efficiency data transmission method of the Internet of things based on FlatBuffers, and therefore the problems that in the prior art, a json-based Payload data storage method is large in data size and low in data processing speed are solved.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
the efficient data transmission method of the Internet of things based on FlatBuffers comprises the following steps:
s1) the publisher acquires the message and carries out serialization processing on the message by using FlatBuffers;
s2) transmitting the serialized messages;
s3) the MQTT server acquires the information after the serialization processing, and carries out the deserialization processing on the information after the serialization processing by using FlatBuffers;
s4) the Subscriber acquires the serialized message and deserializes the serialized message using the FlatBuffers.
The FlatBuffers is an open-source cross-platform data serialization library, can be applied to almost any language (C + +, C #, Go, Java, JavaScript, PHP, Python), and stores the structured object by using a flattened (Flat) buffer, namely, stores the memory object data in a one-dimensional array. Such a design allows the FlatBuffers to have the following advantages: the access to the serialized data does not need packaging and unpacking, and the FlatBuffers stores the serialized data in a cache, and the data can be stored in a file and transmitted through a network without any analysis overhead; the only memory requirement is a buffer area when data is accessed, extra memory allocation is not needed, the memory efficiency is high, and the speed is high; only a small amount of automatically generated code and a single header file are required to be relied on, and the system is easy to integrate into the existing system; the system can be used in a cross-platform mode, and has strong type design and good expansibility. The data protocol of FlatBuffers is used in the protocol Payload of the MQTT, so that the length of data carried in a message can be greatly reduced, and meanwhile, the serialization cost and the deserialization cost of the data are greatly reduced.
Further, the message in step S1) includes a Fixed header (Fixed header), a Variable header (Variable header), and a packet Payload (Payload).
Further, the message payload in the message is serialized by using FlatBuffers.
Further, the message payload in the message is deserialized using FlatBuffers.
Further, the serialization process comprises the following steps:
s11) defining a data structure and writing a schema file;
s12) compiling the schema file by using a FlatBuffers compiler flatc to obtain a compiled file;
s13) compiling the source code of the FlatBuffers by using a tool to obtain a jar package of the FlatBuffers, wherein the tool comprises a maven tool;
s14) adding jar packages and compiled files of FlatBuffers in the MQTT project;
s15) constructing a serialized object by using the FlatBufferBuilder, and putting the message effective load of the MQTT into the serialized object;
s16) storing or sending the data in the serialized object buffer.
Further, the deserializing process includes reading corresponding data from the serialized object cache region, where the corresponding data includes a packet payload.
The invention has the beneficial effects that: the invention utilizes the FlatBuffers data protocol to carry out serialization processing and deserialization, can reduce the length of data carried in the data message, and simultaneously reduces the serialization cost and the deserialization cost of the data.
Drawings
FIG. 1 is an overall flow chart of an embodiment of the present invention.
Fig. 2 is a schematic diagram of data transmission of MQTT communication protocol according to an embodiment of the present invention.
FIG. 3 is a graph comparing the performance of FlatBuffers and json according to an embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention.
In a first embodiment, as shown in fig. 1 and fig. 2, a method for efficient data transmission of an internet of things based on a FlatBuffers is characterized by including the following steps:
s1) the publisher obtains a message, where the message includes a fixed header, a variable header and a packet payload, and performs serialization processing on the packet payload by using flitbuffers, as shown in fig. 2, including the steps of:
s11) defining a data structure and writing a schema file;
s12) compiling the schema file by using a FlatBuffers compiler flatc to obtain a compiled file;
s13) compiling the source code of the FlatBuffers by using a tool to obtain a jar package of the FlatBuffers;
s14) adding jar packages and compiled files of FlatBuffers in the MQTT project;
s15) constructing a serialized object by using the FlatBufferBuilder, and putting the message effective load of the MQTT into a serialized object cache region;
s16) storing or sending the data in the serialized object buffer.
S2) transmitting the serialized messages;
s3) the MQTT server obtains the information after the serialization processing, and carries out the deserialization processing on the message payload in the information after the serialization processing by using FlatBuffers, wherein the deserialization processing comprises reading corresponding data from a serialization object buffer area, and the corresponding data comprises the message payload.
S4) the Subscriber obtains the serialized message and performs deserialization on the serialized message using the FlatBuffers, including reading corresponding data from the serialized object cache, where the corresponding data includes a packet payload.
Taking Java as an example, using flitbuffers to perform data serialization and deserialization in a payload entity of a message payload of MQTT according to the following steps:
1) compiling a schema file;
2) compiling the schema file by using a FlatBuffers compiler, namely, a flash file, and generating a JavaBean file, wherein the JavaBean is a reusable Java component and comprises Properties such as Properties (Properties), Methods (Methods), Events (Events) and the like.
3) Compiling the source code of the FlatBuffers by using a maven tool to obtain a jar packet of the FlatBuffers;
4) constructing a serialized object (ByteBuffer) by using a FlatBufferBuilder, and putting a message payload of the MQTT into a serialized object cache region;
5) storing or sending data in the serialized object cache region;
6) the buffer where the serialized objects reside is read and deserialized by the code getriotasxxx. XXX stands for object name.
As shown in fig. 3, the ordinate represents different types of data, and the abscissa represents the processing speed. Jackson is a Java class library used for processing JSON format data, and for the aspect of serialization processing speed of various types of data, FlatBuffers are faster than JSON in the serialization processing process and the deserialization processing process, which shows that the FlatBuffers are greatly improved in serialization performance and the deserialization performance.
By adopting the technical scheme disclosed by the invention, the following beneficial effects are obtained:
the invention uses the FlatBuffers data protocol to carry out serialization processing and deserialization, can reduce the length of the carried data in the message and simultaneously reduce the serialization cost and the deserialization cost of the data.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that it will be apparent to those skilled in the art that various modifications and improvements can be made without departing from the principle of the present invention, and such modifications and improvements should also be considered within the scope of the present invention.
Claims (5)
1. The efficient data transmission method of the Internet of things based on FlatBuffers is characterized by comprising the following steps:
s1) the publisher acquires the message, and carries out serialization processing on the message by using FlatBuffers, comprising the following steps:
s11) defining a data structure and writing a schema file;
s12) compiling the schema file by using a FlatBuffers compiler flatc to obtain a compiled file;
s13) compiling the source code of the FlatBuffers by using tools, and obtaining jar packages of the FlatBuffers, wherein the tools comprise maven tools;
s14) adding jar packages and compiled files of FlatBuffers in the MQTT project;
s15) constructing a serialized object by using the FlatBufferBuilder, and putting the message payload of the MQTT into a serialized object cache region;
s16) storing or sending the data in the serialized object buffer;
s2) transmitting the serialized messages;
s3) the MQTT server acquires the information after the serialization processing, and carries out the deserialization processing on the information after the serialization processing by using FlatBuffers;
s4) the Subscriber acquires the serialized message and deserializes the serialized message using the FlatBuffers.
2. The method for efficient data transmission of the FlatBuffers-based Internet of things of claim 1, wherein the message in the step S1) comprises a fixed header, a variable header and a message payload.
3. The method for efficient data transmission of the Internet of things based on FlatBuffers as claimed in claim 1 or 2, wherein the message payloads in the messages are serialized by using FlatBuffers.
4. The method for efficient data transmission of the Internet of things based on FlatBuffers as claimed in claim 3, wherein the FlatBuffers is used for deserializing the message payload in the message.
5. The method for efficient data transmission of the Internet of things based on FlatBuffers as claimed in claim 1, wherein the deserialization process comprises reading corresponding data from the serialized object buffer, wherein the corresponding data comprises a message payload.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911317809.6A CN111131219B (en) | 2019-12-19 | 2019-12-19 | Efficient data transmission method for Internet of things based on FlatBuffers |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911317809.6A CN111131219B (en) | 2019-12-19 | 2019-12-19 | Efficient data transmission method for Internet of things based on FlatBuffers |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111131219A CN111131219A (en) | 2020-05-08 |
| CN111131219B true CN111131219B (en) | 2022-04-05 |
Family
ID=70500107
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911317809.6A Active CN111131219B (en) | 2019-12-19 | 2019-12-19 | Efficient data transmission method for Internet of things based on FlatBuffers |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111131219B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111984679B (en) * | 2020-07-02 | 2021-06-04 | 中科驭数（北京）科技有限公司 | Access method, device, host, system and medium of hardware acceleration database |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103970737A (en) * | 2013-01-24 | 2014-08-06 | 腾讯科技（深圳）有限公司 | Data constitution method and device |
| CN106713423A (en) * | 2016-12-06 | 2017-05-24 | 上海斐讯数据通信技术有限公司 | Distributed data processing method and device for cloud access point controller |
| CN107294778A (en) * | 2017-06-23 | 2017-10-24 | 济南浪潮高新科技投资发展有限公司 | A kind of method that industrial equipment collection is realized based on cloud Internet of Things platform |
| CN108243259A (en) * | 2018-02-08 | 2018-07-03 | 北京车和家信息技术有限公司 | Transmission method, the apparatus and system of car networking data |
| CN108809972A (en) * | 2018-06-01 | 2018-11-13 | 南京邮电大学 | A kind of Internet of Things Comprehensive Experiment and application development platform and framework based on the ecosystem of increasing income |
| CN109032699A (en) * | 2018-07-23 | 2018-12-18 | 北京轻元科技有限公司 | A kind of method and terminal for modifying application environment variable |
| CN109257337A (en) * | 2018-08-28 | 2019-01-22 | 科大国创软件股份有限公司 | Internet of things equipment method for converting protocol and lamp stand operation platform based on this method |
| CN105407024B (en) * | 2015-09-23 | 2019-04-09 | 中国电子科技集团公司第二十九研究所 | A kind of isomeric data interoperability methods and device based on distribution subscription communication mechanism |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040076166A1 (en) * | 2002-10-21 | 2004-04-22 | Patenaude Jean-Marc Guy | Multi-service packet network interface |
| US9930103B2 (en) * | 2015-04-08 | 2018-03-27 | Amazon Technologies, Inc. | Endpoint management system providing an application programming interface proxy service |
| US10812582B2 (en) * | 2016-03-10 | 2020-10-20 | Vmware, Inc. | Management of applications across nodes using exo-clones |
-
2019
- 2019-12-19 CN CN201911317809.6A patent/CN111131219B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103970737A (en) * | 2013-01-24 | 2014-08-06 | 腾讯科技（深圳）有限公司 | Data constitution method and device |
| CN105407024B (en) * | 2015-09-23 | 2019-04-09 | 中国电子科技集团公司第二十九研究所 | A kind of isomeric data interoperability methods and device based on distribution subscription communication mechanism |
| CN106713423A (en) * | 2016-12-06 | 2017-05-24 | 上海斐讯数据通信技术有限公司 | Distributed data processing method and device for cloud access point controller |
| CN107294778A (en) * | 2017-06-23 | 2017-10-24 | 济南浪潮高新科技投资发展有限公司 | A kind of method that industrial equipment collection is realized based on cloud Internet of Things platform |
| CN108243259A (en) * | 2018-02-08 | 2018-07-03 | 北京车和家信息技术有限公司 | Transmission method, the apparatus and system of car networking data |
| CN108809972A (en) * | 2018-06-01 | 2018-11-13 | 南京邮电大学 | A kind of Internet of Things Comprehensive Experiment and application development platform and framework based on the ecosystem of increasing income |
| CN109032699A (en) * | 2018-07-23 | 2018-12-18 | 北京轻元科技有限公司 | A kind of method and terminal for modifying application environment variable |
| CN109257337A (en) * | 2018-08-28 | 2019-01-22 | 科大国创软件股份有限公司 | Internet of things equipment method for converting protocol and lamp stand operation platform based on this method |
Non-Patent Citations (4)
| Title |
|---|
| Design and Implementation of Push Notification System Based on the MQTT Protocol;Konglong Tang;《International Conference on Information Science and Computer Applications (ISCA 2013）》;20130930;全文 * |
| FlatBuffers入门简介;泽_渊;《简书https://www.jianshu.com/p/8df23cd182ec》;20170420;正文第1页第2段-第5页倒数第1段 * |
| 基于MQTT的安全通信服务器的研究与实现;钱玉磊;《CNKI硕士论文全文数据库》;20151231;摘要、正文第10-14页 * |
| 钱玉磊.基于MQTT的安全通信服务器的研究与实现.《CNKI硕士论文全文数据库》.2015,摘要、正文第10-14页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111131219A (en) | 2020-05-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110943911B (en) | High-efficiency data transmission method for Internet of things based on protobuf | |
| CN110022289B (en) | Data transmission method, device and system | |
| KR101996598B1 (en) | Priority-based MQTT System and method to provide Differentiated IoT Services | |
| KR101159364B1 (en) | Mapping between object oriented and service oriented representations of a distributed application | |
| KR101213873B1 (en) | Flexibly transferring typed application data | |
| FI125393B (en) | A method, apparatus and system for use in a web service | |
| CN108769017B (en) | Data communication method and device | |
| US20090276451A1 (en) | Method and apparatus for processing messages | |
| CN110086759B (en) | Method and device for realizing message transmission between heterogeneous systems | |
| US8898220B2 (en) | Remote method invocation tunneling over hypertext transfer protocol | |
| CN104468704A (en) | Web server system supporting content center network and processing method | |
| CN111131219B (en) | Efficient data transmission method for Internet of things based on FlatBuffers | |
| US8819135B2 (en) | Method of performing data mediation, and an associated computer program product, data mediation device and information system | |
| CN110247926A (en) | A kind of exchange method and system | |
| CN1929463A (en) | Object request broker, method and device for providing resource high-efficiency transmission | |
| WO2023246309A1 (en) | Data processing method, apparatus and system | |
| Pradana et al. | Flatbuffers implementation on mqtt publish/subscribe communication as data delivery format | |
| CN112417016A (en) | Data exchange method, system, equipment and storage medium | |
| CN112769741B (en) | Message communication method and electronic equipment | |
| US20150081774A1 (en) | System and method for implementing augmented object members for remote procedure call | |
| CN112817539A (en) | Industrial data storage method and system, electronic device and storage medium | |
| CN111935135B (en) | AMQP protocol proxy method based on CMSP | |
| CN108737525A (en) | A kind of Web service system based on REST frameworks | |
| CN115941454A (en) | Log transmission and storage method and device | |
| CN116962319A (en) | Calling method, calling device, network equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231121 Address after: 5089, 5th Floor, Building 2, China Agricultural University International Entrepreneurship Park, No. 10 Tianxiu Road, Haidian District, Beijing, 100193 Patentee after: Fenomen array (Beijing) Technology Co.,Ltd. Address before: 100000 a5-036, maker Plaza, 338 East Street, Huilongguan town, Changping District, Beijing Patentee before: BEIJING QINGYUAN TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right |