CN108965483B - System implementation method for storing and pushing data of mass equipment in Internet of things system - Google Patents
System implementation method for storing and pushing data of mass equipment in Internet of things system Download PDFInfo
- Publication number
- CN108965483B CN108965483B CN201811143300.XA CN201811143300A CN108965483B CN 108965483 B CN108965483 B CN 108965483B CN 201811143300 A CN201811143300 A CN 201811143300A CN 108965483 B CN108965483 B CN 108965483B
- Authority
- CN
- China
- Prior art keywords
- data
- push
- topic
- url
- okhttp
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/55—Push-based network services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a system implementation method for storing and pushing data of a large number of devices in an Internet of things system, which comprises the following steps: the method comprises the following steps: dividing each device into groups to which the device belongs; step two: inquiring a device-topic table according to the source of data acquired from equipment to obtain a group topic to which the data belongs, and then correspondingly inserting the group topic into a push-data table; step three: establishing a mapping relation between topoc and url, and inserting the mapping relation into an okhttp-push table; step four: querying the push-data table by using values of topic and mid of each line in the okhttp-push table, wherein data which are not successfully received by a user corresponding to the url address are not yet below the topic, and then sequentially transmitting the data by using the okhttp; step five: and judging whether the user successfully receives the push message or not through response data of the http, and if the user successfully receives the push message, updating mids corresponding to the topic and the url in the http-push table. The invention realizes that a large amount of data collected from various devices, such as gateway devices, terminal nodes and other arbitrary devices, is stored in the database and is pushed to each user.
Description
Technical Field
The invention relates to the technical field of data storage and pushing. More specifically, the invention relates to a system implementation method for data storage and push of a large number of devices in an internet of things system.
Background
In the internet of things system, a large number of various devices exist, and a large amount of data gathered from the various devices needs to be stored in a database and also needs to be classified and pushed to various users. The traditional implementation method is that each piece of data gathered from the device is bound with the url address to be pushed by the data and then inserted into the database. When the same data is pushed to different url addresses, records with the same message content need to be inserted into the push-data table push-data, namely the total number of records in the database is equal to the number of messages multiplied by the number of url addresses, and meanwhile, when another process is used for pushing the data stored in the push-data table to each url address, a field which is used for judging whether the data is successfully sent or not in the table needs to be updated. Namely, a large amount of concurrent insertion and modification operations exist on the push-data table, and as the device data increases, the database becomes a bottleneck of system performance.
Disclosure of Invention
The invention aims to provide a system implementation method for storing and pushing data of a large number of devices in an Internet of things system, which is used for storing a large number of data collected from various devices (including but not limited to) such as gateway devices, terminal nodes and the like into a database and pushing the data to various users.
To achieve these objects and other advantages in accordance with the present invention, there is provided a system implementation method for data storage and push of a plurality of devices in an internet of things system, comprising the steps of:
the method comprises the following steps: constructing a database table structure, including a device-topic table, a push-data table, an okhttp-push table and a company url address table;
step two: according to different receivers to which the device data are to be sent, dividing each device into groups to which the device data belong, and inserting the corresponding relation of the device and the group to which the device data belong into a device-topic relation table;
step three: determining a topic parameter of the data according to the dev-eui of the equipment from which the data comes from the equipment through a device-topic table, and then correspondingly inserting the topic parameter into the push-data table;
step four: establishing a mapping relation between the topoc and the url, inserting the mapping relation into an okhttp-push table, and determining a url address to which data under the same topoc is sent;
step five: for each record in the okhttp-push table, respectively starting a thread: inquiring the push-data table by using values of topic and mid in the okhttp-push table, wherein the device data which are not successfully received by the user corresponding to the url address are not under the topic, and transmitting the device data by using the okhttp sequence according to the size of the primary key id;
step six: judging whether the user successfully receives the push message or not through response data of the http, and if so, updating the cursor mid corresponding to the topic and the url in the http-push table; and if the data is not received, the okhttp-push table is not updated, and the thread repeatedly reads the unsent device data for retransmission.
Preferably, the device-topic table includes parameters: the device table comprises a non-service primary key id, a unique device number address dev-eui and a group topic to which the device belongs.
Preferably, the push-data push data table includes parameters: the device table comprises a non-service primary key id, creation time create-time of device information, a unique device number address app-eui, a group topic to which the device belongs and gateway gateways of the device.
Preferably, the okhttp-push configuration table includes parameters: the device table includes a non-service primary key id, a group topic to which the device belongs, an id of url pushed by the message, i.e. url-id, and a position cursor mid where the message is successfully received.
Preferably, the corporate url address table includes the parameters: company table non-business primary key id, company name, url address url of company.
Preferably, the fourth step is specifically: and pushing equipment data to each user in an http synchronous mode, and configuring a corresponding relation between topic and a corresponding pushing address url in a company url address table through an sql statement.
The invention at least comprises the following beneficial effects:
1. the separation of the push-data table and the data push url address configuration table okhttp-push-config is realized, so that the data volume in the push-data table can be reduced;
2. the method and the device realize the insertion operation of the device data and the update operation of the data sending state of the same database table, and decompose the insertion operation and the update operation into the insertion operation and the update operation of two database tables respectively, thereby reducing the concurrent writing operation of the push-data table and improving the system performance;
3. the address url of the packet topic to which the device belongs and data transmission under the topic is flexibly configurable;
4. the local persistence of the data collected from the equipment is supported, the analysis and statistics of the data are convenient to carry out, and the data can be based on the current mainstream relational database.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention.
Detailed Description
The present invention is described in further detail below to enable those skilled in the art to practice the invention with reference to the description.
The invention provides a system implementation method for storing and pushing data of a large number of devices in an Internet of things system, which comprises the following steps:
the method comprises the following steps: constructing a database table structure, including a device-topic table, a push-data table, an okhttp-push table and a company url address table;
step two: according to different receivers to which the device data are to be sent, dividing each device into groups to which the device data belong, and inserting the corresponding relation of the device and the group to which the device data belong into a device-topic relation table;
step three: determining a topic parameter of the data according to the dev-eui of the equipment from which the data comes from the equipment through a device-topic table, and then correspondingly inserting the topic parameter into the push-data table;
step four: establishing a mapping relation between the topoc and the url, inserting the mapping relation into an okhttp-push table, and determining a url address to which data under the same topoc is sent; the method specifically comprises the following steps: pushing equipment data to each user in an http synchronous mode, configuring a corresponding relation between topic and a corresponding pushing address url in a company url address table through an sql statement, wherein the sql statement is as follows: "insert int okhttp-push (topoic, url-id) values ('value of topoic', 'id of url')";
step five: for each record in the okhttp-push table, respectively starting a thread: inquiring the push-data table by using values of topic and mid in the okhttp-push table, wherein the device data which are not successfully received by the user corresponding to the url address are not under the topic, and transmitting the device data by using the okhttp sequence according to the size of the primary key id;
step six: judging whether the user successfully receives the push message or not through response data of the http, and if so, updating the cursor mid corresponding to the topic and the url in the http-push table; and if the data is not received, the okhttp-push table is not updated, and the thread repeatedly reads the unsent device data for retransmission.
In another technical solution, the device-topic table includes parameters: the device table comprises a non-service primary key id, a unique device number address dev-eui and a group topic to which the device belongs.
The device-topic relation table stores grouping information of the equipment, and the related fields are explained by the following table structure:
in another technical solution, the push-data push data table includes parameters: the device table comprises a non-service primary key id, creation time create-time of device information, a unique device number address app-eui, a group topic to which the device belongs and gateway gateways of the device.
The push-data push data table stores data collected from equipment and a core table pushing the data to a user, all information reported by each equipment is stored in the table, and the table structure is as follows:
| name of field | Type of field | Description of field | Remarks for note |
| id | bigint | Non-service primary key of equipment table | Automatic increase |
| create-time | datetime | Creation time of device information | |
| application-id | bigint(20) | ||
| app-eui | varchar(255) | Device unique numbering address | |
| data | varchar(8000) | ||
| data-type | Int(11) | ||
| mac | varchar(255) | ||
| reserver | varchar(255) | ||
| v-id | varchar(255) | ||
| topic | int | Group to which the device belongs | |
| gateways | varchar(8000) | Gateway of equipment | |
| create-times | varchar(50) |
In another technical solution, the company url address table includes parameters: company table non-business primary key id, company name, url address url of company.
A company url address table, mgr-company url address table, url addresses for storing data pushes of each company
In another technical scheme, the okhttp-push configuration table includes parameters: the device table includes a non-service primary key id, a group topic to which the device belongs, an id of url pushed by the message, i.e. url-id, and a position cursor mid where the message is successfully received.
okhttp-push configuration table
| id | bigint | Non-service key | Automatic increase |
| topic | int | Group to which the device belongs | |
| url-id | bigint | Id of url of message push | |
| mid | bigint | Position cursor for successful message receipt |
While embodiments of the invention have been described above, it is not limited to the applications set forth in the description and the embodiments, which are fully applicable to various fields of endeavor for which the invention may be embodied with additional modifications as would be readily apparent to those skilled in the art, and the invention is therefore not limited to the details given herein and to the embodiments shown and described without departing from the generic concept as defined by the claims and their equivalents.
Claims (4)
1. The system implementation method for storing and pushing data of a large number of devices in the Internet of things system is characterized by comprising the following steps:
the method comprises the following steps: constructing a database table structure, including a device-topic table, a push-data table, an okhttp-push table and a company url address table;
step two: according to different receivers to which the device data are to be sent, dividing each device into groups to which the device data belong, and inserting the corresponding relation of the device and the group to which the device data belong into a device-topic relation table;
step three: determining a topic parameter of the data according to the dev-eui of the equipment from which the data comes from the equipment through a device-topic table, and then correspondingly inserting the topic parameter into the push-data table;
step four: establishing a mapping relation between the topoc and the url, inserting the mapping relation into an okhttp-push table, and determining a url address to which data under the same topoc is sent;
step five: for each record in the okhttp-push table, respectively starting a thread: inquiring the push-data table by using values of topic and mid in the okhttp-push table, wherein the device data which are not successfully received by the user corresponding to the url address are not under the topic, and transmitting the device data by using the okhttp sequence according to the size of the primary key id;
step six: judging whether the user successfully receives the push message or not through response data of the http, and if so, updating the cursor mid corresponding to the topic and the url in the http-push table; if not, not updating the okhttp-push table, and repeatedly reading unsent device data by the thread for retransmission;
the push-data table comprises parameters: the method comprises the following steps that a device table non-service main key id, creation time create-time of device information, a device unique number address app-eui, a group topic to which the device belongs and a gateway of the device;
the okhttp-push table comprises the parameters: the device table includes a non-service primary key id, a group topic to which the device belongs, an id of url pushed by the message, i.e. url-id, and a position cursor mid where the message is successfully received.
2. The system implementation method for data storage and push of a plurality of devices in an internet of things system according to claim 1, wherein the device-topic table comprises parameters: the device table comprises a non-service primary key id, a unique device number address dev-eui and a group topic to which the device belongs.
3. The method for implementing data storage and push of a plurality of devices in an internet of things system according to claim 1, wherein the company url address table includes parameters: company table non-business primary key id, company name, url address url of company.
4. The system implementation method for storing and pushing data of a large number of devices in an internet of things system according to claim 1, wherein the fourth step is specifically: and pushing equipment data to each user in an http synchronous mode, and configuring a corresponding relation between topic and a corresponding pushing address url in a company url address table through an sql statement.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811143300.XA CN108965483B (en) | 2018-09-28 | 2018-09-28 | System implementation method for storing and pushing data of mass equipment in Internet of things system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811143300.XA CN108965483B (en) | 2018-09-28 | 2018-09-28 | System implementation method for storing and pushing data of mass equipment in Internet of things system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108965483A CN108965483A (en) | 2018-12-07 |
| CN108965483B true CN108965483B (en) | 2021-04-23 |
Family
ID=64472307
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811143300.XA Active CN108965483B (en) | 2018-09-28 | 2018-09-28 | System implementation method for storing and pushing data of mass equipment in Internet of things system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108965483B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109918393A (en) * | 2019-01-28 | 2019-06-21 | 武汉慧联无限科技有限公司 | The data platform and its data query and multilist conjunctive query method of Internet of Things |
| CN111767276B (en) * | 2020-06-29 | 2024-03-15 | 北京百度网讯科技有限公司 | Data storage method, device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101106528A (en) * | 2007-07-31 | 2008-01-16 | 杭州华三通信技术有限公司 | Packet forward system and method based on secure device and its secure device |
| CN102325061A (en) * | 2011-09-16 | 2012-01-18 | 北京星网锐捷网络技术有限公司 | Method for monitoring network, equipment and system |
| CN106896737A (en) * | 2017-03-10 | 2017-06-27 | 南京邮电大学 | A kind of intelligent domestic Integrated Information Processing System |
| WO2018106159A1 (en) * | 2016-12-05 | 2018-06-14 | Telefonaktiebolaget Lm Ericsson (Publ) | Methods and apparatus for operating and managing a constrained device within |
-
2018
- 2018-09-28 CN CN201811143300.XA patent/CN108965483B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101106528A (en) * | 2007-07-31 | 2008-01-16 | 杭州华三通信技术有限公司 | Packet forward system and method based on secure device and its secure device |
| CN102325061A (en) * | 2011-09-16 | 2012-01-18 | 北京星网锐捷网络技术有限公司 | Method for monitoring network, equipment and system |
| WO2018106159A1 (en) * | 2016-12-05 | 2018-06-14 | Telefonaktiebolaget Lm Ericsson (Publ) | Methods and apparatus for operating and managing a constrained device within |
| CN106896737A (en) * | 2017-03-10 | 2017-06-27 | 南京邮电大学 | A kind of intelligent domestic Integrated Information Processing System |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108965483A (en) | 2018-12-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107332876B (en) | Method and device for synchronizing block chain state | |
| US6751627B2 (en) | Method and apparatus to facilitate accessing data in network management protocol tables | |
| US7400625B2 (en) | Protocol for multicast communication | |
| CN101286936B (en) | Method and apparatus for data message processing | |
| AU2010265607B2 (en) | Method for finding, updating and synchronizing modified record item and data synchronizing device | |
| CN103873451B (en) | A kind of method, apparatus and system of data syn-chronization | |
| CN101345707B (en) | Method and apparatus for implementing IPv6 packet classification | |
| CN102110132A (en) | Uniform resource locator matching and searching method, device and network equipment | |
| CN104484295B (en) | Receiver sliding window-based data transmission method in parallel computer system | |
| CN108965483B (en) | System implementation method for storing and pushing data of mass equipment in Internet of things system | |
| CN112003740B (en) | Automatic multicast configuration method and system for substation switch | |
| WO2017000592A1 (en) | Data processing method, apparatus and system | |
| CN101557380A (en) | Data synchronization method, client terminal and server | |
| US8984100B2 (en) | Data downloading method, terminal, server, and system | |
| CN110020243A (en) | Querying method, device, Internet of Things server and the storage medium of internet of things data | |
| CN111245725B (en) | Future network implementation method based on distribution | |
| CN113609518A (en) | Message protocol overtime retransmission method and system based on associated container map | |
| CN109634757B (en) | Data acquisition method for seismic industry big data processing | |
| CN113794994B (en) | Information gathering method and device based on multicast domain name system and application thereof | |
| CN105630983A (en) | Resource obtaining and optimizing device and method | |
| Xiang et al. | Impact of multi-query optimization in sensor networks | |
| CN114884893A (en) | Forwarding and control definable cooperative traffic scheduling method and system | |
| CN107040455A (en) | A kind of instant messaging communication method and system based on redis | |
| CN113343034A (en) | IP searching method, system and storage medium | |
| CN101799803A (en) | Method, module and system for processing information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: System implementation method of data storage and push of a large number of equipment in Internet of things system Effective date of registration: 20220125 Granted publication date: 20210423 Pledgee: Bank of Communications Ltd. Wuhan East Lake New Technology Development Zone sub branch Pledgor: WUHAN EASYLINKIN TECHNOLOGY CO.,LTD Registration number: Y2022420000034 |
|
| PC01 | Cancellation of the registration of the contract for pledge of patent right | ||
| PC01 | Cancellation of the registration of the contract for pledge of patent right |
Date of cancellation: 20230201 Granted publication date: 20210423 Pledgee: Bank of Communications Ltd. Wuhan East Lake New Technology Development Zone sub branch Pledgor: WUHAN EASYLINKIN TECHNOLOGY CO.,LTD Registration number: Y2022420000034 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: System implementation method for data storage and push of a large number of devices in the Internet of Things system Effective date of registration: 20230206 Granted publication date: 20210423 Pledgee: Bank of Communications Ltd. Wuhan East Lake New Technology Development Zone sub branch Pledgor: WUHAN EASYLINKIN TECHNOLOGY CO.,LTD Registration number: Y2023420000024 |