CN114401269A - Business data distribution method and system and Internet of things management platform - Google Patents
Business data distribution method and system and Internet of things management platform Download PDFInfo
- Publication number
- CN114401269A CN114401269A CN202111489684.2A CN202111489684A CN114401269A CN 114401269 A CN114401269 A CN 114401269A CN 202111489684 A CN202111489684 A CN 202111489684A CN 114401269 A CN114401269 A CN 114401269A
- Authority
- CN
- China
- Prior art keywords
- data
- message
- management platform
- internet
- edge
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 20
- 230000003993 interaction Effects 0.000 claims abstract description 21
- 239000012634 fragment Substances 0.000 claims abstract description 16
- 238000007726 management method Methods 0.000 claims description 56
- 230000004044 response Effects 0.000 claims description 8
- 238000007781 pre-processing Methods 0.000 claims description 5
- 230000003139 buffering effect Effects 0.000 claims description 2
- 238000013500 data storage Methods 0.000 claims description 2
- 238000005192 partition Methods 0.000 abstract description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000009466 transformation Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/141—Setup of application sessions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/146—Markers for unambiguous identification of a particular session, e.g. session cookie or URL-encoding
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Telephonic Communication Services (AREA)
Abstract
The invention discloses a service data distribution method, a system and an internet of things management platform, wherein the method comprises the following steps: receiving terminal equipment data reported by each edge equipment through cloud edge interaction topic; the terminal equipment data comprises management data and service data; writing the data of the terminal equipment into a message queue for caching; performing professional division according to the unique equipment identification in the cloud-edge interaction of the Internet of things management platform and the edge equipment; and for each divided specialty, matching the terminal equipment data written into the message queue with consumers according to the preset Cartesian product of the number of message topic fragments and the number of message topic partitions to perform message consumption, and distributing the message consumption to each service system. The invention can improve the data distribution performance of the Internet of things management platform and ensure the high efficiency and low ductility of the business data distributed from the Internet of things management platform to the business system.
Description
Technical Field
The invention relates to a service data distribution method, a service data distribution system and an internet of things management platform, and belongs to the technical field of data processing.
Background
The electric power internet of things management platform is an indispensable part of an electric power internet of things, plays a role in starting and ending, is mainly used for realizing the unified access, management and application of various professional and various types of terminal equipment, provides standardized data and platform capacity for enterprise middlings and business systems in an open interface mode, and mainly comprises functions of connection management, equipment management, message processing, northbound interface service, platform management and the like.
Aiming at the problems that more and more terminal devices are accessed into the current power internet of things management platform, more and more reported data are reported, and the data caching and distributing capacity of the power internet of things management platform faces huge challenges. The service system has higher and higher requirements on the real-time performance of terminal equipment data, and the improvement of the message processing capability of the power internet of things management platform is the most important problem in the prior art.
Disclosure of Invention
The invention aims to overcome the defects in the prior art, and provides a service data distribution method, a service data distribution system and an internet of things management platform, which can ensure the high school and low delay of service data distribution.
In order to achieve the purpose, the invention is realized by adopting the following technical scheme:
in a first aspect, the present invention provides a service data distribution method, where the method includes:
receiving terminal equipment data reported by each edge equipment through cloud edge interaction topic; the terminal equipment data comprises management data and service data;
caching the terminal equipment data according to the divided professional write message queue;
and matching the terminal equipment data written into the message queue with consumers according to the preset Cartesian product of the number of the message topic fragments and the number of the message topic fragments to perform message consumption, and distributing the message consumption to each service system.
With reference to the first aspect, further, the specialties are divided according to unique device identifiers in the interaction between the internet of things management platform and the edge device cloud side.
With reference to the first aspect, further, before receiving the terminal device data reported by the edge device, the internet of things management platform establishes a long connection with the edge device based on an MQTT protocol.
With reference to the first aspect, further, before writing the terminal device data into the message queue for buffering, preprocessing the terminal device data, where the preprocessing includes: and adding message header information of the Internet of things management platform.
With reference to the first aspect, the management data includes command issuing and response, request and response, and is used to implement real-time command interaction or service processing between the internet of things management platform and the edge device; the service data includes event data and state data of the terminal device.
In a second aspect, the present invention provides an internet of things management platform, including:
a connection unit: the cloud-side interaction topic is used for receiving terminal equipment data reported by the edge equipment through the cloud-side interaction topic and writing the received terminal equipment data into the message queue according to the divided professional topic; the terminal equipment data comprises management data and service data;
message queue: the data storage module is used for writing and caching the terminal equipment data;
a data distribution unit: and the system is used for matching the messages written into the message queue with consumers according to the preset Cartesian product of the number of the message topic fragments and the number of the message topic fragments to perform message consumption and distributing the messages to each service system.
With reference to the second aspect, further, the connection unit is further configured to establish a long connection with the edge device based on an MQTT protocol before receiving the terminal device data reported by the edge device.
In a third aspect, the present invention provides a business data distribution system, including the internet of things management platform according to any one of the second aspects, a plurality of edge devices, and a business system;
the edge device is used for reporting terminal device data through the connection unit;
and the service system is used for receiving the message distributed by the data distribution unit and executing related processing tasks according to the message.
Compared with the prior art, the invention has the following beneficial effects:
and matching the terminal equipment data written into the message queue with consumers for message consumption according to the preset Cartesian product of the number of message topic fragments and the number of message topic fragments, so that the multi-consumer parallel consumption of the terminal equipment data can be realized, and the high efficiency and low ductility of the service data distributed to a service system by the Internet of things management platform are ensured.
Drawings
Fig. 1 is a schematic block diagram of a service data distribution system according to a third embodiment of the present invention.
Detailed Description
The invention provides a business data distribution method, an internet of things management platform capable of realizing the method and a distribution system adopting the internet of things management platform based on an electric power internet of things cloud side interaction specification topic and taking a message queue component kafka as an example, so that the data distribution performance of the internet of things management platform can be improved, and the high efficiency and low ductility of business data distribution from the internet of things management platform to the business system are ensured.
The technical solutions of the present invention are described in detail below with reference to the drawings and specific embodiments, and it should be understood that the specific features in the embodiments and examples of the present invention are described in detail in the technical solutions of the present application, and are not limited to the technical solutions of the present application, and the technical features in the embodiments and examples of the present application may be combined with each other without conflict.
The first embodiment is as follows:
the embodiment of the invention provides an internet of things management platform which comprises a connecting unit, a message queue and a data distribution unit which are connected in sequence.
The connecting units are arranged in one-to-one correspondence with external edge devices, and the connecting units can be connected with the terminal devices through the edge devices and are used for collecting terminal device data uploaded by the terminal devices through the edge devices. .
The terminal device data is mainly divided into management data and service data. The management data comprises command issuing and response, request and response and the like, the data is mainly used for realizing real-time command interaction or service processing of the Internet of things management platform and the edge device, and the data magnitude is small. The service data mainly includes event data and state data of the terminal device, which belong to periodic sampling data for actively reporting the terminal device, and the data is large in volume and frequent, is data which is continuously concerned by the service system in real time, and is also a message processing flow which needs key guarantee on the side of the internet of things management platform.
The message queue is used for writing in terminal equipment data and caching the terminal equipment data; the data of the terminal equipment is also required to be preprocessed before being stored in the message queue, and the preprocessing comprises adding a message header of the IOT management platform. It should be noted that before caching terminal device data in a message queue, professional division is performed according to a unique device identifier (edgeID) in the interaction between the internet of things management platform and the edge device cloud side;
and the data distribution unit is used for matching the messages written into the message queue with consumers according to the preset Cartesian product of the number of the message topic fragments and the number of the message topic partitions to perform message consumption and distributing the messages to each service system.
The cloud side interaction of the power internet of things stipulates a protocol specification for interaction between edge equipment and an internet of things management platform in an MQTT mode, a universal topic format and a message format which are interactively used are stipulated in the specification, and in the actual application process, the internet of things management platform needs to perform equipment access and data distribution according to dimensions of various specialties, projects, regions and the like in the power industry.
Example two:
the embodiment of the invention provides a service data distribution method, which can be realized by adopting the internet of things management platform of the embodiment I, and specifically comprises the following steps:
the method comprises the following steps: receiving terminal equipment data reported by each edge equipment through cloud edge interaction topic; the terminal equipment data comprises management data and service data;
the management data comprises command issuing and response, request and response and the like, the data is mainly used for realizing real-time command interaction or service processing of the Internet of things management platform and the edge device, and the data magnitude is small. The service data mainly includes event data and state data of the terminal device, which belong to periodic sampling data for actively reporting the terminal device, and the data is large in volume and frequent, is data which is continuously concerned by the service system in real time, and is also a message processing flow which needs key guarantee on the side of the internet of things management platform.
Step two: caching the terminal equipment data according to the divided professional write message queue;
the data of the terminal equipment is also required to be preprocessed before being stored in the message queue, and the preprocessing comprises adding a message header of the IOT management platform.
It should be noted that before caching the terminal device data, professional division needs to be performed according to the unique device identifier in the interaction between the internet of things management platform and the edge device cloud edge. Take the specialty in the power field as an example: the power transmission 10, the transformation 11, the power distribution 12 and the power utilization 13, wherein the power transmission, the transformation, the power distribution and the power utilization represent professional types, and the equipment number sections are corresponding professional equipment number sections which can be customized according to needs, wherein the equipment number sections are 10, 11, 12 and 13.
Step three: and matching the terminal equipment data written into the message queue with consumers according to the preset Cartesian product of the number of the message topic fragments and the number of the message topic fragments to perform message consumption, and distributing the message consumption to each service system.
Suppose that: setting the message topic fragment number segment to be 2 and the message topic partition number partition to be 3, and then the cloud edge interaction topic and the information queue topic of the internet of things management platform are correspondingly shown in table 1:
table 1:
according to the number of the fragments and the number of the partitions set in table 1, the message processing service of the internet of things management platform can be extended to 2 × 3 — 6 consumers to concurrently process the messages of the message queue, so that the message processing performance is improved. Taking transmission and transformation specialties in table 1 as an example, an internet of things management platform topic shown in table 2 can be established:
table 2:
example three:
an embodiment of the present invention further provides a service data distribution system, as shown in fig. 1, including an internet of things management platform, a plurality of edge devices, and a service system. The internet of things management platform according to the first embodiment may be adopted, and specifically includes: the device comprises a connecting unit, a message queue and a data distribution unit which are connected in sequence.
The edge device mainly refers to a switch, a router, a routing switch, an IAD and various MAN/WAN devices installed on an edge network, and is used for realizing data transmission. The edge device can be provided with a plurality of corresponding connecting units, the corresponding connecting units and the edge device are configured according to the same number, and the edge device can report the terminal device data to the Internet of things management platform through the connecting units.
And the service system is used for receiving the message distributed by the data distribution unit and executing related processing tasks according to the message.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.
Claims (8)
1. A method for distributing service data, the method comprising:
receiving terminal equipment data reported by each edge equipment through cloud edge interaction topic; the terminal equipment data comprises management data and service data;
caching the terminal equipment data according to the divided professional write message queue;
and matching the terminal equipment data written into the message queue with consumers according to the preset Cartesian product of the number of the message topic fragments and the number of the message topic fragments to perform message consumption, and distributing the message consumption to each service system.
2. The business data distribution method of claim 1, wherein the specialties are divided according to unique device identifiers in the interaction between the IOT management platform and the cloud edge of the edge device.
3. The service data distribution method according to claim 1, wherein the IOT platform establishes a long connection with the edge device based on an MQTT protocol before receiving the terminal device data reported by the edge device.
4. The service data distribution method according to claim 1, wherein before writing the terminal device data into the message queue for buffering, the terminal device data is preprocessed, and the preprocessing includes: and adding message header information of the Internet of things management platform.
5. The service data distribution method according to claim 1, wherein the management data includes command issuing and response, request and response, and is used for implementing real-time command interaction or service processing between the internet of things management platform and the edge device; the service data includes event data and state data of the terminal device.
6. An internet of things management platform, comprising:
a connection unit: the cloud-side interaction topic is used for receiving terminal equipment data reported by the edge equipment through the cloud-side interaction topic and writing the received terminal equipment data into the message queue according to the divided professional topic; the terminal equipment data comprises management data and service data;
message queue: the data storage module is used for writing and caching the terminal equipment data;
a data distribution unit: and the system is used for matching the messages written into the message queue with consumers according to the preset Cartesian product of the number of the message topic fragments and the number of the message topic fragments to perform message consumption and distributing the messages to each service system.
7. The IOT management platform of claim 6, wherein the connection unit is further configured to establish a long connection with the edge device based on an MQTT protocol before receiving the terminal device data reported by the edge device.
8. A business data distribution system comprising the internet of things management platform of claim 6 or 7, a plurality of edge devices and a business system;
the edge device is used for reporting terminal device data through the connection unit;
and the service system is used for receiving the message distributed by the data distribution unit and executing related processing tasks according to the message.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111489684.2A CN114401269A (en) | 2021-12-08 | 2021-12-08 | Business data distribution method and system and Internet of things management platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111489684.2A CN114401269A (en) | 2021-12-08 | 2021-12-08 | Business data distribution method and system and Internet of things management platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114401269A true CN114401269A (en) | 2022-04-26 |
Family
ID=81226268
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111489684.2A Pending CN114401269A (en) | 2021-12-08 | 2021-12-08 | Business data distribution method and system and Internet of things management platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114401269A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115913809A (en) * | 2022-09-26 | 2023-04-04 | 重庆长安汽车股份有限公司 | Data distribution communication method, system, computer device and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140372486A1 (en) * | 2013-06-12 | 2014-12-18 | Oracle International Corporation | In-Database Sharded Queue |
| CN109451072A (en) * | 2018-12-29 | 2019-03-08 | 广东电网有限责任公司 | A kind of message caching system and method based on Kafka |
| CN110019240A (en) * | 2017-12-29 | 2019-07-16 | 中国移动通信集团北京有限公司 | A kind of service data interaction method, apparatus and system |
| CN110162410A (en) * | 2018-02-12 | 2019-08-23 | 北京京东尚科信息技术有限公司 | A kind of message treatment method and device |
| CN110618860A (en) * | 2019-09-09 | 2019-12-27 | 中盈优创资讯科技有限公司 | Spark-based Kafka consumption concurrent processing method and device |
| CN110648178A (en) * | 2019-09-24 | 2020-01-03 | 四川长虹电器股份有限公司 | Method for increasing kafka consumption capacity |
| CN111158931A (en) * | 2019-12-30 | 2020-05-15 | 联想(北京)有限公司 | Dynamic data balancing method and device for Kafka partition and storage medium |
| CN111432295A (en) * | 2020-03-18 | 2020-07-17 | 北京科东电力控制系统有限责任公司 | Power consumption information acquisition master station system based on distributed technology |
| CN112261080A (en) * | 2020-09-14 | 2021-01-22 | 国网江苏省电力有限公司信息通信分公司 | Edge Internet of things agent method applied to power Internet of things |
| CN112671934A (en) * | 2021-03-15 | 2021-04-16 | 国网江苏省电力有限公司信息通信分公司 | Electric power internet of things system |
-
2021
- 2021-12-08 CN CN202111489684.2A patent/CN114401269A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140372486A1 (en) * | 2013-06-12 | 2014-12-18 | Oracle International Corporation | In-Database Sharded Queue |
| CN110019240A (en) * | 2017-12-29 | 2019-07-16 | 中国移动通信集团北京有限公司 | A kind of service data interaction method, apparatus and system |
| CN110162410A (en) * | 2018-02-12 | 2019-08-23 | 北京京东尚科信息技术有限公司 | A kind of message treatment method and device |
| CN109451072A (en) * | 2018-12-29 | 2019-03-08 | 广东电网有限责任公司 | A kind of message caching system and method based on Kafka |
| CN110618860A (en) * | 2019-09-09 | 2019-12-27 | 中盈优创资讯科技有限公司 | Spark-based Kafka consumption concurrent processing method and device |
| CN110648178A (en) * | 2019-09-24 | 2020-01-03 | 四川长虹电器股份有限公司 | Method for increasing kafka consumption capacity |
| CN111158931A (en) * | 2019-12-30 | 2020-05-15 | 联想(北京)有限公司 | Dynamic data balancing method and device for Kafka partition and storage medium |
| CN111432295A (en) * | 2020-03-18 | 2020-07-17 | 北京科东电力控制系统有限责任公司 | Power consumption information acquisition master station system based on distributed technology |
| CN112261080A (en) * | 2020-09-14 | 2021-01-22 | 国网江苏省电力有限公司信息通信分公司 | Edge Internet of things agent method applied to power Internet of things |
| CN112671934A (en) * | 2021-03-15 | 2021-04-16 | 国网江苏省电力有限公司信息通信分公司 | Electric power internet of things system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115913809A (en) * | 2022-09-26 | 2023-04-04 | 重庆长安汽车股份有限公司 | Data distribution communication method, system, computer device and storage medium |
| CN115913809B (en) * | 2022-09-26 | 2024-05-03 | 重庆长安汽车股份有限公司 | Data distribution communication method, system, computer device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100470517C (en) | System and method for information handling system PCI express advanced switching | |
| CN102480426A (en) | Communication method based on peripheral component interconnect-express (PCIE) switching bus and PCIE switching system | |
| CN100456693C (en) | Data transmitting system used in electronic equipment with multiple service unit | |
| CN101610216A (en) | Ethernet message forwarding method, Apparatus and system | |
| CN114401269A (en) | Business data distribution method and system and Internet of things management platform | |
| CN107086937A (en) | A kind of monitoring method and equipment | |
| CN112770358B (en) | Multi-rate mode data transmission control method and device based on service data | |
| CN213904561U (en) | Automatic edge calculation node system of parking | |
| CN105897653A (en) | High-performance mobile device framework | |
| CN205003725U (en) | Classroom intelligence attendance management system | |
| CN110309031B (en) | Load balancing micro-computing cluster architecture | |
| CN203537418U (en) | Electric-network cloud computing management system | |
| CN111782322A (en) | Intranet and extranet message communication server and system based on cloud desktop server | |
| CN111835574A (en) | SDN-based virtualized data center network management method and device | |
| CN111541667A (en) | Method, equipment and storage medium for intersystem message communication | |
| CN111190746A (en) | VxWorks 653-based multi-core partition real-time operating system and communication method | |
| CN115225484B (en) | Management system of DPU networking | |
| CN202353605U (en) | Macro control system for cloud equipment | |
| CN104052665A (en) | Method and equipment for determining flow forwarding path | |
| CN113282431B (en) | Abnormal data processing method and device, storage medium and electronic equipment | |
| CN112751756B (en) | Data forwarding method, device, equipment and computer readable storage medium | |
| CN113849508A (en) | Data storage method, device, equipment and medium | |
| CN117749606A (en) | Power switch abnormal message real-time recording system and method | |
| CN103139238A (en) | Distributed multi-dimensional outdoor server of application plug-in based on active data push | |
| CN111314186A (en) | Communication method, system, computer equipment and medium for accessing superior command post |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220426 |
|
| RJ01 | Rejection of invention patent application after publication |