CN116193295A - Multi-element Internet of things data fusion and unified data processing method - Google Patents

Abstract

Embodiments of the present disclosure provide a data processing method for fusion and unification of multiple IoT data. The method includes the adaptive collection of sensing device data by the Internet of Things gateway, and matching and filtering of the collected data; the data fusion of the matched and filtered data by the Internet of Things gateway, and real-time analysis and calculation and persistent data access; The networking gateway transmits the fused data to the IoT cloud platform through a unified application layer protocol. In this way, the unified scheduling and processing of multi-source heterogeneous data can be realized through the cooperative control of active distribution network integrated edge agents.

Description

A data processing method for multi-item data fusion and unification

technical field

The present disclosure relates to the field of the Internet of Things, and in particular to the technical field of multi-IoT data fusion and unified data processing.

Background technique

With the rapid development of Internet of Things technology, the demand for data collection of various intelligent terminals is constantly increasing. Traditional IoT gateway program development is mainly for specific business scenarios and data protocols. Due to the different data transmission protocols and formats of various smart devices, if the device, protocol or business scenario changes, the corresponding gateway program needs to be changed to make business changes, and the data collection of each device needs targeted adaptation development. Acquisition equipment cannot be reused, resulting in the complexity of secondary development; the development cycle and development difficulty are relatively large. The increase in data collection will also increase the data transmission bandwidth and increase the pressure on the cloud.

Contents of the invention

The present disclosure provides a data processing method for merging and unifying multiple Internet of Things data.

According to the first aspect of the present disclosure, a data processing method for merging and unifying multiple IoT data is provided. The method includes: the Internet of Things gateway adaptively collects sensing device data, and performs matching and filtering on the collected data; the Internet of Things gateway performs data fusion on the matched and filtered data, and performs real-time analysis and calculation and persistent data access; The networking gateway transmits the fused data to the IoT cloud platform through a unified application layer protocol.

According to the above-mentioned aspect and any possible implementation mode, an implementation mode is further provided, wherein the self-adaptive collection of sensing device data by the IoT gateway includes: collecting various types of equipment in real time according to the data collection tasks issued by the IoT cloud platform data.

According to the above-mentioned aspect and any possible implementation mode, an implementation mode is further provided. The data fusion performed by the IoT gateway on the matched and filtered data includes: Static model, realize collection model fusion and upload model fusion; real-time analysis calculation and data persistent access include: IoT gateway executes service algorithm according to the equipment and data service topics concerned by the algorithm subscribed by IoT cloud platform, and completes data Analysis and association processing; according to the equipment and data service topics subscribed by the Internet of Things cloud platform, perform corresponding storage/retrieval services for the fused data.

According to the above-mentioned aspect and any possible implementation mode, an implementation mode is further provided. The IoT gateway transmits the fused data to the IoT cloud platform through a unified application layer protocol, including: the IoT gateway subscribes according to the IoT cloud platform Configured devices and data upload service topics, obtain corresponding upload model data, and perform protocol conversion and upload.

According to the above aspect and any possible implementation manner, an implementation manner is further provided, wherein the protocol conversion includes: performing adaptive conversion of the protocol for the uploaded model data.

According to the above aspect and any possible implementation, an implementation is further provided, the sensing device performs equipment encoding and data encoding according to the unified preset equipment encoding and data encoding specifications; the model is based on the preset data Modeling specification modeling.

According to the above-mentioned aspect and any possible implementation, an implementation is further provided, the model is downloaded to the gateway of the Internet of Things after the training of the cloud platform of the Internet of Things is completed; the gateway of the Internet of Things uses real-time data to update the model Follow up training.

According to the foregoing aspect and any possible implementation manner, an implementation manner is further provided, the device code is a device-aware address, and the data code includes a data body and a data item.

According to the above aspect and any possible implementation, an implementation is further provided, the method further includes: modeling the object model of the IoT device; the object model includes basic information and device services of the IoT device information.

According to the above aspect and any possible implementation, an implementation is further provided, the method further includes: establishing an association relationship between the monitoring object of the object model and the unique ID of the grid resource of the holographic grid model, and linking the Internet of Things The physical model of the equipment is fused with the holographic model of the power grid.

It should be understood that what is described in the Summary of the Invention is not intended to limit the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood through the following description.

Description of drawings

The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description when taken in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution, and do not constitute a limitation to the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

FIG. 1 shows a schematic diagram of an exemplary operating environment in which embodiments of the present disclosure can be implemented;

FIG. 2 shows a schematic diagram of a data processing method in which multiple Internet of Things data fusion and unification according to an embodiment of the present disclosure can be implemented;

FIG. 3 shows a schematic diagram of a structure framework of an object model according to an embodiment of the present disclosure;

FIG. 4 shows a schematic diagram of an analysis and calculation of an Internet of Things gateway according to an embodiment of the present disclosure;

Fig. 5 shows a schematic diagram of a data model cloud-edge collaboration mechanism according to an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments It is a part of the embodiments of the present disclosure, but not all of them. Based on the embodiments in the present disclosure, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present disclosure.

In addition, the term "and/or" in this article is only an association relationship describing associated objects, which means that there may be three relationships, for example, A and/or B may mean: A exists alone, A and B exist at the same time, There are three cases of B alone. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship.

FIG. 1 shows a schematic diagram of an exemplary operating environment 100 in which embodiments of the present disclosure can be implemented;

In some embodiments, the edge proxy terminal (Internet of Things gateway) uniformly transmits the data of each sensor device through a unified transmission protocol, and uploads it to the Internet of Things cloud platform for processing by the digital twin system platform.

In some embodiments, the edge proxy terminal (Internet of Things gateway) sends the control instructions issued by the cloud platform to photovoltaics, energy storage, adjustable loads, and power loads, and implements adjustment, control, and integration for the macro-control of the entire station area. Multiple ports.

FIG. 2 shows a schematic diagram of a data processing method 200 in which multiple IoT data fusion and unification according to embodiments of the present disclosure can be implemented.

In block 210, the IoT gateway adaptively collects sensing device data, and performs matching filtering on the collected data;

In some embodiments, the IoT gateway device communicates with the sensing device (smart sensor) based on a preset communication definition specification to collect data from the sensing device.

In some embodiments, taking power Internet of Things devices as an example, the sensing devices include charging piles, air source air conditioners, photovoltaics, temperature and humidity sensors, water immersion sensors, smoke sensors, video ball machines, intelligent track robots, power distribution Voltage switchgear partial discharge monitoring sensor, infrared imaging monitoring device, transformer temperature sensor, etc. The sensing devices form a sensing network.

The main access methods can be divided into wired access and wireless access.

In some embodiments, since the collected sensing information is encapsulated in different protocol encapsulation forms in different sensing networks, this will lead to soft isolation of data between different sensing networks. On the one hand, sensing networks that are physically connected to each other On the other hand, the perception network cannot complete the communication with the core switching network, resulting in the remote access to the perception data. Therefore, the smart gateway of the Internet of Things needs to have the ability to communicate with heterogeneous networks, as well as the ability to monitor, control and manage The ability to be compatible with new node access; at the same time, all intelligent gateway nodes are required to use standardized communication methods. In summary, combined with the application scenarios of the Internet of Things, in order to standardize the address allocation of sensor devices connected to the Internet of Things and unify the communication between the smart gateway device and the downlink device, the embodiment of the present disclosure adopts the method suitable for smart sensors and the Internet of Things gateway device. Communication definition rules between.

In some embodiments, the uniqueness of IoT device coding and the standardization of data coding are ensured according to the unified IoT device data coding specification, providing support for asset management and data modeling of the digital twin platform. Among them, the data coding of IoT devices is mainly divided into two parts: device coding and data coding. The coding objects include: IoT devices, including IoT gateway devices and sensing devices. The coding system is as follows: Device coding: mainly codes IoT devices , which is used to identify the unique identity code of the device in the digital twin platform. Data coding: mainly to standardize and code the data capabilities of IoT devices, including data subject and data item definitions. Data topics are mainly used for data classification to identify different data types of IoT devices. For example, the data topics of electric energy meters can be divided into operating data, meter reading data, and alarm data. A data item is the smallest granular data description of an IoT device. It is used to describe a specific data of an IoT device, including data name, data description, data type, data length, etc., similar to a database data dictionary.

In some embodiments, the device code can be the address of the sensing device, that is, the address of the sensor device; the address of the sensor device is assigned according to the sensor manufacturer, category, and function. The conventional configuration of the intelligent gateway provides multiple RS485 serial ports and network ports, which can be freely adapted. The sensors used in the networking are uniformly connected to the intelligent gateway device through the RS485 serial port or network port. Among them, the address range of the RS485 communication sensor device should be in the range of 1 to 128. The principle of address allocation is to allocate different addresses according to different manufacturers, as shown in Table 1.

Table 1 Distribution Internet of Things sensor address allocation table

Note: The default address refers to the initial address of the device, which will be modified according to the actual access situation in actual use.

In some embodiments, the object model of the above-mentioned Internet of Things device is integrated with the holographic model of the power grid, such as SG-CIM (State Grid Enterprise Common Information Model), to realize the integration of the Internet of Things device and the primary equipment of the power grid, the object model and the holographic grid model Data fusion provides basic support for grid business applications to use IoT device data. .

In some embodiments, the object model, that is, a data model defining feature information and service capabilities of IoT devices, includes basic device information, device services (attributes, events, commands) and other information. The equipment encoding and data capability attributes of the physical model follow the encoding rules of the "Power Internet of Things Equipment Data Encoding Specification", as shown in Figure 3 of the attached drawing.

In some embodiments, object model modeling of IoT devices is performed using an object-oriented approach. Through object model modeling, the digital twin platform can quickly and comprehensively identify IoT devices and their service capabilities, laying the foundation for the rapid access of IoT devices.

In some embodiments, based on the production system ledger, the SG-CIM (that is, the State Grid Enterprise Common Information Model) modeling specification is adopted to construct a holographic grid that focuses on power primary equipment and covers the fields of power generation, power transmission, power distribution, and power consumption Model. Through the establishment of an association relationship between the monitoring object of the physical model and the unique ID of the grid resource of the holographic grid model, the data fusion of the IoT device and the primary grid equipment, the physical model and the holographic grid model is realized.

Based on the equipment ledger of the production system ledger of the power grid, sort out the equipment category and functional location structure of the main distribution network primary equipment; then sort out and establish the corresponding relationship between the equipment category of the production ledger and the SGCIM model of the power grid (such as the production ledger The transformer is mapped to the Transformer object of the SGCIM model of the power grid), and at the same time, the data information necessary for holographic modeling (such as function location number, superior function location number, etc.) is sorted out from the ledger equipment; the sorted out primary distribution network primary equipment function structure And data information, according to the SG-CIM modeling specification and the mapping relationship between the production ledger equipment category and the power grid SG-CIM model, a unified holographic power grid model is generated. When modeling, the "rdf:ID" (that is, the unique ID of grid resources) in the holographic grid model should be consistent with the functional location code of the production ledger. In the basic equipment information of the IoT equipment object model, the monitoring object and monitoring location information are expanded to describe the primary equipment and its location information monitored by the IoT equipment. Through the establishment of an association relationship between the monitoring object of the physical model and the "rdf:ID" of the holographic grid model (that is, the unique ID of the grid resource), the data fusion between the IoT device and the primary equipment of the grid, and the physical model and the holographic grid model is realized. Use IoT device data to provide basic support.

In some embodiments, the IoT gateway device performs matching filtering on the collected data for later edge computing; wherein the matching filtering includes:

String matching parsing: due to the self-configuration of sensing devices, matching parsing is required. Judging whether the collected data matches, if so, analyze the matching mode; if not, analyze the non-matching mode; finally analyze to get the real data. Wherein, the parsing is character string matching, in order to facilitate data input and reading, byte matching is used to parse required bytes. It is mainly divided into matching mode and non-matching mode. The matching mode is all matches of a box of data frames, such as 0203 0405 06; the non-matching mode is mainly used for long data frames, which can be extracted by moving the byte number of data bytes .

Edge data filtering: Edge data filtering mainly uses two methods: incremental threshold and maximum and minimum values. Make a difference between the data obtained this time and the data obtained once, and calculate the percentage of the difference in this collection value, keep the data when it is greater than the incremental threshold, otherwise discard the data; use the comparison method for the maximum and minimum values, and analyze the converted results Compare the preset maximum and minimum values to determine whether the data is retained. In order to upload the retained data to the Internet of Things cloud platform (power distribution networking platform).

In some embodiments, according to the southbound device data model, data collection, analysis and calculation, and data access are all initiated by the IoT cloud platform (power distribution networking platform) to the IoT gateway (edge data center) in the form of services Requests and Subscriptions. Among them, the collection service collects various equipment data in real time, and releases data collection tasks to the edge data center in real time. The edge computing service subscribes to the edge-side data center for the equipment and data service topics that the algorithm cares about, so that the edge-side data center can execute the service algorithm and complete data analysis and association processing. Data persistence storage/retrieval service, subscribe to the configured device and data service topics to the edge data center, so that the edge data center can perform corresponding storage/retrieval services. Data upload service, subscribe to the configured device and data upload service topic to the edge data center, so that the edge data center can obtain the corresponding upload model data, and perform protocol conversion and upload.

In some embodiments, the Internet of Things gateway collects various types of device data in real time according to the data collection task issued by the Internet of Things cloud platform (power distribution networking platform).

In some embodiments, the adaptive acquisition is the adaptability of the acquisition interval, that is, the data is preprocessed on the edge side, targeted data filtering is performed, and the acquisition interval is adaptively adjusted, while reducing the amount of data, it will not cause critical data loss.

In block 220, the Internet of Things gateway performs data fusion on the matched and filtered data, and performs real-time analysis calculation and data persistent access;

In some embodiments, as shown in FIG. 4 , the IoT gateway performs data fusion on the matched and filtered data, and performs real-time analysis calculation and persistent data access.

In some embodiments, the Internet of Things gateway realizes collection model fusion and upload model fusion according to the static model issued by the data model service subscribed by the Internet of Things cloud platform (power distribution networking platform).

In some embodiments, the Internet of Things gateway executes the service algorithm and completes data analysis and association processing according to the equipment and data service topics concerned by the algorithm subscribed by the Internet of Things cloud platform (distribution networking platform); Take calculation data and historical data for real-time analysis and calculation, and obtain calculation result data.

In some embodiments, as shown in FIG. 5 , a schematic diagram of the cloud-edge collaboration mechanism of the data model, the calculation model of the IoT gateway is based on the cloud-edge collaboration technology, and is downloaded to the IoT gateway after being trained by the IoT cloud platform. While performing computing tasks, the IoT gateway will also use the data collected in real time to further train the model, so as to send high-level early warning data to the IoT cloud platform first, which solves the pressure problem of the IoT cloud platform and reduces The amount of data transmitted by the network ensures that the early warning data can be sent quickly. Through collaboration with the cloud data center, an iterative update mechanism for the existing model (active distribution network application scenario model) on the IoT gateway (edge side) is established to realize efficient linkage of cloud-side data flow and consistent data services.

In some embodiments, the Internet of Things gateway performs corresponding storage/retrieval services for the fused data according to the equipment and data service topics subscribed by the Internet of Things cloud platform (power distribution networking platform); including depositing historical data and retrieving historical data .

In block 230, the IoT gateway transmits the fused data to the IoT cloud platform through a unified application layer protocol;

In some embodiments, the Internet of Things gateway subscribes to the configured device and data upload service topics according to the Internet of Things cloud platform (power distribution networking platform), obtains the corresponding upload model data, and performs protocol conversion and upload.

In some embodiments, the Internet of Things gateway (intelligent gateway for power distribution Internet of Things equipment) is uniformly connected to the Internet of Things cloud platform (power distribution networking platform) using the MQTT protocol, and follows the preset equipment coding, data coding and data modeling specifications . Among them, the preset data modeling specification includes: when modeling, the subject and load should adopt the format. Modeling takes sensors (such as Internet of Things gateways, smoke sensors, temperature sensors, etc.) as objects, and try to use data items with similar purposes and data items with the same collection frequency as much as possible. Modeling should use the same theme to report data. Wherein, the equipment encoding and data encoding are as described above, and will not be repeated here.

According to the embodiments of the present disclosure, based on cloud-edge collaboration, starting from the hardware resources, data format, communication interface, security protocol, transmission throughput rate, etc. of the IoT gateway (edge proxy device) and the IoT cloud platform (intelligent perception cloud) , realizing the data processing between the edge proxy device and the intelligent perception cloud, reducing the amount of data transmitted by the network, improving the computing efficiency of cloud computing, and collaboratively completing user business needs. Through collaborative caching of data, rapid response, and realization of " Perceived autonomy" enhances the real-time nature of collaborative work.

According to the embodiments of the present disclosure, the following technical effects are achieved:

Aiming at the problems of different manufacturers and different models of sensing layer equipment with inconsistent specifications, difficult access, and time-consuming protocol analysis, a unified equipment data model is established to realize a unified equipment model, unified coding specification, and unified communication protocol to meet the needs of distribution network scenarios. application.

The Internet of Things gateway supports automatic switching of various communication protocols of electric power, intelligently adapts to various interface types of electric power Internet of Things terminals, and supports fast and flexible access to various Internet of Things devices.

For the access and forwarding of terminal equipment data in the active distribution network, the connection between the cloud platform of the Internet of Things and energy storage, photovoltaic and other subsystems is realized, and the data acquisition and processing and control strategies of the active distribution network are transferred to the Internet of Things gateway. Issued.

It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the described action sequence. Because of this disclosure, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, each step described in the present disclosure may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in the present disclosure can be achieved, no limitation is imposed herein.

The specific implementation manners described above do not limit the protection scope of the present disclosure. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims (10)

1. A multi-element Internet of things data fusion and unified data processing method comprises the following steps:

the gateway of the Internet of things adaptively collects sensing equipment data and performs matching filtering on the collected data;

the gateway of the Internet of things performs data fusion on the matched and filtered data, and performs real-time analysis and calculation and data persistence access;

and the gateway of the Internet of things transmits the fused data to the cloud platform of the Internet of things through a unified application layer protocol.

2. The method of claim 1, wherein the internet of things gateway adaptively collecting sensing device data comprises: and acquiring various device data in real time according to the data acquisition task issued by the cloud platform of the Internet of things.

3. The method of claim 1, wherein data fusion of matching filtered data by the internet of things gateway comprises:

the gateway of the Internet of things realizes acquisition model fusion and uploading model fusion according to a static model issued by a data model service subscribed by the cloud platform of the Internet of things;

real-time analytics computing and data persistence access includes:

the gateway of the Internet of things executes a service algorithm according to the equipment and the data service subject concerned by the algorithm subscribed by the cloud platform of the Internet of things, and completes data analysis and association processing; and executing corresponding access/fetch service on the fused data according to the equipment subscribed by the Internet of things cloud platform and the data service theme.

4. The method of claim 1, wherein the internet of things gateway transmitting the fused data to the internet of things cloud platform through a unified application layer protocol comprises:

and the Internet of things gateway subscribes the configured equipment and the data uploading service theme according to the Internet of things cloud platform, acquires corresponding uploading model data and performs protocol conversion uploading.

5. The method of claim 4, wherein the protocol conversion comprises:

and carrying out protocol self-adaptive conversion of the uploading model data.

6. The method according to claim 3 or 4, wherein the method further comprises:

carrying out Internet of things equipment coding and data coding according to unified preset equipment coding and data coding specifications; the model is modeled according to a preset data modeling specification.

7. The method of claim 6, wherein the model is downloaded to an internet of things gateway after training by an internet of things cloud platform is completed; the internet of things gateway utilizes real-time data to train the model subsequently.

8. The method of claim 6, wherein the device encoding is a perceiving device address, the data encoding comprising a data body and a data item.

9. The method of claim 6, wherein the method further comprises:

modeling an object model of the Internet of things equipment; the object model comprises basic information and equipment service information of the Internet of things equipment.

10. The method of claim 9, wherein the method further comprises:

and establishing an association relation between the monitored object of the object model and the unique ID of the power grid resource of the holographic power grid model, and fusing the object model of the Internet of things equipment with the power grid holographic model.

Images