CN117648379A - Method and device for collecting and synchronizing time sequence data of different protocols - Google Patents

Abstract

The invention provides a method and a device for collecting and synchronizing time sequence data of different protocols, which are applied to a client, wherein the method comprises the following steps: initializing each device based on metadata of each of one or more devices, and generating a virtual shadow device for each device, wherein the metadata comprises description information of a protocol, a product and the device, and the product is represented by a digitized object model; determining corresponding virtual shadow equipment according to the received acquisition instruction, and enabling the corresponding virtual shadow equipment to acquire time sequence data from the designated equipment using the corresponding protocol through a predefined interface; based on the object model, converting the time sequence data into a predefined data format, packaging by utilizing an MQTT protocol and transmitting to a time sequence database; and synchronizing the time sequence data to a cloud database through a predefined database interface. In the invention, protocol compatibility is increased in data acquisition and synchronization, heterogeneous database synchronization capability is realized, cloud edge synchronization efficiency is improved, and data transmission required by cloud edge data synchronization is reduced.

Description

Method and device for collecting and synchronizing time sequence data of different protocols

Technical Field

The present invention relates to the field of data processing, and in particular, to a method and apparatus for collecting and synchronizing time series data of different protocols, and a computing device cluster, a computer program product, and a medium.

Background

In current industrial production, data acquisition has become a very important part. Various data generated in the production process are collected, so that the production process can be comprehensively monitored and managed, the production process is further optimized, the production efficiency is improved, and the production cost is reduced. However, in practical applications, industrial data collection has many bottlenecks and challenges, such as low data collection efficiency, incompatibility of different industrial protocols, and the like, due to the diversity of industrial protocols and the isomerism of data collection devices. Also, because different protocols have different requirements and limitations, the edge nodes may need to have different processing capabilities and hardware devices, which further increases the cost and complexity of the system.

Therefore, searching for a more flexible and efficient edge collection mode is a problem to be solved in the current edge computing field. With the popularization of cloud computing technology and the wide application of cloud services, cloud computing resources are more abundant than edge nodes, more and more enterprises and organizations select to store data of the cloud computing technology in a cloud database, and analysis and calculation are performed in the cloud, so that cloud edge synchronization is needed for the database. Database cloud edge synchronization refers to keeping a cloud database in synchronization with a local database on edge equipment so as to ensure consistency and instantaneity of data. However, data synchronization between cloud and local databases remains a challenging problem. There is a need to address some of the issues such as heterogeneous database synchronization, data loss, data inconsistencies, network delays, etc.

The existing industrial acquisition scene has a plurality of different protocols, the existing solution strategies cannot well consider multi-protocol expansibility and acquisition performance, and the consistency of the universality of the edge node information acquisition and cloud edge data synchronization is supported in the absence of a proper model. The computational performance of the edge nodes is not well exploited. And with the continuous development of time sequence databases in recent years, a batch of databases suitable for different scenes are emerging. Selecting a proper bottom time sequence database for different business scenes is a more general and product competitive method. Therefore, cloud edge node heterogeneous databases are synchronized, great challenges are made for high-performance cloud edge database synchronization, and the existing cloud edge synchronization technology cannot meet the requirements of high-performance cloud edge heterogeneous database synchronization.

Disclosure of Invention

The invention aims to provide a method and a device for collecting and synchronizing time sequence data of different protocols, and a computing device cluster, a computer program product and a medium.

The first aspect of the invention discloses a method for collecting and synchronizing time sequence data of different protocols, which is applied to a client, and comprises the following steps:

Initializing each device based on metadata of one or more devices communicated with the client, and generating virtual shadow devices for each device, wherein the metadata comprises the devices, protocols corresponding to the devices and descriptive information of products, and the products are represented by a digitalized object model;

a collection step of determining virtual shadow equipment corresponding to designated equipment in a collection instruction according to the received collection instruction, and enabling the corresponding virtual shadow equipment to collect the time sequence data from the designated equipment using a corresponding protocol through a predefined interface;

a transmission step, based on the object model, converting the time sequence data into a predefined data format, and transmitting the time sequence data to a time sequence database in the client after packaging by utilizing an MQTT protocol;

and synchronizing, by a predefined database interface of the time sequence database, the time sequence data to a cloud database in communication with the client.

In a possible implementation of the above first aspect, the object model defines properties, methods and events of the products, each product comprising a corresponding one or more devices,

Wherein the metadata includes descriptive information of the protocol, the device, and attributes, methods, events of the product.

In a possible implementation of the above first aspect, the predefined interface is defined based on an operation of the object model, the predefined interface comprising a plurality of methods implemented by the virtual shadow device,

in the collecting step, the corresponding virtual shadow device accesses the designated device using the corresponding protocol through the methods to collect the time sequence data.

In a possible implementation of the first aspect, in the transmitting step, the time series data is converted into a predefined data format based on the description information of the attribute, the method and the event of the product, and the time series data in the predefined data format and the association information related to the time series data are packaged into a message in MQTT format and then transmitted to the time series database.

In a possible implementation of the first aspect, the predefined database interface is defined based on the metadata, the predefined database interface being compatible with different timing databases.

In a possible implementation manner of the first aspect, in the synchronizing step, after the time-series data in the time-series database is converted into a preset data format through the predefined database interface, the time-series data is synchronized to the cloud database.

In a possible implementation of the first aspect, after the transmitting step, the method further includes a downsampling step for downsampling the time series data in the time series database based on a dynamic downsampling frequency.

In a possible implementation of the first aspect, in the step of downsampling, an outlier detection algorithm is used to detect an outlier segment in the time-series database, downsampling is performed according to an outlier downsampling frequency when the outlier segment is detected, and downsampling is performed according to a default downsampling frequency when the outlier segment is not detected.

In a possible implementation of the first aspect described above, the client comprises a device driver service for implementing the initializing step and a device management service for implementing the collecting step, the transmitting step, and the synchronizing step,

When a device change instruction is received, the device change instruction is packaged into an MQTT format message and then transmitted to the device driving service, so that the device driving service operates according to the device change instruction.

The second aspect of the present invention discloses a device for collecting and synchronizing time sequence data of different protocols, applied to a client, the device comprising:

an initializing unit, which is used for initializing each device based on metadata of one or more devices communicated with the client, and generating a virtual shadow device for each device, wherein the metadata comprises the device, a protocol corresponding to the device and a product, and the product is represented by a digitalized object model;

the acquisition unit is used for determining virtual shadow equipment corresponding to designated equipment in the acquisition instruction according to the received acquisition instruction, and enabling the corresponding virtual shadow equipment to acquire the time sequence data from the designated equipment using a corresponding protocol through a predefined interface;

the transmission unit is used for converting the time sequence data into a predefined data format based on the object model, packaging the time sequence data by utilizing an MQTT protocol and transmitting the packaged time sequence data to a time sequence database in the client;

And the synchronization unit is used for synchronizing the time sequence data to a cloud database communicated with the client through a predefined database interface of the time sequence database.

A third aspect of the invention discloses a cluster of computing devices, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of collecting and synchronizing time series data of different protocols according to any one of claims 1-9.

A fourth aspect of the invention discloses a computer program product comprising instructions which, when executed by a cluster of computing devices, cause the cluster of computing devices to perform a method of collecting and synchronizing time series data of different protocols according to the first aspect.

A fifth aspect of the invention discloses a computer readable storage medium comprising computer program instructions which, when executed by a cluster of computing devices, perform a method of collecting and synchronizing time series data of different protocols according to the first aspect.

Compared with the prior art, the embodiment of the invention has the main differences and effects that: in the invention, protocol compatibility is increased in data acquisition and synchronization; utilizing a predefined database interface to realize heterogeneous database synchronization capability; the Apache part is utilized to improve cloud edge synchronization efficiency and reduce bandwidth use of an edge end (edge node); the dynamic downsampling frequency is utilized for data preprocessing, and data transmission required by cloud edge data synchronization can be reduced to the greatest extent on the premise that analysis accuracy is not affected.

Drawings

Fig. 1 illustrates an application scenario diagram for collecting and synchronizing time series data of different protocols according to an embodiment of the present application.

Fig. 2 shows a flow chart of a method of collecting and synchronizing time series data of different protocols according to a first embodiment of the present application.

Fig. 3 shows a flow chart of a method of collecting and synchronizing time series data of different protocols according to a second embodiment of the present application.

Fig. 4 shows a block diagram of an apparatus for collecting and synchronizing time series data of different protocols according to an embodiment of the present application.

Fig. 5 shows a block diagram of a client according to an embodiment of the present application.

Fig. 6 shows a block diagram of a computing device according to an embodiment of the present application.

FIG. 7 illustrates a block diagram of a cluster of computing devices, according to an embodiment of the application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

In view of the fact that the expansibility of multiple protocols cannot be well considered in the prior art, the performance of collecting and synchronizing data is low, the application provides a method and a device for collecting and synchronizing time sequence data of different protocols, a computing device cluster, a computer program product and a medium.

FIG. 1 illustrates an application scenario diagram for acquisition and synchronization of time series data of different protocols according to an embodiment of the present invention. As shown in fig. 1, there are Edge nodes in industrial production, which are implemented by Edge-Device-SDK (EDS) 100, for example, the EDS100 communicates with a plurality of internet of things devices 101a, 101b, 101c through wired or wireless connection, and the EDS100 collects time series data of these internet of things devices and synchronizes the time series data to cloud database 1021 in cloud 102.

It is understood that the internet of things devices 101a, 101b, 101c are for illustration only and that the number of internet of things devices in communication with the EDS100 may be greater or lesser and is not limited.

In order to realize unified access of internet of things protocols used by different internet of things devices, the invention abstracts operation of the devices based on the object model, namely, the operation of the devices comprises a device driving service and a device management service, wherein the device driving service is used for realizing an initialization step, and the device management service is used for realizing an acquisition step, a transmission step, a downsampling step and a synchronization step. These steps will be described in detail hereinafter.

First, the terms involved in EDS100 are as follows:

1. metadata: the metadata includes a Protocol (Protocol), a Product (Product), and a Device (Device). Each device belongs to a product, wherein the protocol corresponds to the description information of a specific protocol, the product corresponds to the description information of a specific product, and the device corresponds to the description information of a specific device. In the metadata of each device, a protocol and a product correspond to the device.

2. Virtual shadow device: the physical devices (i.e. the internet of things devices) exist in the device driving service in a corresponding logic manner, and each virtual shadow device corresponds to one physical device and is responsible for establishing connection, health detection, read-write attribute, calling method, monitoring event and disconnection with the physical device.

3. Soft reading of equipment: read, read the attribute of virtual shadow equipment buffer memory.

4. Device hard reading: hardRead reads the attribute from the physical device through the virtual shadow device and updates the cache corresponding to the attribute in the virtual shadow device;

5. subscribing to device properties: and the subscore devices provide the expansion functions for the virtual shadow equipment, and the equipment is periodically read and pushed.

6. Subscribing to device events: the primary function provided by the physical device (part of the protocol may not be supported) is that the device pushes certain custom events when a threshold condition triggers.

Fig. 2 shows a flow chart of a method of collecting and synchronizing time series data of different protocols according to a first embodiment of the invention, applied to a client, for example implemented by EDS100 in fig. 1. As shown in fig. 2, the method includes:

an initializing step S200, initializing each device based on metadata of each of one or more devices in communication with the EDS100, and generating a virtual shadow device for each device, wherein the metadata includes the device, a protocol corresponding to the device, and a product, and wherein the product is represented by a digitized object model.

The devices are for example the internet of things devices 101a, 101b, 101c shown in fig. 1. Each device belongs to a product, such as a sensor, thermometer, on-board device, etc. In the present embodiment, for example, the device 101a is a thermometer 1 belonging to a thermometer product, the device 101b is a sensor 1 belonging to a sensor product, and the device 101c is a sensor 2 also belonging to a sensor product.

The object model is a digital representation of an entity (such as a sensor, a thermometer, an on-board device, a building, a factory and the like) in a physical space at the cloud, and three dimensions of a attribute, a method and an event describe what the entity can do, what can be done, and which information can be provided to the outside respectively. These three dimensions of the object model are defined, i.e. the definition of the product is completed. Wherein the attributes, methods, events are specifically defined as follows.

Attribute (property): for describing specific information and states of the device during operation, such as the current ambient temperature, intelligent lamp switch state, electric fan wind power level, etc., read by the environment monitoring device. Attributes can be classified into two types, read-write and read-only. The read-write type supports reading and setting attribute values, and the read-only type only supports reading attribute values;

method (method): refers to an instruction or method that a device may provide for external invocation. Input and output parameters may be set in the service call. The input parameters are parameters at the time of service execution, and the output parameters are results after service execution. In contrast to attributes, a service may implement more complex business logic, such as performing a particular task, through one instruction.

Event (event): the information for describing the operation of the device, which is actively reported to the cloud, generally includes information, alarms and faults that need to be perceived and processed externally. The event may contain a number of output parameters, which must be some "attribute". For example, notification information after a certain task is completed; temperature and time information when equipment fails; the operating state of the device when the device alarms, etc.

EDS100 defines metadata based on the concept of an object model, which includes devices (devices), protocols (protocols) corresponding to the devices, and products (products). As described above, the product is defined by the attributes, methods, and events, so the metadata of each device includes the device, the protocol corresponding to the device, and the description information of the attributes, methods, and events of the product corresponding to the device. Specific description information is shown in tables 1 to 6 below. Tables 1-6 form a unified data model of the present invention.

TABLE 1 protocol-description information defining a particular protocol

Table 2 products-description information defining a particular product, including attributes, methods, and events

Fields	Description of the invention
		ID	Product identification
Name	Product name
		Desc	Product description information
Creator	Product owners
		Protocol	Unique identification of protocol
DataFormat	Data format, optionally Json
		Properties	Product-supported Property list comprising one or more properties
Events	Product-supported Event list containing zero or more events
		Methods	Product-supported methods list containing zero or more methods

Table 3 device-the description information defining a specific device, which corresponds to the real internet of things device one to one, must belong to and only belongs to a certain product.

Table 4 attributes-descriptive information defining attributes of a product.

Fields	Description of the invention
		ID	Product attribute identification
Name	Product attribute names
		Desc	Product attribute description
Unit	Product attribute unit
		ReportMode	Product reporting mode, optional periodical, onchange
Interval	The reporting period of the product takes effect when the report mode is periodic
		FieldType	Product attribute types, optionally ' int ', ' uint ', ' float ', ' body ' and ' string
Writeable	Whether or not the product attributes are writable
		AuxProps	Product attribute extension configuration defined in protocol

Table 5 events-description information of events defining one product.

Fields	Description of the invention
		ID	Product event identification
Name	Product event name
		Desc	Product event description
Unit	Product attribute unit
		Outs	Product event attribute list, independent of product attributes
AuxProps	Product attribute extension configuration defined in protocol

Table 6 method-description information defining the method of a product.

Fields	Description of the invention
		ID	Product method identification
Name	Product method name
		Desc	Description of the product methods
Ins	Product method input list, independent of product attribute
		AuxProps	Product attribute extension configuration defined in protocol

Each device is initialized, i.e. the internet of things devices 101a, 101b, 101c are registered in the EDS100, respectively, as defined in tables 1-6. It will be appreciated that the protocols used by each device may be the same or different.

During the initialization process, a virtual shadow device is generated for each device. The virtual shadow devices are logically existing in the EDS100 corresponding to the internet of things device, and each virtual shadow device corresponds to one internet of things device and is responsible for establishing connection, health detection, read-write properties, calling methods, monitoring events, disconnection and the like with the internet of things device. As shown in fig. 1, for example, virtual shadow Va is generated for device 101a (i.e., thermometer 1), virtual shadow Vb is generated for device 101b (i.e., sensor 1), and virtual shadow Vc is generated for device 101c (i.e., sensor 2).

It will be appreciated that in this step, the internet of things devices using different protocols are respectively registered in the EDS100 in the manner defined in tables 1-6 so as to collect time series data from the registered devices. The generated virtual shadow devices are in one-to-one correspondence with the Internet of things devices.

In the collecting step S202, according to the received collecting instruction, a virtual shadow device corresponding to the designated device in the collecting instruction is determined, so that the corresponding virtual shadow device collects time sequence data from the designated device using the corresponding protocol through the predefined interface.

The user may enter an acquisition order, such as data of the reading device 101a, through the EDS 100. It will be appreciated that the designated device in the acquisition instruction is device 101a, and thus it can be determined that the virtual shadow device corresponding to device 101a is virtual shadow device Va.

In order to realize the capability of quickly accessing the protocols of different internet of things devices, a predefined interface is defined for each virtual shadow device based on some common operations of an object model, wherein the predefined interface comprises a plurality of methods realized by the virtual shadow device, and only the interface needs to be realized when devices of other protocols are accessed.

A specific description of these methods in the predefined interface is shown in table 7.

TABLE 7

The virtual shadow device Va accesses a specified device (for example, the device 101 a) using the corresponding protocol by these methods defined in table 7 to acquire (read) time series data of the device 101 a.

In a transmission step S204, the time series data is converted into a predefined data format based on the object model, and is packaged by the MQTT protocol and transmitted to the time series database 1002 (shown in fig. 1) in the EDS 100.

Specifically, the time series data is converted into a predefined data format based on descriptive information of attributes, methods and events of the product. Tables 4-6 show description information of attributes, methods and events of the products, and the collected time series data are converted into predefined data formats shown in tables 4-6 based on the definitions in tables 4-6.

The timing data in the predefined data format and the associated information related to the timing data are then encapsulated into a MQTT format message and transmitted to the timing database 1002.

The MQTT is an asynchronous communication message protocol constructed based on a TCP/IP protocol stack, and is a lightweight type release and subscription information transmission protocol. Originally developed by IBM in 1999, it was possible to extend in unreliable network environments, suitable for scenarios where device hardware storage space or network bandwidth is limited. Using the MQTT protocol, the message sender and receiver are not limited in time and space. The internet of things platform supports device access using MQTT protocol.

MQTT employs a publish/subscribe model, in which there are two main roles: publishers (publishers) and subscribers (subscribers). Publishers are responsible for sending messages to specific topics (topics), while subscribers receive messages by subscribing to topics of interest. This model allows devices to communicate asynchronously without directly establishing a point-to-point connection.

Theme (Topic): the subject is a message class identifier in the MQTT to distinguish between different types of messages. Subscribers may subscribe to one or more topics to receive messages related to those topics. Topics typically take a hierarchical structure, similar to a file system path. For example, "sensors/temperature" and "sensors/sensitivity" are two different topics.

In this embodiment, the message in MQTT format is Topic, and the general format of Topic is:

DATA/${Version}/${OptMode}/${ProtocolID}/${ProductID}/${DeviceID}/${FuncID}/${OptType}[/${ReqID}]`：

-Version: version of DATA Topic;

- 'OptMode': an operation mode, optionally down|up|up-ERR: 'DOWN': an operation initiated by a corresponding device management service (north) to a device driver service (south); normal data fed back by the device driving service (southbound) to the device management service (northbound) corresponds to 'UP'; error information fed back by the device driving service (southbound) to the device management service (northbound) corresponds to the 'UP-ERR';

-protocol id': UUID (Universally Unique Identifier, universal unique identification code) of protocol metadata, protocol (device driver services) unique;

- 'ProductID': UUID of the metadata of the product, the product is unique;

-DeviceID': UUID of the device metadata, device unique;

- 'FuncID': an ID of the current operation, an ID corresponding to the 'property|method|event';

-OptType', type of operation: 1) Bidirectional operation: for 'property', optional 'read|hard-read|write', corresponding to soft READ, HARD READ, and WRITE operations, respectively, of the device attribute; for 'method', selecting 'CALL', and calling corresponding equipment method; 2) Unidirectional operation: for 'property', selecting 'PROP' and reporting corresponding equipment attribute; for 'EVENT', the reporting operation of the corresponding device EVENT (the difference between the reporting operation and the reporting operation of 'PROP' is that 'PROP' is actively acquired by a device driver to a real device, and 'EVENT' is actively pushed by the real device to the device driver); in particular, for the device state detection function, a specific operation type 'STATUS' is defined for reporting the device state;

- 'ReqID': for bi-directional operations, if a request and a response belonging to the operation cannot be bound, when multiple operations are performed concurrently, the request and the response may be confused, and thus a unique UUID is required to indicate that the same group of operations is identified.

It will be appreciated that DATA in Topic represents time series DATA in a predefined DATA format, the remainder being associated information relating to the time series DATA. The encapsulated Topic is transferred to the timing database 1002.

Data exchange between the device driver service and the device management service can be facilitated by encapsulating the message Topic in MQTT format. It will be appreciated that data exchange between the virtual shadow Va and the timing database 1002 is also facilitated.

In a synchronization step S206, the time series data is synchronized to a cloud database 1021 in communication with the EDS100 through a predefined database interface of the time series database.

A predefined database interface is defined based on the metadata, the predefined database interface being compatible with different time series databases.

Specifically, to mask the underlying timing database implementation details up (i.e., to cloud 102), the present invention provides a unified predefined database interface, compatible with the mainstream timing databases, timeLyre, influxDB and TDengine, etc., that EDS100 encapsulates the data operations of these mainstream timing databases. A specific description of the predefined database interface is as follows.

For data storage, EDS100 defines a 'Device Data Record' structure for a unified data model (as shown in tables 1-6), which includes: protocol id, UUID of protocol metadata, protocol (device driver service) unique; product ID, UUID of product metadata, product unique; UUID of deviceID device metadata, device unique; funcID, currently operated ID, ID corresponding to property method event; optType, type of operation corresponding to the current data, e.g., READ, WRITE, EVENT, CALL; properties, the attribute list corresponding to the current data operation, and each attribute corresponds to one Property.

When storing the collected data, EDS110 tabulates the information collected by each specific physical device (i.e., internet of things device) ID in the format of $ { ProtocolID }/$ { ProductID }/$ { DeviceID }. Then according to the data insertion rules of the mainstream timing database, EDS100 classifies Device Data Record fields into two categories: tags and Fields. The Tags comprise protocol metadata UUIDs, product metadata UUIDs, device metadata UUIDs, current operation IDs and operation types corresponding to the current data. Fields include corresponding Properties for the current operating parameter, and are written into the target database with the attribute name key and the attribute Value.

During data Query, EDS100 extracts Query rules of the mainstream timing database, abstracts a Query structure Query Request, and defines the following table 8.

TABLE 8Query Request definition

Wherein clase. Select represents the field to be queried and given the type of operation of the query, comprising: raw, avg, sum, max, min, count; clase, white defines the data range of the current query operation, including the start time introduction time. Or inquiring the data of a specific time node; clase. It is also specified how to fill in missing values, optional filling modes are: the method comprises the steps of not returning data without values, filling data of a previous section, filling data of a next section, filling missing values in a linear interpolation mode, filling null values and filling specified values.

EDS100 defines a unified data representation format Query Response for different formats of data returned by different time series databases. The Query Response consists of one or more Series structures. Each Series includes Columns, consists of one or more Columns, and defines Column attributes for each row of query results, as shown in Table 9 below. Rows consists of one or more Rows, which describe specific values of query results per Row, and in which interface data types are stored to describe specific values of multiple types of data.

Table 9Column definition

Fields	Description of the invention
		Index	Column index corresponding to Column
Name	The column name
		DataType	The column data type
Tag	Identifying whether the column is a Tag column

Based on the predefined database interfaces described above, EDS100 is capable of supporting cloud-edge heterogeneous database data synchronization. For synchronization of large amounts of non-real time data in an edge acquisition scenario, EDS100 defines a Sync Task and a Sync Task Runner. The sync task is a specific sync task configuration, and designates a table name of a database to be synchronized, an automatic sync Crontab configuration, cluster ID identifiers of both sync parties, a sync file storage path, and the like. The Sync Task Runner is an executor of a specific synchronous Task, and automatically executes the synchronous Task and monitors the progress of the synchronous Task according to the Sync Task configuration. In order to manage the Sync Task and the Sync Task Runner, the EDS100 defines a Sync manager, which is responsible for adding, deleting, modifying, and checking synchronization tasks.

Table 10 shows the definition of the Sync Task and table 11 shows the definition of the Sync Task Runner.

Table 10

Fields	Description of the invention
		TaskID	Specifying the name of a synchronous task
TableName	Table names for performing synchronization tasks
		From	Sender edge EDS Cluster ID
To	Receiver cloud EDS cluster ID
		Crontab	Synchronizing task timingExecution configuration

TABLE 11

In addition, in order to reduce bandwidth consumption of edge non-real time sequential data synchronization, EDS100 employs Apache part (open source storage format) as an intermediate format for data transmission, as compared to the conventional Json data format. That is, after converting the time-series data in the time-series database 1002 into a predetermined data format (e.g. Apache part format) through the predefined database interface, the time-series data is synchronized to the cloud database 1021.

Specifically, as the EDS100 of the synchronous data sender, query Response results are queried from the unified predefined database interface of the time sequence database 1002, and are converted into an array (an efficient data exchange format) memory data format according to the data type of Column, and finally compressed into a part file format to perform data transmission on cloud edges, that is, to transmit to the cloud database 1021. The cloud database 1021, which is the synchronous data receiver, converts the received data from the compressed part file format to the arow format, and converts the arow memory data format to the Device Data Record format corresponding to the unified data model of the EDS100 according to the list identifier of the arow format. In this way, the time series data is synchronized (written) into the cloud database 1021 through the unified predefined database interface.

Fig. 3 shows a flow chart of a method of collecting and synchronizing time series data of different protocols according to a second embodiment of the invention. Steps S200, S202, S204, S206 in the second embodiment described in fig. 3 are identical to steps S300, S302, S304, S306 in the first embodiment shown in fig. 2, and thus are not described in detail here. Step S305 in fig. 3 is described in detail below.

As shown in fig. 3, after the transmission step S304, the method of the present invention further includes a downsampling step S305 for downsampling the time series data in the time series database 1002 based on a dynamic downsampling frequency.

Specifically, an outlier detection algorithm is used to detect outlier segments in the timing database 1002, downsampling at an outlier downsampling frequency when outlier segments are detected, and downsampling at a default downsampling frequency when outlier segments are not detected.

In order to dynamically adjust the edge-side downsampling frequency according to the result of the preprocessing algorithm in an offline manner, the EDS 100 of the present invention is designed with DSRunner (Down Sampling Runner) for performing specific downsampling tasks, and automatically performs the downsampling tasks on the target data table at intervals of 24 hours, as defined in the following table 12.

Table 12DSR filter key definition

TaskID is a unique identification of DSR filter, and TableName and TargetColumns specify table names and Fields column names in the timing database 1002 to be preprocessed. The TableName specifies the specific physical device to be preprocessed as specified by the unified predefined database interface. One or more attribute names in the particular device Fields are specified in TargetColumns. Dsluner automatically preprocesses the designated target column every day, and establishes an attached Table ds_table for each Table to be subjected to preprocessing task to store the preprocessed data for cloud edge synchronization. The specific implementation process is as follows: the DSR unit extracts the target column value in a sliding window mode and then detects the target column value according to a specified abnormal point detection algorithm. Wherein the configuration of the outlier detection algorithm, sliding window, is specified by algorithm options. If an abnormal point is found, marking the section of window as an abnormal data section, and inserting the abnormal data section into the DS_Table according to an abnormal downsampling frequency (usually the full quantity); if no outlier is found, the method inserts according to the default downsampling frequency. The abnormal downsampling frequency is, for example, less than a default downsampling frequency.

EDS100 provides built-in digital Outlier (numerical Outlier) Outlier detection algorithms, Z-score and Box Plot (Box Plot), respectively. If the two built-in digital outliers cannot meet the outlier detection requirement of the specific scene of the user, the EDS100 derives the Rows of the Query Response through the unified predefined database interface of the time sequence database and reflects the Rows as the original data type set. EDS100 provides the ability to customize outlier detection algorithms based on the raw data set.

Through the downsampling step, the amount of data synchronized to the cloud database 1021 can be reduced, and the transmission bandwidth is reduced.

In addition, as described above, the EDS100 of the present invention includes a device driver service and a device management service (edge-device-manager).

The main functions of the device driver services include:

-protocol registration: registering a current device protocol with a device management service;

-device initialization: acquiring product and device metadata from a device management service, and loading a device driver;

metadata listening: monitoring the change of the device management service product and the device metadata, and loading/unloading/reloading the driver;

-device attribute read-write: reading attribute data defined by a product from real equipment or a drive cache;

-device method invocation: calling a method supported by the equipment;

-device event listening: forwarding the data actively pushed by the device to a MessageBus, wherein the encapsulation MQTT is used as a middleware MessageBus for data exchange between the device driving service and the device management service;

-device status check: checking the health state of real equipment, and periodically pushing the health state to a message bus;

-device driver service status check: checking the health status of the device driver services and pushing the health status to the MessageBus periodically.

In order to manage the device driver services that have been accessed, a device management service is defined, and main functions include:

protocol management: receiving a registration request of equipment service;

-product management: product & product CRUD (add, read, update, delete) are defined based on a specific protocol;

-device management: the device & device CRUD is defined based on a specific product.

The device management services are responsible for protocol, product and device management. EDS100 defines resources under an edge-device-manager directory, which is used to store metadata in json format for products and devices. The manager client is defined for managing metadata and data operations of the corresponding driven devices based on the metadata and object model operation specifications mentioned above and the encapsulated msgbus. Manager client is defined in table 13 below.

Table 13 ManagerClient operation definition

/>

And the device management service acts as a definer of device attributes and events, EDS100 has devised that manager service is responsible for accepting events and reported attributes published by the device driver, as shown in table 14.

Table 14 ManagerService operation definition

In the invention, the virtual shadow equipment is realized by a driver layer and is responsible for communication with physical equipment of a specific protocol. The EDS100 deposits different twinrunners defined for different protocols in this layer as specific implementations of virtual shadow devices. For example, EDS100 has provided opcua-twin-runner, s7-twin-runner, or custom-written device-twin. The driver layer designs a cache structure and stores various devices, products and TwinRunner based on unique IDs of the product devices. A driveservice (table 15) is also defined at the driver layer, and the msgbus based on encapsulation operating as metadata and object model receives and performs the operation of device management service delivery.

Table 15 driiveservice operation definition

A driverClient (table 16) is also defined at the driver layer, which is responsible for publishing device attributes and events of manager subscriptions to the device management service.

Table 16 drive client operation definition

Operation of	Function of
		PublishDriverStatus	Push-driven status update message
PublishDeviceStatus	Push device status update message
		PublishDeviceProps	Attribute update message for push device
PublishDeviceEvent	Push device event

The above description is for a device registered in the EDS100, and a procedure of modifying the registered device is described below. When the EDS100 receives the device change instruction, the device change instruction is encapsulated into a message in MQTT format and then transmitted to the device driver service, so that the device driver service operates according to the device change instruction.

The device change instruction is, for example, an add device instruction, a delete device instruction, or an update device instruction, etc., for modifying a physical device that has been registered in the EDS 100.

Specifically, the device change instruction is encapsulated into a message Topic in MQTT format, where the general format of Topic is 'META/$ { Version }/$ { OptMode }/$ { protocol ID }/$ { OptType [/$ { ID }) ]':

-Version: versions of META Topic;

-an 'OptMode' operation mode, optionally down|up: 'DOWN': an operation initiated by a corresponding device management service (north) to a device driver service (south); normal data fed back by the device driving service (southbound) to the device management service (northbound) corresponds to 'UP';

-OptType' operation type, optional init|product|device: 1) The device driver service judges the specific operation type according to the cache corresponding to the adding/deleting/modifying operation of the PRODUCT: firstly judging whether the Payload is empty or not, and if so, indicating that the product is deleted; otherwise, judging whether the equipment driving service cache exists or not; "prod_id", if not present, means product creation; otherwise, the product update is represented; 2) The DEVICE driver service judges whether the Payload is empty or not according to the specific operation type by the buffer memory according to the adding/deleting/modifying operation of the DEVICE, and if the Payload is empty, the deletion of the DEVICE is indicated; otherwise, judging whether the equipment driving service cache exists or not; "prod_id" if not present, means device creation, otherwise means device update; 3) "STATUS", corresponding to the STATUS reporting operation of the device driver services, needs to include a STATUS code, protocol metadata, and a timestamp (initially 0) of the last "APPEND" operation; 4) The device management service sends updated products & devices to the device driver service increment according to the timestamp in the STATUS corresponding to the product & device metadata adding operation initiated by the device management service to the device driver service;

-ID: for unidirectional operation, an operation object for specifying addition/deletion/modification of a product/device.

Technical effects of the invention

In terms of protocol support, the invention can realize acquisition and synchronization of time sequence data of equipment using a plurality of different protocols, including common internet of things protocol support in 6: modbusRTU/ModbusTCP/OPC-UA/COAP/BACNet/S7.

In terms of a unified predefined database interface of a timing database, the present invention can interface to, for example, three mainstream timing databases: influxDB/TimeLyre/TDengine, so heterogeneous database synchronization capability can be achieved.

Through the test of a protocol simulator, the invention supports the adoption frequency of the equipment attribute with the millisecond basic, thereby greatly improving the performance, the expansibility and the protocol support quantity.

Cloud edge synchronization efficiency contrast

In order to simulate a real cloud edge synchronous scene, the experiment generates test data according to a designed unified data model at a frequency of one piece per second, and the test data are inserted into a time sequence database of an edge end, and 60 ten thousand high-frequency thermometer acquisition scenes based on OPC-UA protocol are simulated at the edge end. The data format content of the data set is shown in table 16 below.

Table 17 data formats of datasets

Data field	Description of the invention
		timestamp	Timestamp (int, 10 bit timestamp)
Protocol_id	Corresponding protocol id (Nchar, 256 bits)
		Product_id	Corresponding product id (Nchar, 256 bits)
device_id	Corresponding device id (Nchar, 256 bits)
		func_id	Corresponding operation id (Nchar, 256 bits)
opt_type	Corresponding operation type (Nchar, 256 bits)
		temperature	Attribute value temperature (float)

Except for temperature, all the other attributes are collection tag attributes in the EDS100, and are fixed values. For temperature, a normal air temperature data value at a certain place corresponding to the moment of the data time stamp is adopted, the normal air temperature data value is uniformly generated in units of the temperature, and the data range normally fluctuates within the range of-5-40 ℃. In addition, 20 abnormal points deviating from normal air temperature are randomly inserted, and abnormal values in a real environment are simulated.

The data set is respectively applied to the parquet format of the method and the synchronization comparison performed for json format by adopting the traditional data export. The execution time, the synchronous data file size, and the data compression rate are recorded, respectively. Specific indexes and experimental efficiencies are shown in tables 18 and 19 below.

Table 18 introduction to experimental index

Table 19 comparative table of synchronous experimental efficiency

Through calculation, the ratio (namely, the data compression rate) of the data size of cloud edge synchronization adopting the part format of the invention to the data size adopting the json of the traditional format is 28/136=20.58%.

Downsampling at dynamic downsampling frequency (data preprocessing efficiency vs.)

The dynamic downsampling frequency strategy (the data outlier detection algorithm adopts Z-SCORE, the outlier downsampling frequency is the whole, the normal downsampling frequency is 1m, the window size is configured to be 1h, the sliding step length is 30 m) and the fixed downsampling frequency is 1m, which are used in the invention, are respectively used for processing the data set. The execution time, the size after the downsampling process, the data compression rate and the number of abnormal points contained in the processed data are recorded respectively.

Table 20 comparison Table of pretreatment experiment efficiencies

As can be seen from the comparison of cloud edge synchronization efficiency and data preprocessing efficiency, in terms of synchronization efficiency, as shown in Table 19, the synchronization strategy adopted by the invention is 11s in execution time, and the execution time of the traditional synchronization strategy is 22s, which is twice that of the invention. In addition, in terms of the size of the synchronous data file, the invention has 28MB for a 60-kilo-scale data set, and the traditional synchronous strategy reaches 136MB. Therefore, the synchronous execution efficiency of the method adopted by the patent is improved, and the bandwidth use of the edge end (edge node) can be reduced.

In terms of data preprocessing efficiency, as shown in table 20, the preprocessing strategy of dynamic downsampling frequency adopted by the invention has execution time 104.37898s which is not much different from the execution time 100.16437s of fixed downsampling frequency. And finally, dynamically downsampling the 27866 pieces of data in the processed size, and fixing the 10000 pieces of data in the downsampling frequency. The processed data contains the number of abnormal points, the dynamic downsampling frequency is 20, and the fixed downsampling frequency is 0. The dynamic downsampling algorithm adopted by the invention can upload the data in the window in full when the existence of the abnormal point is detected under the condition that the abnormal point exists in the data set, and can better and more accurately reflect the real situation of the data near the abnormal point of the data set, so that the data transmission required by cloud edge data synchronization can be reduced maximally on the premise of not influencing the analysis precision.

It can be seen that the invention can realize the acquisition and synchronization of time sequence data of a plurality of different protocols; utilizing a unified predefined database interface to realize heterogeneous database synchronization capability; the Apache part is utilized to improve cloud edge synchronous execution efficiency and reduce bandwidth use of an edge end (edge node); the dynamic downsampling frequency is utilized for data preprocessing, and data transmission required by cloud edge data synchronization can be reduced to the greatest extent on the premise that analysis accuracy is not affected.

The present invention also provides a device for collecting and synchronizing time sequence data of different protocols, which is applied to a client, as shown in fig. 4, the device 40 includes:

an initializing unit 401, configured to initialize each device based on metadata of each of one or more devices in communication with the client, and generate a virtual shadow device for each device, where the metadata includes a protocol, a product, and a device, and the product is represented by a digitized object model;

the acquisition unit 402 determines virtual shadow equipment corresponding to designated equipment in the acquisition instruction according to the received acquisition instruction, and enables the corresponding virtual shadow equipment to acquire the time sequence data from the designated equipment using a corresponding protocol through a predefined interface;

The transmission unit 403 converts the time sequence data into a predefined data format based on the object model, encapsulates the time sequence data by using an MQTT protocol and transmits the encapsulated time sequence data to a time sequence database in the client;

and the synchronization unit 404 synchronizes the time sequence data to a cloud database communicated with the client through a predefined database interface of the time sequence database.

Referring now to fig. 5, fig. 5 is a block diagram illustrating a client according to an implementation of the present application. The clients may include one or more processors 702, system control logic 708 coupled to at least one of the processors 702, system memory 704 coupled to the system control logic 708, non-volatile memory (NVM) 706 coupled to the system control logic 708, and input/output (I/O) interface 710 coupled to the system control logic 708.

The processor 702 may include one or more single-core or multi-core processors. The processor 702 can include any combination of general-purpose and special-purpose processors (e.g., graphics processor, application processor, baseband processor, etc.). In implementations herein, the processor 702 may be configured to perform the aforementioned methods of collecting and synchronizing time series data of different protocols.

In some implementations, system control logic 708 may include any suitable interface controller to provide any suitable interface to at least one of processors 702 and/or any suitable device or component in communication with system control logic 708.

In some implementations, system control logic 708 may include one or more memory controllers to provide an interface to system memory 704. The system memory 704 may be used for loading and storing data and/or instructions. In some implementations, the client's system memory 704 may include any suitable volatile memory, such as suitable dynamic random access memory (Dynamic Random Access Memory, DRAM).

NVM/memory 706 may include one or more tangible, non-transitory computer-readable media for storing data and/or instructions. In some implementations, NVM/memory 706 may include any suitable nonvolatile memory, such as flash memory, and/or any suitable nonvolatile storage device, such as at least one of a Hard Disk Drive (HDD), compact Disc (CD) Drive, digital versatile Disc (Digital Versatile Disc, DVD) Drive.

NVM/memory 706 may include a portion of a storage resource installed on the client's device or it may be accessed by, but not necessarily part of, the apparatus. For example, NVM/memory 706 may be accessed over a network via an input/output (I/O) interface 710.

In particular, system memory 704 and NVM/storage 706 may each include: a temporary copy and a permanent copy of instruction 712. The instructions 712 may include: instructions that, when executed by at least one of the processors 702, cause the client to implement the aforementioned method of collecting and synchronizing time series data of different protocols. In some implementations, instructions 712, hardware, firmware, and/or software components thereof may additionally/alternatively be disposed in system control logic 708, input/output (I/O) interface 710, and/or processor 702.

In one implementation, at least one of the processors 702 may be packaged together with logic for one or more controllers of the system control logic 708 to form a system package (System In a Package, siP). In one implementation, at least one of the processors 702 may be integrated on the same die with logic for one or more controllers of the System control logic 708 to form a System on Chip (SoC).

The present application also provides a computing device 80. As shown in fig. 6, the computing device 80 includes: bus 802, processor 804, and memory 80. Communication between processor 804, memory 806, and communication interface 808 is via bus 802. The computing device 80 may be a server or a terminal device. It should be understood that the present application is not limited to the number of processors, memories in computing device 80.

As shown in fig. 6, the memory 806 has stored therein executable program code that is executed by the processor 804 to implement the aforementioned methods of collecting and synchronizing time series data of different protocols, respectively. That is, the memory 806 has instructions stored thereon for performing the method of collecting and synchronizing time series data of different protocols.

The embodiment of the application also provides a computing device cluster. The cluster of computing devices includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop, notebook, or smart phone.

As shown in fig. 7, the cluster of computing devices includes at least one computing device 80. The same instructions for performing the methods of collecting and synchronizing time series data of different protocols may be stored in memory 806 in one or more computing devices 80 in the cluster of computing devices.

The present application further provides a computer program product comprising instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored in any useful medium. The computer program product, when run on at least one computing device, causes the at least one computing device to perform a method of collecting and synchronizing time series data of different protocols.

Implementations of the present application also provide a computer-readable storage medium. Computer readable storage media can be any available media that can be stored by a computing device or data storage device such as a data center containing one or more available media. Usable media may be magnetic media (e.g., floppy disks, hard disks, magnetic tape), optical media (e.g., DVD), or semiconductor media (e.g., solid state disk), among others. The computer-readable storage medium includes instructions that instruct a computing device to perform a method of collecting and synchronizing time series data of different protocols.

In this application implementation, the terms "first," "second," and the like are used merely to distinguish between descriptions and are not to be construed as indicating or implying relative importance.

In the implementations of the application, some structural or methodological features may be shown in a particular arrangement and/or order in the drawings. However, it should be understood that such a particular arrangement and/or ordering may not be required. Rather, in some embodiments, these features may be arranged in a different manner and/or order than shown in the illustrative figures. Additionally, the inclusion of structural or methodological features in a particular figure is not meant to imply that such features are required in all embodiments, and in some embodiments, may not be included or may be combined with other features.

While the application has been shown and described with reference to certain embodiments thereof, it will be understood by those of ordinary skill in the art that the foregoing is a further detailed description of the application in conjunction with the specific embodiments and is not intended to limit the practice of the application to such descriptions. Various changes in form and detail may be made therein by those skilled in the art, including a few simple inferences or alternatives, without departing from the spirit and scope of the present application.

Claims (13)

1. A method for collecting and synchronizing time sequence data of different protocols, which is applied to a client, and is characterized in that the method comprises the following steps:

An initializing step, namely initializing each device based on metadata of one or more devices communicated with the client, and generating a virtual shadow device for each device, wherein the metadata comprises the device, a protocol corresponding to the device and a product, and the product is represented by a digitalized object model;

a collection step of determining virtual shadow equipment corresponding to designated equipment in a collection instruction according to the received collection instruction, and enabling the corresponding virtual shadow equipment to collect the time sequence data from the designated equipment using a corresponding protocol through a predefined interface;

a transmission step, based on the object model, converting the time sequence data into a predefined data format, and transmitting the time sequence data to a time sequence database in the client after packaging by utilizing an MQTT protocol;

and synchronizing, by a predefined database interface of the time sequence database, the time sequence data to a cloud database in communication with the client.

2. The method of claim 1, wherein the object model defines attributes, methods and events of the products, each product comprising a corresponding one or more devices,

Wherein the metadata includes the device, the protocol corresponding to the device, and attributes, methods, events of the product.

3. The method of claim 2, wherein the predefined interface is defined based on operation of the object model, the predefined interface comprising a plurality of methods implemented by the virtual shadow device,

in the collecting step, the corresponding virtual shadow device accesses the designated device using the corresponding protocol through the methods to collect the time sequence data.

4. The method according to claim 2, wherein in the transmitting step, the time series data is converted into a predefined data format based on the description information of the attribute, method and event of the product, and the time series data in the predefined data format and the associated information related to the time series data are packaged into a message in MQTT format and then transmitted to the time series database.

5. The method of claim 2, wherein the predefined database interface is defined based on the metadata, the predefined database interface being compatible with different timing databases.

6. The method of claim 5, wherein in the synchronizing step, the time series data in the time series database is synchronized to the cloud database after being converted into a preset data format through the predefined database interface.

7. The method of claim 1, further comprising, after the transmitting step, a downsampling step for downsampling the time series data in the time series database based on a dynamic downsampling frequency.

8. The method of claim 7, wherein in the downsampling step, an outlier detection algorithm is used to detect outlier segments in the temporal database, downsampling is performed at an outlier downsampling frequency when the outlier segments are detected, and downsampling is performed at a default downsampling frequency when the outlier segments are not detected.

9. The method of claim 1, wherein the client comprises a device driver service for implementing the initializing step and a device manager service for implementing the collecting step, the transmitting step, and the synchronizing step,

When a device change instruction is received, the device change instruction is packaged into an MQTT format message and then transmitted to the device driving service, so that the device driving service operates according to the device change instruction.

10. An apparatus for collecting and synchronizing time sequence data of different protocols, applied to a client, characterized in that the apparatus comprises:

an initializing unit, which is used for initializing each device based on metadata of one or more devices communicated with the client, and generating a virtual shadow device for each device, wherein the metadata comprises the device, a protocol corresponding to the device and a product, and the product is represented by a digitalized object model;

the acquisition unit is used for determining virtual shadow equipment corresponding to designated equipment in the acquisition instruction according to the received acquisition instruction, and enabling the corresponding virtual shadow equipment to acquire the time sequence data from the designated equipment using a corresponding protocol through a predefined interface;

the transmission unit is used for converting the time sequence data into a predefined data format based on the object model, packaging the time sequence data by utilizing an MQTT protocol and transmitting the packaged time sequence data to a time sequence database in the client;

And the synchronization unit is used for synchronizing the time sequence data to a cloud database communicated with the client through a predefined database interface of the time sequence database.

11. A cluster of computing devices, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of collecting and synchronizing time series data of different protocols according to any one of claims 1-9.

12. A computer program product containing instructions which, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of collecting and synchronizing time series data of different protocols as claimed in any one of claims 1 to 9.

13. A computer readable storage medium comprising computer program instructions which, when executed by a cluster of computing devices, perform the method of collecting and synchronizing time series data of different protocols according to any of claims 1-9.