CN115357397A - Equipment data processing method and system for industrial internet platform - Google Patents

Abstract

The invention relates to a method and a system for processing equipment data of an industrial internet platform, wherein the method comprises the following steps: and establishing a data processing service which is parallel to the link where the access machine is located and can convert the original data into standard data, wherein the data processing service acquires the original data uploaded by the access machine and performs data processing, so that the link where the access machine is located only performs data transmission. By establishing data processing service, the access machine is not responsible for service processing, and only needs to receive and upload data to realize the decoupling of data uploading and data processing; therefore, on the premise of not disconnecting the connection between the equipment and the platform and not losing data, the development logic of hardware does not need to be changed, the data processing is not needed to be additionally carried out by a third-party application terminal, and the equipment data which are different in data model and expression but have the same meaning can be processed into standardized data for the analysis and use of the third-party application terminal.

Description

Equipment data processing method and system for industrial internet platform

Technical Field

The invention relates to the field of industrial internet platforms, in particular to a device data processing method and system of an industrial internet platform.

Background

In recent years, with the wide and deep application of information communication technology, the fusion development of the manufacturing industry and the internet is deepened continuously, a good foundation is laid for the further development of the industrial internet, and a plurality of industrial internet platforms are also developed.

The equipment in the industrial scene is based on an MQTT protocol, the equipment is used as an MQTT client, an industrial internet platform is used as an MQTT server, and access, data report and the like of the equipment are realized. When the equipment is used as the MQTT client, the interactive logic of the equipment and the platform needs to be written into a hardware program through embedded development, and then the hardware is welded on the equipment. Therefore, when the device uploads data to the industrial internet platform, the data is transferred by a fixed data model, including field names, field types, data lengths, units and the like.

The devices used by enterprises are from different manufacturers, and even if the devices are from the same manufacturer, the models and versions of the devices may be different, so that the data models used in reporting data may be different, however, the meanings expressed by the devices may be the same. For example, the electricity consumption of the equipment, the reported data of the equipment A is { "50001":2000}, the reported data of the equipment B is { "electric":2.0}, and all the reported data represent that the electricity consumption of the equipment is 2kw/h, but the field names, the data types and the units are different. Therefore, data representing the same meaning needs to be integrated to form standardized data, so that a third-party application terminal can conveniently acquire data information.

In order to standardize data, it is impractical to require different enterprises to purchase equipment of the same manufacturer, the same model and the same version; if the hardware program on the equipment is modified to achieve standardization, a large amount of manpower and financial resources are consumed; if a third-party application terminal is required to analyze data to form standardization, a lot of repetitive work can be done, the analysis modes are different, the consistency of the data cannot be guaranteed, the fault tolerance rate is low, and the workload is large.

Some proposals have been made to complete data receiving, processing and reporting at the server side, and report the data after the server side processes the data uniformly, which increases the load of the server side, affects the stability of the server side, and causes the server side to disconnect from the device and lose data; data is also easily lost when updating the data processing function.

The above problems are worth solving.

Disclosure of Invention

In order to deal with the condition that data are not uniform due to the fact that reported meanings are the same but data models are different during data acquisition, and meanwhile avoid the risk that the load of a server is increased, the stability of a system is influenced, and data are lost, the invention provides a simple and flexible equipment data processing method and system of an industrial internet platform, which have strong expansibility and minimum influence on users and the platform, and the standardized processing of the data is realized.

The technical scheme of the invention is as follows:

a device data processing method of an industrial Internet platform establishes a data processing service which is parallel to a link where an access machine is located and can convert original data into standard data, the data processing service acquires the original data uploaded by the access machine and performs data processing, so that the link where the access machine is located only performs data transmission, and the method comprises the following steps:

step 1, the access machine serves as a production end of the Kafka system and sends original data to a first theme of the Kafka system;

step 2, establishing a data processing service as a first consumption end of the Kafka system, and consuming original data from a first theme of the Kafka system;

step 3, the data processing service inquires a data conversion script uploaded in advance;

and 4, the data processing service executes a data conversion script corresponding to the original data, converts the original data into standard data, and sends the standard data to a second theme of the Kafka system to be consumed by other consuming terminals.

According to the invention of the scheme, the access machine is an MQTT server, and different devices send respective original data to the MQTT server through an MQTT protocol.

According to the invention of the above scheme, the step 3 comprises the following steps:

s301, uploading a data conversion script in advance;

s302, the data service inquires the corresponding data conversion script.

Further, the uploading of the data conversion script in advance specifically includes:

sending the data conversion script to one of the message queues of the RabbitMQ system in a message form by using a console, subscribing the message queue by using a data processing service, and acquiring a script message;

and the data processing service analyzes the script message, and updates the script message to a local cache device by taking the combination of the productKey and the streamName as a key name.

According to the invention of the scheme, the console also comprises a message queue for deleting script messages in the RabbitMQ system message queue.

Further, the data service queries a corresponding data conversion script, specifically:

inquiring script information in a local cache device;

judging whether a script message corresponding to the original data exists or not, wherein the corresponding rule comprises the following steps: the original data is consistent with the key name carried by the script message;

if yes, entering step 4, and executing the script message;

if not, the raw data is sent directly to the second topic of the Kafka system.

According to the scheme, the data conversion script is a JS script, and the method name of the JS script is dataStream _ product Key _ streamName (byte), wherein dataStream represents that the script acts on a data stream, product Key represents a unique identifier of a product to which the equipment belongs, streamName represents an identifier of a data stream model, and byte represents a binary stream of original data of the equipment.

Furthermore, the method content of the data conversion script comprises the steps of modifying the structure of original data, modifying the data type and modifying the value, wherein the return value of the data conversion script is standard data of equipment.

The present invention according to the above aspect further includes, after step 4: and 5, taking the third-party application terminal as a second consumption terminal of the Kafka system, and consuming data from a second theme of the Kafka system.

The invention also provides a device data processing system of the industrial internet platform, which is used for operating the device data processing method of the industrial internet platform of the scheme, and comprises a Kafka system and a RabbitMQ system, and further comprises:

the equipment is used for generating and transmitting original data;

the access machine is used for receiving the original data of the equipment and producing the original data to the Kafka system as a production end of the Kafka system;

the data processing service is used as a first consuming end of the Kafka system and used for consuming data from a first theme of the Kafka system and sending the data to a second theme of the Kafka system; the data processing service is also used as a consumption end of the RabbitMQ system and is used for acquiring and analyzing messages from the RabbitMQ system;

the local cache device is used for storing the script message analyzed by the data processing service;

the console is used as a production end of the RabbitMQ system and used for sending script messages;

and the third party application terminal is used as a second consumption terminal of the Kafka system and is used for consuming data from the second theme of the Kafka system.

The invention according to the scheme has the beneficial effects that:

the invention not only realizes the direct generation of standardized data, solves the troubles of different data models brought by different devices, for example, reduces the workload of a third-party application end, and avoids data inconsistency caused by the difference of data processing modes of each application end;

the data processing service consumes data for logic processing by establishing a data processing service which is parallel to a link where the access machine is located and can convert original data into standard data, and the service end, namely the access machine, is not responsible for service processing and only needs to upload data of receiving equipment, so that decoupling of data uploading and data processing is finally realized; therefore, even if the logic for processing data is changed and other functions are expanded, only the data processing service is influenced, and the access machine is not influenced to cause the disconnection of equipment and the loss of data; when the data processing service is updated, only short consumption delay occurs, and data cannot be lost.

Drawings

FIG. 1 is a flow chart of data processing according to the present invention.

Detailed Description

For better understanding of the objects, technical solutions and effects of the present invention, the present invention will be further explained with reference to the accompanying drawings and examples. It is to be noted that the following examples are provided only for illustrating the present invention and are not intended to limit the present invention.

It will be understood that when an element is referred to as being "secured to" or "disposed on" another element, it can be directly on the other element or intervening elements may also be present. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements may also be present.

The terms "first", "second" and "first" are used merely for descriptive purposes and are not to be construed as indicating or implying relative importance or to implicitly indicate a number of technical features. The meaning of "a number" is two or more unless specifically defined otherwise.

In order to make the technical solutions of the embodiments of the present invention easier to understand for those skilled in the art, the technical terms related to the present invention and the technical flows in the previous stages are explained first:

the Kafka system, a high throughput distributed publish-subscribe messaging system, can handle all the flow data of consumers in a web site, including:

producer, the Producer/Producer of messages and data, may issue a message to one topic of Kafka;

Consumer-Consumer/Consumer of messages and data, can subscribe to topic and handle consumption of its publication;

topic (subject) -a physical grouping, a Topic may be divided into multiple parts (partitions), each part being an ordered queue; each message in part is assigned an ordered Id (offset);

message-is the basic unit of communication, and each producer may publish some messages to one topic.

The RabbitMQ system is message-oriented middleware which realizes an Advanced Message Queuing Protocol (AMQP); each producer (producer) or consumer (consumer) is to send and consume messages through the RabbitMQ system, and first, a Connection is established with the RabbitMQ system, where the Connection is a Connection that is a long TCP Connection.

And the third-party application terminal refers to other micro-service/application platforms which need to acquire data information from the system.

As shown in fig. 1, the present invention provides a device data processing method for an industrial internet platform, and specifically, a data-processing-service (data-processing-service) is established that is parallel to a link where an access machine is located and can convert original data into standard data, and the data-processing-service acquires the original data uploaded by the access machine and performs data processing, so that the link where the access machine is located only performs data transmission, including the following steps:

step 1, the access machine serves as a production end of the Kafka system and sends original data to a first theme (namely an original data theme, origin-data-topic) of the Kafka system; the access machine is an MQTT server, and different devices send respective original data to the MQTT server through an MQTT protocol.

And 2, establishing a data processing service as a first consumption end of the Kafka system, and consuming the original data from the first theme of the Kafka system.

Step 3, the data processing service inquires a data conversion script uploaded in advance; the method specifically comprises the following steps:

step 301, uploading a data conversion script in advance:

the data conversion script is sent to one of the message queues of the RabbitMQ system in a message form by using the console, the data processing service subscribes to the message queue, and the message is immediately pushed to the data processing service after reaching the RabbitMQ and the script message is obtained; the console also comprises a script message used for deleting the message queue of the RabbitMQ system; meanwhile, in order to avoid malicious attack by someone through the script, the script uploaded by the user needs to be audited by a manager, or only some roles have the authority to upload the script.

And the data processing service analyzes the script message, and updates the script message to a local cache device by taking the combination of the productKey and the streamName as a key name.

Step S302, inquiring a corresponding data conversion script by the data service:

inquiring script information in a local cache device;

because when equipment original data uploads, can carry productKey and streamName equally, so can judge whether there is correspondence between script message and the original data, the rule of correspondence includes: the original data is consistent with the key name carried by the script message;

if the script message corresponding to the original data exists, executing step 4, executing a data conversion script corresponding to the original data by the data processing service, converting the original data into standard data, sending the standard data to a second theme (namely a standard-data-topic) of the Kafka system, and waiting for consumption by other consumption terminals; specifically, the method comprises the following steps:

executing the corresponding script message, and sending the executed return value to a second theme of the Kafka system to finish the conversion of the original data into standard data; the data conversion script is a JS script, and the JS script is named as dataStream _ product key _ streamName (byte), wherein dataStream represents that the script acts on a data stream, the product key represents a unique identifier of a product to which the equipment belongs, streamName represents an identifier of a data stream model, and byte represents a binary stream of original data of the equipment; the code segment for the JS script is as follows:

original data { "a":100, "b":200} is converted into standard data { "a":100, "aa":100, "c":200};

the method content of the data conversion script comprises the steps of modifying the structure of original data, modifying the data type and modifying the value, wherein the return value of the data conversion script is standard data of equipment.

If the script message corresponding to the original data does not exist, the original data is directly sent to a second theme of the Kafka system; at this time, two cases are: in the first case, the data model of the original data is a standard data model, and other consumption terminals consume successfully; in the second situation, the data model of the original data is not the model of the standard data, after other consumption terminals consume, the server terminal obtains error information to remind the user that the data conversion script is missing, and a console is used for adding a new data conversion script to be put in storage (a local cache device).

And 5, taking the third-party application terminal as a second consumption terminal of the Kafka system, and consuming data from a second theme of the Kafka system.

In summary, the present invention establishes a data processing service that is parallel to the link where the access machine is located and can convert the original data into the standard data, and the data processing service consumes the data for logic processing, while the service end, i.e., the access machine, is not responsible for service processing, and only needs to upload the data of the receiving device, thereby finally realizing the decoupling of data uploading and data processing.

The invention also provides a device data processing system of the industrial internet platform, which is used for operating the device data processing method of the industrial internet platform of the scheme, and comprises a Kafka system and a RabbitMQ system, and further comprises:

the equipment is used for generating and transmitting original data;

the access machine is used for receiving the original data of the equipment and producing the original data to the Kafka system as a production end of the Kafka system;

the data processing service is used as a first consuming end of the Kafka system and used for consuming data from a first theme of the Kafka system and sending the data to a second theme of the Kafka system; the data processing service is also used as a consumption end of the RabbitMQ system and is used for acquiring and analyzing messages from the RabbitMQ system;

the local cache device is used for storing the script message analyzed by the data processing service;

the console is used as a production end of the RabbitMQ system and used for sending script messages;

and the third party application terminal is used as a second consumption terminal of the Kafka system and is used for consuming data from the second theme of the Kafka system.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above examples only show some embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent should be subject to the appended claims.

Claims (10)

1. A device data processing method of an industrial Internet platform is characterized in that a data processing service which is parallel to a link where an access machine is located and can convert original data into standard data is established, the data processing service acquires the original data uploaded by the access machine and performs data processing, so that the link where the access machine is located only performs data transmission, and the method comprises the following steps:

step 1, the access machine serves as a production end of the Kafka system and sends original data to a first theme of the Kafka system;

step 2, establishing a data processing service as a first consumption end of the Kafka system, and consuming original data from a first theme of the Kafka system;

step 3, the data processing service inquires a data conversion script uploaded in advance;

and 4, the data processing service executes a data conversion script corresponding to the original data, converts the original data into standard data, and sends the standard data to a second theme of the Kafka system to be consumed by other consuming terminals.

2. The device data processing method of industrial internet platform as claimed in claim 1, wherein the access machine is MQTT server, and different devices transmit respective raw data to MQTT server through MQTT protocol.

3. The device data processing method of the industrial internet platform as claimed in claim 1, wherein the step 3 comprises the steps of:

s301, uploading a data conversion script in advance;

s302, the data service inquires the corresponding data conversion script.

4. The device data processing method of the industrial internet platform according to claim 3, wherein the upload data conversion script specifically comprises:

sending the data conversion script to one of the message queues of the RabbitMQ system in a message form by using a console, subscribing the message queue by using a data processing service, and acquiring a script message;

and the data processing service analyzes the script message, and updates the script message to a local cache device by taking the combination of the productKey and the streamName as a key name.

5. The device data processing method of the industrial internet platform as claimed in claim 4, wherein the console further comprises a message queue for deleting script messages in the RabbitMQ system message queue.

6. The device data processing method of the industrial internet platform as claimed in claim 4, wherein the data service queries a corresponding data conversion script, specifically:

inquiring script information in a local cache device;

judging whether a script message corresponding to the original data exists or not, wherein the corresponding rule comprises the following steps: the original data is consistent with the key name carried by the script message;

if yes, entering step 4, and executing the script message;

if not, the raw data is sent directly to the second topic of the Kafka system.

7. The method for processing the data of the device on the industrial internet platform as recited in claim 1, wherein the data conversion script is a JS script, and a method name of the JS script is dataStream _ product key _ streamName (byte), wherein dataStream represents that the script acts on the data stream, product key represents a unique identifier of a product to which the device belongs, streamName represents an identifier of a data stream model, and byte represents a binary stream of original data of the device.

8. The device data processing method of the industrial internet platform as claimed in claim 7, wherein the method contents of the data conversion script include modification of structure of original data, modification of data type, modification of value, and the return value of the data conversion script is standard data of a device.

9. The device data processing method of the industrial internet platform as claimed in claim 1, further comprising, after the step 4: and 5, the third party application terminal serves as a second consumption terminal of the Kafka system, and the data are consumed from a second theme of the Kafka system.

10. A device data processing system of an industrial internet platform for operating the device data processing method of the industrial internet platform according to any one of claims 1 to 9, comprising a Kafka system and a RabbitMQ system, further comprising:

the equipment is used for generating and transmitting original data;

the access machine is used for receiving the original data of the equipment and producing the original data to the Kafka system as a production end of the Kafka system;

the data processing service is used as a first consuming end of the Kafka system and used for consuming data from a first theme of the Kafka system and sending the data to a second theme of the Kafka system; the data processing service is also used as a consumption end of the RabbitMQ system and is used for acquiring and analyzing messages from the RabbitMQ system;

the local cache device is used for storing the script message analyzed by the data processing service;

the console is used as a production end of the RabbitMQ system and used for sending script messages;

and the third party application end, which is a second consumption end of the Kafka system, is used for consuming data from the second theme of the Kafka system.

Images