CN111596628A - Acquisition method and acquisition system for mass industrial data - Google Patents
Acquisition method and acquisition system for mass industrial data Download PDFInfo
- Publication number
- CN111596628A CN111596628A CN202010477319.9A CN202010477319A CN111596628A CN 111596628 A CN111596628 A CN 111596628A CN 202010477319 A CN202010477319 A CN 202010477319A CN 111596628 A CN111596628 A CN 111596628A
- Authority
- CN
- China
- Prior art keywords
- data
- server
- module
- opc
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B19/00—Programme-control systems
- G05B19/02—Programme-control systems electric
- G05B19/418—Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS], computer integrated manufacturing [CIM]
- G05B19/4183—Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS], computer integrated manufacturing [CIM] characterised by data acquisition, e.g. workpiece identification
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B2219/00—Program-control systems
- G05B2219/30—Nc systems
- G05B2219/31—From computer integrated manufacturing till monitoring
- G05B2219/31282—Data acquisition, BDE MDE
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Manufacturing & Machinery (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Automation & Control Theory (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention provides a method and a system for acquiring massive industrial data, wherein the method comprises the following steps: acquiring digital data, document data, video data, image data, audio data, structured data and semi-structured data; establishing data transmission interaction of the structured data, the semi-structured data, the text data and the image data by utilizing an OPC-UA protocol; in the industrial data acquisition process, establishing an MQTT communication protocol between a sensor for acquiring the industrial data and a server, and carrying out communication transmission on the industrial data acquired by the sensor; establishing a text-based protocol between a client and a server, and transmitting text data, video data, image data and audio data; and carrying out data cleaning, data coding and decoding and data integration on the acquired data. The system comprises modules corresponding to the method steps.
Description
Technical Field
The invention provides a method and a system for acquiring massive industrial data, and belongs to the technical field of internet.
Background
The industrial network is a full digital, two-way and multi-station communication system installed in an industrial production environment, and is mainly divided into a special closed industrial network, an open industrial network and a standard industrial network, the industrial network system is an office and monitoring network which is indispensable in the production operation process of large enterprises or factories at present, and the generation of network technology has the following advantages for industrial control: (1) the installation and wiring are convenient; (2) modularization; (3) easy to diagnose; (4) self-construction; (5) and (5) carrying out enterprise management. Although the industrial control network has these advantages, in practice, the industrial control network often has data transmission errors and failures due to too much collected data during the actual operation process.
Disclosure of Invention
The invention provides a method and a system for acquiring massive industrial data, which are used for solving the problems of high transmission error rate and failure rate of acquired data, and the method comprises the following steps:
a method for acquiring massive industrial data, comprising the following steps:
acquiring digital data, document data, video data, image data, audio data, structured data and semi-structured data;
establishing data transmission interaction of the structured data, the semi-structured data, the text data and the image data by utilizing an OPC-UA protocol;
in the industrial data acquisition process, establishing an MQTT communication protocol between a sensor for acquiring the industrial data and a server, and carrying out communication transmission on the industrial data acquired by the sensor;
establishing a text-based protocol between a client and a server, and transmitting text data, video data, image data and audio data;
and carrying out data cleaning, data coding and decoding and data integration on the acquired data.
Further, the establishing data transmission interaction of the structured data, the semi-structured data, the text data and the image data by using the OPC-UA protocol includes:
the interface definition module is used for self-defining an interface and reading the structured data, the semi-structured data, the text data and the image data which are acquired by the data acquisition module by using the defined interface;
reading and writing the structured data, the semi-structured data, the text data and the image data through an OPC-UA protocol;
storing the definition, data value and state value of the specific Item, and generating an object of an OPC Item class;
storing Group information consisting of objects of a plurality of OPC Item classes, and returning OPC Item class objects generated by the OPC Item modules;
and obtaining other objects and initial classes of the service except the OPC Group objects by utilizing the OPC starting server, and returning the OPC Group objects generated by the OPC Group module.
Further, in the industrial data acquisition process, an MQTT communication protocol between a sensor for acquiring the industrial data and a server is established, and the communication transmission of the industrial data acquired by the sensor is performed, including:
appointing the sensor to transmit the appointed information of the machine in the form of Topic and Payload; wherein, Topic is equivalent to data type, Payload is equivalent to data content;
receiving data sent by the sensor, and sending the received data to a subscription module;
associating with the session; wherein, after being connected with the MQTT server, each sensor is regarded as a session for representing state interaction between the sensor and the server; a session contains multiple subscriptions, each subscription in each session having a different topic filter.
Further, a text-based protocol is established between the client and the server, and transmission of text data, video data, image data and audio data is carried out, including:
acquiring a transmission mode provided by a server;
acquiring description information of a session to be established between the client and the server;
establishing a session between the client and the server, and establishing a data transmission mode;
the control client initiates a request for closing the session to the server;
the control client initiates a request for playing the related data content to the server.
Further, the obtaining the transmission mode provided by the server includes:
calculating the maximum value of the capacity of the server, which can be called in the residual energy, by using formula (1), wherein the formula (1) is;
wherein G ismaxRepresenting the maximum value of the capacity of the server which can be called in the residual energy, and n representing the number of the servers; c represents a constant, C ═ 1.5; hmaxRepresenting the maximum bandwidth required in the data transmission, HminRepresents the minimum bandwidth used in data transmission;
designating a server according to the maximum capability value of the server;
calculating the data volume to be transmitted by the client by using a formula (2) to obtain a data transmission quantity value;
wherein M represents a data transmission magnitude; m isiThe data transmission quantity when the server transmits the ith data file of the client is represented, and k represents the number of files contained in the data transmitted by the client;
describing data information to be transmitted when the client requests to establish a session to obtain description information, and determining a transmission mode between the client and the server by combining the description information and a data transmission quantity value as a response.
A system for collection of mass industrial data, the system comprising:
the data acquisition module is used for acquiring digital data, document data, video data, image data, audio data, structured data and semi-structured data;
the OPC-UA protocol module is used for establishing data transmission interaction of the structured data, the semi-structured data, the text data and the image data by utilizing an OPC-UA protocol;
the MQTT protocol module is used for establishing an MQTT communication protocol between a sensor for acquiring the industrial data and a server in the industrial data acquisition process and carrying out communication transmission on the industrial data acquired by the sensor;
the RTSP module is used for establishing a text-based protocol between the client and the server and transmitting text data, video data, image data and audio data;
and the data preprocessing module is used for carrying out data cleaning, data coding and decoding and data integration on the acquired data.
The data cleaning is to convert data which is not in accordance with standards, such as missing data and noise data mixed in the data, into data which is in accordance with standards; the data coding and decoding are used for reducing data dimensionality; data integration refers to the integration of distributed heterogeneous data sources that are related to each other, so that users can access the data sources in a transparent manner.
Further, the OPC-UA protocol module includes:
the interface definition module is used for self-defining an interface and reading the structured data, the semi-structured data, the text data and the image data which are acquired by the data acquisition module by using the defined interface;
the read-write module is used for performing read-write operation on the structured data, the semi-structured data, the text data and the image data through an OPC-UA protocol;
the OPC Item module is used for storing the definition, the data value and the state value of the concrete Item and generating an object of an OPC Item class;
the OPC Group module is used for storing Group information consisting of objects of a plurality of OPC Item classes and returning the OPC Item class objects generated by the OPC Item module;
and the OPC Server module is used for acquiring other objects and initial classes of services except the OPC Group objects by utilizing the OPC starting Server and returning the OPC Group objects generated by the OPC Group module.
Wherein, OPC Item refers to the object related to the management process variable, OPC Group refers to the management process variable; the OPC Server refers to an OPC service interface; OPC refers to OLE for Process Control, an interface standard established for communication between industrial Control system applications.
Further, the MQTT protocol module includes:
the information transmission module is used for appointing the sensor to transmit the appointed information of the machine in the forms of Topic and Payload; wherein, Topic is equivalent to data type, Payload is equivalent to data content;
the MQTT server is used for receiving the data sent by the sensor and sending the received data to the subscription module;
a subscription module for associating with a session;
wherein, after being connected with the MQTT server, each sensor is regarded as a session for representing state interaction between the sensor and the server; a session contains multiple subscriptions, each subscription in each session having a different topic filter.
Further, the RTSP protocol module includes:
the method acquisition module is used for acquiring the transmission mode provided by the server;
the information acquisition module is used for acquiring the description information of the session to be established between the client and the server;
the request establishing module is used for establishing a session between the client and the server and establishing a data transmission mode;
a closing request module, configured to control the client to initiate a request for closing a session to the server;
and the playing request module is used for controlling the client to initiate a request for playing the related data content to the server.
Further, the method acquisition module comprises:
the capacity acquisition module is used for calculating the maximum capacity value of the server, which can be called in the residual energy, by using the formula (1); wherein, the formula (1) is:
wherein G ismaxRepresenting presence of a serverThe maximum value of the called capacity in the residual energy, wherein n represents the number of servers; c represents a constant, C ═ 1.5; hmaxRepresenting the maximum bandwidth required in the data transmission, HminRepresents the minimum bandwidth used in data transmission;
the server distribution module is used for appointing a server according to the maximum capacity value of the server;
the data volume acquisition module is used for calculating the data volume to be transmitted by the client by using a formula (2) to obtain a data transmission quantity value;
wherein M represents a data transmission magnitude; m isiThe data transmission quantity when the server transmits the ith data file of the client is represented, and k represents the number of files contained in the data transmitted by the client;
and the server method determining module is used for describing the data information to be transmitted when the client requests to establish the session to obtain the description information, and determining the transmission mode between the client and the server by combining the description information and the data transmission quantity value as a response.
The invention has the beneficial effects that:
according to the acquisition method and the acquisition system for the mass industrial data, the accuracy and the success rate in the data communication process are improved to a great extent, specific types of data are transmitted in a targeted mode through a plurality of protocols, the data transmission efficiency is effectively improved, and the situation that the transmission efficiency is low due to the large data volume is prevented. Meanwhile, by processing the transmitted data, the data which is invalid or has noise can be effectively restored, and the data transmission quality is improved. On the other hand, the appointed server transmits the related data according to the judgment of the transmission capability of the server, so that the workload processing capability and the data transmission of the server can be balanced, and the server is prevented from being paralyzed to influence the data transmission quality and efficiency.
Drawings
FIG. 1 is a flow chart of the method of the present invention;
fig. 2 is a system block diagram of the system of the present invention.
Detailed Description
The preferred embodiments of the present invention will be described in conjunction with the accompanying drawings, and it will be understood that they are described herein for the purpose of illustration and explanation and not limitation.
The invention provides a method and a system for acquiring massive industrial data, which are used for solving the problems of high transmission error rate and failure rate of acquired data, and the method comprises the following steps:
a method for acquiring massive industrial data, as shown in fig. 1, the method comprising:
s1, collecting digital data, document data, video data, image data, audio data, structured data and semi-structured data;
s2, establishing data transmission interaction of the structured data, the semi-structured data, the text data and the image data by utilizing an OPC-UA protocol;
s3, in the industrial data acquisition process, establishing an MQTT communication protocol between a sensor for acquiring the industrial data and a server, and carrying out communication transmission on the industrial data acquired by the sensor;
s4, establishing a text-based protocol between the client and the server, and transmitting text data, video data, image data and audio data;
and S5, performing data cleaning, data coding and decoding and data integration on the acquired data.
The working principle of the scheme is as follows: firstly, acquiring digital data, document data, video data, image data, audio data, structured data and semi-structured data; then, establishing data transmission interaction of the structured data, the semi-structured data, the text data and the image data by utilizing an OPC-UA protocol; then, in the industrial data acquisition process, establishing an MQTT communication protocol between a sensor for acquiring the industrial data and a server, and carrying out communication transmission on the industrial data acquired by the sensor; establishing a text-based protocol between a client and a server, and transmitting text data, video data, image data and audio data; and finally, carrying out data cleaning, data coding and decoding and data integration on the acquired data.
The technical effects of the scheme are as follows: the accuracy and the success rate in the data communication process are improved to a great extent, specific types of data are transmitted in a targeted mode through a plurality of protocol modes, the data transmission efficiency is effectively improved, and the situation that the transmission efficiency is low due to large data volume is prevented. Meanwhile, by processing the transmitted data, the data which is invalid or has noise can be effectively restored, and the data transmission quality is improved. On the other hand, the appointed server transmits the related data according to the judgment of the transmission capability of the server, so that the workload processing capability and the data transmission of the server can be balanced, and the server is prevented from being paralyzed to influence the data transmission quality and efficiency.
According to one embodiment of the present invention, the establishing data transmission interaction of the structured data, the semi-structured data, the text data and the image data by using the OPC-UA protocol includes:
s201, an interface definition module, which is used for self-defining an interface and reading the structured data, the semi-structured data, the text data and the image data which are acquired by the data acquisition module by using the defined interface;
s202, performing read-write operation on the structured data, the semi-structured data, the text data and the image data through an OPC-UA protocol;
s203, storing the definition, data value and state value of the specific Item, and generating an object of an OPC Item class;
s204, storing Group information consisting of a plurality of objects of the OPC Item class, and returning the OPC Item class objects generated by the OPC Item module;
s205, utilizing the OPC starting server to obtain other objects and initial classes of the service except the OPC Group objects, and returning the OPC Group objects generated by the OPC Group module.
The working principle of the scheme is as follows: the industrial Internet of things monitors the equipment condition; the communication between the data sources such as PLC (programmable logic controller) and the OPC collector is carried out by using OPC-UA protocol. The OPC-UA protocol provides a standard data source and data description scheme, according to which digital data, text data and graphic data in the industrial internet of things can be transmitted and interacted according to rules. And the client data is read through an OPC-UA custom interface. When OPC-UA is used for reading and writing, a Custom interface and an Automation interface are needed. Three key objects are involved in Custom interface development: OpcServer, OpcRoup, OpcItem. The processing procedure of these three key objects is: storing the definition, data value and state value of the specific Item, and generating an object of an OPC Item class; storing Group information consisting of objects of a plurality of OPC Item classes, and returning OPC Item class objects generated by the OPC Item modules; and obtaining other objects and initial classes of services except the OPC Group objects by utilizing the OPC starting server, and returning the OPCGroup class objects generated by the OPC Group module.
The technical effects of the scheme are as follows: the transmission efficiency and the data transmission quality are improved, the phenomena of errors and failures in the data transmission process are avoided, and the accuracy and the success rate of data transmission are improved to a great extent.
In an embodiment of the present invention, in the industrial data acquisition process, establishing an MQTT communication protocol between a sensor for acquiring the industrial data and a server, and performing communication transmission of the industrial data acquired by the sensor includes:
s301, appointing the sensor to transmit the appointed information of the machine in the forms of Topic and Payload; wherein, Topic is equivalent to data type, Payload is equivalent to data content;
s302, receiving data sent by the sensor, and sending the received data to a subscription module;
s303, associating with the session;
wherein, after being connected with the MQTT server, each sensor is regarded as a session for representing state interaction between the sensor and the server; a session contains multiple subscriptions, each subscription in each session having a different topic filter.
The working principle of the scheme is as follows: firstly, appointing the sensor to transmit the appointed information of the machine in the form of Topic and Payload; wherein, Topic is equivalent to data type, Payload is equivalent to data content; then, S302, receiving the data sent by the sensor, and sending the received data to a subscription module; finally, a session is associated. In this embodiment, the message publishing mode of the MQTT protocol is:
the working state information of the sensor is sent to a server in a sending mode at most once;
sending data sampling information monitored by a sensor to a server in at least one sending mode;
the MQTT acquires the working fault and information change data of the sensor through a message subscription mode, and acquires the information of the sensor in at least one acquisition mode.
The technical effects of the scheme are as follows: data transmission between the industrial sensor for collecting data and the network is established through the MQTT protocol, and the success rate of data transmission can be effectively improved by combining the message issuing mode, so that the problem of data monitoring omission caused by unsuccessful output transmission is prevented.
One embodiment of the present invention establishes a text-based protocol between a client and a server, and performs transmission of text data, video data, image data, and audio data, including:
s401, obtaining a transmission mode provided by a server;
s402, obtaining the description information of the session to be established between the client and the server;
s403, establishing a session between the client and the server, and establishing a data transmission mode;
s404, the control client initiates a request for closing the session to the server;
s405, the control client initiates a request for playing the related data content to the server.
The working principle of the scheme is as follows: the RTSP is a text-based protocol, CRLF (carriage return and line feed) is used as an end symbol of each line, and custom parameters can be conveniently added in the using process of the RTSP, and packet capture analysis is convenient. From the message transmission direction, there are two types of RTSP messages: a request message and a response message. The request message refers to a request sent from the client to the server (there are also a small number of requests sent from the server to the client), and the response message refers to a response from the server to the client. In this embodiment, data transmission between the client and the server is established by obtaining a designated server transmission mode.
The technical effects of the scheme are as follows: the data transmission is carried out through the transmission mode provided by the server, the data transmission efficiency and the data compatibility between the server and the client can be improved, meanwhile, the server is appointed to transmit related data according to the judgment of the transmission capability of the server, the balance can be carried out between the workload processing capability of the server and the data transmission, and the server is prevented from being paralyzed to influence the data transmission quality and efficiency.
In an embodiment of the present invention, the obtaining a transmission mode provided by a server includes:
s4011, calculating a maximum value of the capability of the server, which can be called in the residual energy, by using a formula (1), wherein the formula (1) is;
wherein G ismaxRepresenting the maximum value of the capacity of the server which can be called in the residual energy, and n representing the number of the servers; c represents a constant, C ═ 1.5; hmaxRepresenting the maximum bandwidth required in the data transmission, HminRepresents the minimum bandwidth used in data transmission;
s4012, appointing a server according to the maximum capacity value of the server;
s4013, calculating data volume to be transmitted by the client by using formula (2) to obtain data transmission quantity value;
wherein M represents a data transmission magnitude; m isiThe data transmission quantity when the server transmits the ith data file of the client is represented, and k represents the number of files contained in the data transmitted by the client;
s4014, data information to be transmitted when the client requests to establish a session is described to obtain description information, and a transmission mode between the client and the server is determined by taking the description information and a data transmission quantity value as a response.
The working principle of the scheme is as follows: the method comprises the steps that a server used for data transmission is designated through the maximum capacity value capable of being called by the server, and the transmission mode between the client and the server is further determined according to the total amount of data to be transmitted.
The technical effects of the scheme are as follows: the server for data transmission is designated by the maximum capacity value which can be called by the server, so that the rationality of the application of the server can be improved, and the influence on the data transmission efficiency caused by the frequent occurrence of the server with over-high data transmission saturation is avoided; meanwhile, the maximum capacity of the server is calculated according to the formula (1), so that the accuracy of the maximum capacity calculation of the server can be effectively improved, and the efficiency and the accuracy of server distribution are improved. Meanwhile, the matching degree of the transmission mode and the data volume can be improved by determining the transmission mode in a mode of combining the description information and the data transmission quantity value, data transmission failure or error caused by mismatching of the transmission mode and the data transmission volume is avoided, and the data transmission accuracy and the data transmission success rate are improved to a great extent.
A system for acquiring mass industrial data, as shown in fig. 2, the system comprising:
the data acquisition module is used for acquiring digital data, document data, video data, image data, audio data, structured data and semi-structured data;
the OPC-UA protocol module is used for establishing data transmission interaction of the structured data, the semi-structured data, the text data and the image data by utilizing an OPC-UA protocol;
the MQTT protocol module is used for establishing an MQTT communication protocol between a sensor for acquiring the industrial data and a server in the industrial data acquisition process and carrying out communication transmission on the industrial data acquired by the sensor;
the RTSP module is used for establishing a text-based protocol between the client and the server and transmitting text data, video data, image data and audio data;
and the data preprocessing module is used for carrying out data cleaning, data coding and decoding and data integration on the acquired data.
The data cleaning is to convert data which is not in accordance with standards, such as missing data and noise data mixed in the data, into data which is in accordance with standards; the data coding and decoding are used for reducing data dimensionality; data integration refers to the integration of distributed heterogeneous data sources that are related to each other, so that users can access the data sources in a transparent manner.
The working principle of the scheme is as follows: acquiring digital data, document data, video data, image data, audio data, structured data and semi-structured data by using a data acquisition module; and then, respectively transmitting the acquired data of the specified data type according to different protocol modes of the module by an OPC-UA protocol module, an MQTT protocol module and an RTSP protocol module.
The technical effects of the scheme are as follows: the accuracy and the success rate in the data communication process are improved to a great extent, specific types of data are transmitted in a targeted mode through a plurality of protocol modes, the data transmission efficiency is effectively improved, and the situation that the transmission efficiency is low due to large data volume is prevented. Meanwhile, by processing the transmitted data, the data which is invalid or has noise can be effectively restored, and the data transmission quality is improved. On the other hand, the appointed server transmits the related data according to the judgment of the transmission capability of the server, so that the workload processing capability and the data transmission of the server can be balanced, and the server is prevented from being paralyzed to influence the data transmission quality and efficiency.
In an embodiment of the present invention, the OPC-UA protocol module includes:
the interface definition module is used for self-defining an interface and reading the structured data, the semi-structured data, the text data and the image data which are acquired by the data acquisition module by using the defined interface;
the read-write module is used for performing read-write operation on the structured data, the semi-structured data, the text data and the image data through an OPC-UA protocol;
the OPC Item module is used for storing the definition, the data value and the state value of the concrete Item and generating an object of an OPC Item class;
the OPC Group module is used for storing Group information consisting of objects of a plurality of OPC Item classes and returning the OPC Item class objects generated by the OPC Item module;
and the OPC Server module is used for acquiring other objects and initial classes of services except the OPC Group objects by utilizing the OPC starting Server and returning the OPC Group objects generated by the OPC Group module.
Wherein, OPC Item refers to the object related to the management process variable, OPC Group refers to the management process variable; the OPC Server refers to an OPC service interface; OPC refers to OLEforProcessControl, an interface standard established for communication between industrial control system applications.
The working principle of the scheme is as follows: firstly, an interface is defined by an interface definition module, and structured data, semi-structured data, text data and image data acquired by a data acquisition module are read by the defined interface; then, reading and writing the structured data, the semi-structured data, the text data and the image data by adopting a reading and writing module through an OPC-UA protocol; and finally, respectively processing three key objects, namely OpcServer, OpcRoup and OpcItem, by utilizing an OPC Item module, an OPC Group module and an OPCServer module.
The technical effects of the scheme are as follows: the transmission efficiency and the data transmission quality are improved, the phenomena of errors and failures in the data transmission process are avoided, and the accuracy and the success rate of data transmission are improved to a great extent.
In an embodiment of the present invention, the MQTT protocol module includes:
the information transmission module is used for appointing the sensor to transmit the appointed information of the machine in the forms of Topic and Payload; wherein, Topic is equivalent to data type, Payload is equivalent to data content;
the MQTT server is used for receiving the data sent by the sensor and sending the received data to the subscription module;
a subscription module for associating with a session;
wherein, after being connected with the MQTT server, each sensor is regarded as a session for representing state interaction between the sensor and the server; a session contains multiple subscriptions, each subscription in each session having a different topic filter.
The working principle of the scheme is as follows: utilizing an information transmission module to designate the sensor to transmit the designated information of the machine in the forms of Topic and Payload; wherein, Topic is equivalent to data type, Payload is equivalent to data content; then, receiving the data sent by the sensor by using an MQTT server, and sending the received data to a subscription module; and finally, associating with the session through a subscription module.
The technical effects of the scheme are as follows: data transmission between the industrial sensor for collecting data and the network is established through the MQTT protocol, and the success rate of data transmission can be effectively improved by combining the message issuing mode, so that the problem of data monitoring omission caused by unsuccessful output transmission is prevented.
In one embodiment of the present invention, the RTSP protocol module includes:
the method acquisition module is used for acquiring the transmission mode provided by the server;
the information acquisition module is used for acquiring the description information of the session to be established between the client and the server;
the request establishing module is used for establishing a session between the client and the server and establishing a data transmission mode;
a closing request module, configured to control the client to initiate a request for closing a session to the server;
and the playing request module is used for controlling the client to initiate a request for playing the related data content to the server.
The working principle of the scheme is as follows: firstly, respectively acquiring a transmission mode provided by a server and description information of a session to be established between a client and the server by a method acquisition module and an information acquisition module; then, establishing a session between the client and the server by using a request establishing module, and establishing a data transmission mode; and finally, the client respectively initiates a request for closing the session and a request for playing the related data content through the closing request module and the playing request module.
The technical effects of the scheme are as follows: the data transmission is carried out through the transmission mode provided by the server, the data transmission efficiency and the data compatibility between the server and the client can be improved, meanwhile, the server is appointed to transmit related data according to the judgment of the transmission capability of the server, the balance can be carried out between the workload processing capability of the server and the data transmission, and the server is prevented from being paralyzed to influence the data transmission quality and efficiency.
In an embodiment of the present invention, the method obtaining module includes:
the capacity acquisition module is used for calculating the maximum capacity value of the server, which can be called in the residual energy, by using the formula (1); wherein, the formula (1) is:
wherein G ismaxRepresenting the maximum value of the capacity of the server which can be called in the residual energy, and n representing the number of the servers; c represents a constant, C ═ 1.5; hmaxRepresenting the maximum bandwidth required in the data transmission, HminRepresents the minimum bandwidth used in data transmission;
the server distribution module is used for appointing a server according to the maximum capacity value of the server;
the data volume acquisition module is used for calculating the data volume to be transmitted by the client by using a formula (2) to obtain a data transmission quantity value;
wherein M represents a data transmission magnitude; m isiThe data transmission quantity when the server transmits the ith data file of the client is represented, and k represents the number of files contained in the data transmitted by the client;
and the server method determining module is used for describing the data information to be transmitted when the client requests to establish the session to obtain the description information, and determining the transmission mode between the client and the server by combining the description information and the data transmission quantity value as a response.
The working principle of the scheme is as follows:
step 1, calculating the maximum value of the capacity of the server, which can be called in the residual energy, by using a formula (1) through a capacity acquisition module, wherein the formula (1) is;
wherein G ismaxRepresenting the maximum value of the capacity of the server which can be called in the residual energy, and n representing the number of the servers; c represents a constant, C ═ 1.5; hmaxRepresenting the maximum bandwidth required in the data transmission, HminRepresents the minimum bandwidth used in data transmission;
step 2, appointing a server according to the maximum capacity of the server through a server distribution module;
step 3, calculating the data volume to be transmitted by the client by using a formula (2) by using a data volume acquisition module to obtain a data transmission quantity value;
wherein M represents a data transmission magnitude; m isiThe data transmission quantity when the server transmits the ith data file of the client is represented, and k represents the number of files contained in the data transmitted by the client;
and 4, describing data information to be transmitted when the client requests to establish the session by using a server method determining module to obtain description information, and determining a transmission mode between the client and the server by combining the description information and the data transmission quantity value as a response.
The technical effects of the scheme are as follows: the server for data transmission is designated by the maximum capacity value which can be called by the server, so that the rationality of the application of the server can be improved, and the influence on the data transmission efficiency caused by the frequent occurrence of the server with over-high data transmission saturation is avoided; meanwhile, the maximum capacity of the server is calculated according to the formula (1), so that the accuracy of the maximum capacity calculation of the server can be effectively improved, and the efficiency and the accuracy of server distribution are improved. Meanwhile, the matching degree of the transmission mode and the data volume can be improved by determining the transmission mode in a mode of combining the description information and the data transmission quantity value, data transmission failure or error caused by mismatching of the transmission mode and the data transmission volume is avoided, and the data transmission accuracy and the data transmission success rate are improved to a great extent.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. A method for acquiring massive industrial data is characterized by comprising the following steps:
acquiring digital data, document data, video data, image data, audio data, structured data and semi-structured data;
establishing data transmission interaction of the structured data, the semi-structured data, the text data and the image data by utilizing an OPC-UA protocol;
in the industrial data acquisition process, establishing an MQTT communication protocol between a sensor for acquiring the industrial data and a server, and carrying out communication transmission on the industrial data acquired by the sensor;
establishing a text-based protocol between a client and a server, and transmitting text data, video data, image data and audio data;
and carrying out data cleaning, data coding and decoding and data integration on the acquired data.
2. The acquisition method according to claim 1, wherein said establishing data transmission interaction of said structured data, semi-structured data, text data and image data using OPC-UA protocol comprises:
the interface definition module is used for self-defining an interface and reading the structured data, the semi-structured data, the text data and the image data which are acquired by the data acquisition module by using the defined interface;
reading and writing the structured data, the semi-structured data, the text data and the image data through an OPC-UA protocol;
storing the definition, data value and state value of the specific Item, and generating an object of an OPC Item class;
storing Group information consisting of objects of a plurality of OPC Item classes, and returning OPC Item class objects generated by the OPC Item modules;
and obtaining other objects and initial classes of the service except the OPC Group objects by utilizing the OPC starting server, and returning the OPC Group objects generated by the OPC Group module.
3. The method according to claim 1, wherein during the industrial data collection process, establishing an MQTT communication protocol between a sensor for collecting the industrial data and a server, and performing communication transmission of the industrial data collected by the sensor comprises:
appointing the sensor to transmit the appointed information of the machine in the form of Topic and Payload; wherein, Topic is equivalent to data type, Payload is equivalent to data content;
receiving data sent by the sensor, and sending the received data to a subscription module;
associating with the session; wherein, after being connected with the MQTT server, each sensor is regarded as a session for representing state interaction between the sensor and the server; a session contains multiple subscriptions, each subscription in each session having a different topic filter.
4. The acquisition method according to claim 1, wherein a text-based protocol is established between the client and the server, and the transmission of text data, video data, image data and audio data is performed, comprising:
acquiring a transmission mode provided by a server;
acquiring description information of a session to be established between the client and the server;
establishing a session between the client and the server, and establishing a data transmission mode;
the control client initiates a request for closing the session to the server;
the control client initiates a request for playing the related data content to the server.
5. The acquisition method according to claim 4, wherein the obtaining of the transmission mode provided by the server comprises:
calculating the maximum value of the capacity of the server, which can be called in the residual energy, by using formula (1), wherein the formula (1) is;
wherein G ismaxTo representThe maximum value of the capacity of the server which can be called in the residual energy, wherein n represents the number of the servers; c represents a constant, C ═ 1.5; hmaxRepresenting the maximum bandwidth required in the data transmission, HminRepresents the minimum bandwidth used in data transmission;
designating a server according to the maximum capability value of the server;
calculating the data volume to be transmitted by the client by using a formula (2) to obtain a data transmission quantity value;
wherein M represents a data transmission magnitude; m isiThe data transmission quantity when the server transmits the ith data file of the client is represented, and k represents the number of files contained in the data transmitted by the client;
describing data information to be transmitted when the client requests to establish a session to obtain description information, and determining a transmission mode between the client and the server by combining the description information and a data transmission quantity value as a response.
6. A system for acquiring massive industrial data, the system comprising:
the data acquisition module is used for acquiring digital data, document data, video data, image data, audio data, structured data and semi-structured data;
the OPC-UA protocol module is used for establishing data transmission interaction of the structured data, the semi-structured data, the text data and the image data by utilizing an OPC-UA protocol;
the MQTT protocol module is used for establishing an MQTT communication protocol between a sensor for acquiring the industrial data and a server in the industrial data acquisition process and carrying out communication transmission on the industrial data acquired by the sensor;
the RTSP module is used for establishing a text-based protocol between the client and the server and transmitting text data, video data, image data and audio data;
and the data preprocessing module is used for carrying out data cleaning, data coding and decoding and data integration on the acquired data.
7. The acquisition system according to claim 6, wherein said OPC-UA protocol module comprises:
the interface definition module is used for self-defining an interface and reading the structured data, the semi-structured data, the text data and the image data which are acquired by the data acquisition module by using the defined interface;
the read-write module is used for performing read-write operation on the structured data, the semi-structured data, the text data and the image data through an OPC-UA protocol;
the OPC Item module is used for storing the definition, the data value and the state value of the concrete Item and generating an object of an OPC Item class;
the OPC Group module is used for storing Group information consisting of objects of a plurality of OPC Item classes and returning the OPC Item class objects generated by the OPC Item module;
and the OPC Server module is used for acquiring other objects and initial classes of services except the OPC Group objects by utilizing the OPC starting Server and returning the OPC Group objects generated by the OPC Group module.
8. The acquisition system according to claim 6, wherein the MQTT protocol module comprises:
the information transmission module is used for appointing the sensor to transmit the appointed information of the machine in the forms of Topic and Payload; wherein, Topic is equivalent to data type, Payload is equivalent to data content;
the MQTT server is used for receiving the data sent by the sensor and sending the received data to the subscription module;
a subscription module for associating with a session;
wherein, after being connected with the MQTT server, each sensor is regarded as a session for representing state interaction between the sensor and the server; a session contains multiple subscriptions, each subscription in each session having a different topic filter.
9. The acquisition system according to claim 6, characterised in that said RTSP protocol module comprises:
the method acquisition module is used for acquiring the transmission mode provided by the server;
the information acquisition module is used for acquiring the description information of the session to be established between the client and the server;
the request establishing module is used for establishing a session between the client and the server and establishing a data transmission mode;
a closing request module, configured to control the client to initiate a request for closing a session to the server;
and the playing request module is used for controlling the client to initiate a request for playing the related data content to the server.
10. The acquisition system of claim 9, wherein the method acquisition module comprises:
the capacity acquisition module is used for calculating the maximum capacity value of the server, which can be called in the residual energy, by using the formula (1); wherein, the formula (1) is:
wherein G ismaxRepresenting the maximum value of the capacity of the server which can be called in the residual energy, and n representing the number of the servers; c represents a constant, C ═ 1.5; hmaxRepresenting the maximum bandwidth required in the data transmission, HminRepresents the minimum bandwidth used in data transmission;
the server distribution module is used for appointing a server according to the maximum capacity value of the server;
the data volume acquisition module is used for calculating the data volume to be transmitted by the client by using a formula (2) to obtain a data transmission quantity value;
wherein M represents a data transmission magnitude; m isiThe data transmission quantity when the server transmits the ith data file of the client is represented, and k represents the number of files contained in the data transmitted by the client;
and the server method determining module is used for describing the data information to be transmitted when the client requests to establish the session to obtain the description information, and determining the transmission mode between the client and the server by combining the description information and the data transmission quantity value as a response.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010477319.9A CN111596628B (en) | 2020-05-29 | 2020-05-29 | Acquisition method and acquisition system for mass industrial data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010477319.9A CN111596628B (en) | 2020-05-29 | 2020-05-29 | Acquisition method and acquisition system for mass industrial data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111596628A true CN111596628A (en) | 2020-08-28 |
| CN111596628B CN111596628B (en) | 2021-07-30 |
Family
ID=72185837
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010477319.9A Active CN111596628B (en) | 2020-05-29 | 2020-05-29 | Acquisition method and acquisition system for mass industrial data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111596628B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112947290A (en) * | 2021-05-16 | 2021-06-11 | 北京赛博联物科技有限公司 | Edge cloud cooperation-based equipment state monitoring method and system and storage medium |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006058983A (en) * | 2004-08-18 | 2006-03-02 | Yokogawa Electric Corp | Process data collector |
| CN101026633A (en) * | 2007-03-01 | 2007-08-29 | 江西省电力科学研究院 | IEC 61850 and OPC based heterogeneous power quality monitoring device integrating method and gateway |
| CN101719153A (en) * | 2009-12-09 | 2010-06-02 | 中国工商银行股份有限公司 | Device and method for carrying out transmission processing on massive data files |
| CN102520655A (en) * | 2011-12-05 | 2012-06-27 | 大连海事大学 | Marine engine room data acquisition and monitoring system based on OLE for process control (OPC) technology |
| US20140082192A1 (en) * | 2011-05-25 | 2014-03-20 | Huawei Technologies Co., Ltd. | Processing method, distribution server, client, and system for streaming media |
| CN105205687A (en) * | 2015-08-24 | 2015-12-30 | 浪潮通用软件有限公司 | Mass data acquisition method |
| CN105245410A (en) * | 2015-11-18 | 2016-01-13 | 四川神琥科技有限公司 | Data encoding and transmitting method |
| CN206077446U (en) * | 2016-08-30 | 2017-04-05 | 重庆钢铁集团电子有限责任公司 | Industrial real-time data acquisition system |
| CN106773978A (en) * | 2016-12-28 | 2017-05-31 | 武汉科技大学 | A kind of device data acquisition system based on OPC UA agreements |
| CN107437153A (en) * | 2017-08-14 | 2017-12-05 | 长沙变化率信息技术有限公司 | Underground pipe gallery big data visualized O＆M cloud platform |
| CN107992949A (en) * | 2017-11-09 | 2018-05-04 | 北京许继电气有限公司 | Industrial data analysis method and system |
| CN109302623A (en) * | 2018-11-01 | 2019-02-01 | 南京大学 | A kind of transmission method based on QoE model dynamic self-adapting video |
| CN109740908A (en) * | 2018-12-27 | 2019-05-10 | 广州联达信息科技有限公司 | A kind of collecting method of multi-data source, apparatus and system |
| CN111142487A (en) * | 2019-12-30 | 2020-05-12 | 浪潮通用软件有限公司 | Equipment data acquisition system based on OPC UA unified architecture protocol |
-
2020
- 2020-05-29 CN CN202010477319.9A patent/CN111596628B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006058983A (en) * | 2004-08-18 | 2006-03-02 | Yokogawa Electric Corp | Process data collector |
| CN101026633A (en) * | 2007-03-01 | 2007-08-29 | 江西省电力科学研究院 | IEC 61850 and OPC based heterogeneous power quality monitoring device integrating method and gateway |
| CN101719153A (en) * | 2009-12-09 | 2010-06-02 | 中国工商银行股份有限公司 | Device and method for carrying out transmission processing on massive data files |
| US20140082192A1 (en) * | 2011-05-25 | 2014-03-20 | Huawei Technologies Co., Ltd. | Processing method, distribution server, client, and system for streaming media |
| CN102520655A (en) * | 2011-12-05 | 2012-06-27 | 大连海事大学 | Marine engine room data acquisition and monitoring system based on OLE for process control (OPC) technology |
| CN105205687A (en) * | 2015-08-24 | 2015-12-30 | 浪潮通用软件有限公司 | Mass data acquisition method |
| CN105245410A (en) * | 2015-11-18 | 2016-01-13 | 四川神琥科技有限公司 | Data encoding and transmitting method |
| CN206077446U (en) * | 2016-08-30 | 2017-04-05 | 重庆钢铁集团电子有限责任公司 | Industrial real-time data acquisition system |
| CN106773978A (en) * | 2016-12-28 | 2017-05-31 | 武汉科技大学 | A kind of device data acquisition system based on OPC UA agreements |
| CN107437153A (en) * | 2017-08-14 | 2017-12-05 | 长沙变化率信息技术有限公司 | Underground pipe gallery big data visualized O＆M cloud platform |
| CN107992949A (en) * | 2017-11-09 | 2018-05-04 | 北京许继电气有限公司 | Industrial data analysis method and system |
| CN109302623A (en) * | 2018-11-01 | 2019-02-01 | 南京大学 | A kind of transmission method based on QoE model dynamic self-adapting video |
| CN109740908A (en) * | 2018-12-27 | 2019-05-10 | 广州联达信息科技有限公司 | A kind of collecting method of multi-data source, apparatus and system |
| CN111142487A (en) * | 2019-12-30 | 2020-05-12 | 浪潮通用软件有限公司 | Equipment data acquisition system based on OPC UA unified architecture protocol |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112947290A (en) * | 2021-05-16 | 2021-06-11 | 北京赛博联物科技有限公司 | Edge cloud cooperation-based equipment state monitoring method and system and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111596628B (en) | 2021-07-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11252115B2 (en) | Conversation interface agent for manufacturing operation information | |
| US10324983B2 (en) | Interactive visualizations for a recurrent neural network | |
| US20170279874A1 (en) | Translation of messages using sensor-specific and unified protocols | |
| CN111708667B (en) | Business edge calculation method and system | |
| US7142929B2 (en) | Process data management | |
| CN111142487A (en) | Equipment data acquisition system based on OPC UA unified architecture protocol | |
| CN101902505A (en) | Distributed DNS inquiry log real-time statistic device and method thereof | |
| CN112969058B (en) | Industrial video real-time supervision platform and method with cloud storage function | |
| CN111596628B (en) | Acquisition method and acquisition system for mass industrial data | |
| KR102345082B1 (en) | Cloud based iec61850 information processing method | |
| KR20210098219A (en) | Advanced Operator Method for Manufacturing Equipment System using Broker based OPC UA PubSub | |
| CN114024801A (en) | Industrial edge computing service gateway | |
| CN113190524A (en) | Industrial big data acquisition method and system | |
| US9858302B1 (en) | Management of streaming data | |
| CN109392192B (en) | Method and system for device connection of SCADA system for fast communication | |
| CN116506300A (en) | Website traffic data statistics method and system | |
| CN115695090A (en) | Data transmission method, internet of things gateway equipment and computer readable storage medium | |
| CN115391429A (en) | Time sequence data processing method and device based on big data cloud computing | |
| CN112486706B (en) | Internet of things local equipment linkage method based on MQTT message driving mechanism | |
| CN112579675A (en) | Data processing method and device | |
| CN112564984A (en) | Distributed safe operation and maintenance method of Internet of things based on big data | |
| CN113988051B (en) | Vehicle data processing system | |
| CN109257257B (en) | Main and sub station wide area message transmission method and system | |
| JP4756130B2 (en) | Data collection management system | |
| Chen et al. | An OpenDDS cross-platform data exchange module for cloud-edge-based industrial internet of things |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |