CN114039965B - High-speed data acquisition system and method based on edge computing technology - Google Patents
High-speed data acquisition system and method based on edge computing technology Download PDFInfo
- Publication number
- CN114039965B CN114039965B CN202010702443.0A CN202010702443A CN114039965B CN 114039965 B CN114039965 B CN 114039965B CN 202010702443 A CN202010702443 A CN 202010702443A CN 114039965 B CN114039965 B CN 114039965B
- Authority
- CN
- China
- Prior art keywords
- data
- module
- edge
- uploading
- pushing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000005516 engineering process Methods 0.000 title claims abstract description 25
- 235000019504 cigarettes Nutrition 0.000 claims abstract description 108
- 238000012545 processing Methods 0.000 claims abstract description 68
- 230000008569 process Effects 0.000 claims abstract description 41
- 230000010354 integration Effects 0.000 claims abstract description 25
- 230000005540 biological transmission Effects 0.000 claims abstract description 15
- 238000013474 audit trail Methods 0.000 claims abstract description 9
- 238000004140 cleaning Methods 0.000 claims abstract description 9
- 238000003860 storage Methods 0.000 claims description 48
- 238000007726 management method Methods 0.000 claims description 15
- 238000001514 detection method Methods 0.000 claims description 8
- 238000007405 data analysis Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 4
- 238000012937 correction Methods 0.000 claims description 4
- 238000013500 data storage Methods 0.000 claims description 4
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 230000008901 benefit Effects 0.000 abstract description 4
- 238000004519 manufacturing process Methods 0.000 description 7
- 238000003384 imaging method Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000002688 persistence Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000000779 smoke Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/10—Network architectures or network communication protocols for network security for controlling access to devices or network resources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/06—Notations for structuring of protocol data, e.g. abstract syntax notation one [ASN.1]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Storage Device Security (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a high-speed data acquisition system and method based on an edge computing technology, comprising a cigarette making machine shared file, an edge side upper computer system, an Internet of things platform and a database; the shared file of the cigarette making machine is used for inputting real-time data of cigarettes of the cigarette making machine; the edge side upper computer system is used for acquiring real-time cigarette data in a shared file of the cigarette making machine, automatically processing failed uploading data through data pushing, and uploading the pushed data to the Internet of things platform through a custom protocol after cleaning specification, audit trail, data splitting and integration are carried out on the pushed data; and the Internet of things platform instantiates the real-time cigarette data and stores the data into a database. The invention can collect the process quality data of specific single cigarettes in the whole batch; the method has the advantages that the full automation of data uploading is realized by utilizing the edge computing technology, the data transmission efficiency is improved, and the method has the advantage of instantaneity in the aspect of data processing, so that the real-time processing and execution of local business can be better supported.
Description
Technical Field
The invention belongs to the field of industrial Internet of things and edge computing, and particularly relates to a high-speed data acquisition system and method based on an edge computing technology.
Background
With rapid progress in internet technology, information digital technology and artificial intelligence technology, the internet of things and intelligent manufacturing have been widely popularized. The release of industrial Internet of things white paper in China indicates that the on-demand execution of the manufacturing process and the reasonable optimization of the manufacturing process are realized through the network interconnection, the data intercommunication and the system interoperation of industrial resources, so that the efficient utilization of the resources is achieved, and a service-driven novel industrial ecological system is constructed. Edge computing is an internet of things oriented distributed computing infrastructure that extends computing power and data analysis applications to the "edges" of the network, enabling clients to analyze and manage data locally, thereby obtaining immediate insight through joins.
Today, the cigarette making machines in the production workshops are divided into low-speed cigarette making machines pt70/pt1-8 with the speed of 7000/8500 cigarettes/min and high-speed cigarette making machines pt2-2/m5/m8 with the speed of 16000/15000/20000 cigarettes/min. The data acquisition unit time of the high-speed cigarette making machine has huge data volume, most traditional data acquisition methods only sample and detect cigarettes in the whole batch or only acquire average values of the cigarettes, the data acquisition of single cigarettes cannot be performed finely, and the acquisition speed and the uploading speed cannot meet the requirement of real-time intelligent control.
Disclosure of Invention
The invention aims to provide a high-speed data acquisition system and a method based on the edge computing technology, which are used for realizing full automation of data uploading, avoiding manual detection of system data uploading failure, realizing high data transmission efficiency, solving the problems of data acquisition quality, data authenticity and instantaneity in the intelligent control process of a high-speed cigarette machine and further improving the cigarette process quality.
In order to achieve the above purpose, the technical scheme adopted by the invention is as follows:
a high-speed data acquisition system based on an edge computing technology comprises a cigarette making machine shared file, an edge side upper computer system, an Internet of things platform and a database; the method is characterized in that: the cigarette making machine sharing file is used for inputting real-time data of cigarettes of the cigarette making machine; the edge side upper computer system is used for acquiring real-time cigarette data in the shared file of the cigarette making machine, automatically processing uploading failure data through data pushing, and uploading the data to the Internet of things platform through a custom protocol after cleaning specification, audit trail, data splitting and integration are carried out on the pushed data; the Internet of things platform instantiates the real-time cigarette data and stores the data in a database; the database is used for providing data storage service;
the edge side upper computer system comprises a cache module, a data pushing module, an edge storage module, a failure data processing module, an edge data integration module and a data security module;
the high-speed buffer module is connected with the shared file of the cigarette making machine and used for reading data in the shared file of the cigarette making machine;
the data pushing module is connected with the high-speed cache module and is used for triggering a pushing task when the high-speed cache reaches a threshold value, pushing real-time cigarette data to the edge data processing module, and if the data uploading is successful, calibrating the data uploading to be successful; if the data uploading fails, calibrating the data uploading failure, triggering a data local storage task, and uploading the data to the edge storage module;
the edge storage module is connected with the data pushing module and is used for classifying and recording the data uploaded by the data pushing module and storing the data in a local database;
the failure data processing module is connected with the edge storage module and the data pushing module and is used for polling and detecting pushing failure data stored by the edge storage module and uploading the pushing failure data to the data pushing module again;
the edge data processing module is connected with the data pushing module through an edge data interface and is used for carrying out data standardization, data analysis, cleaning and audit trail on real-time data pushed by the data pushing module according to a set processing rule; the edge data processing module is close to the equipment end and can process data at a millisecond speed;
the edge data integration module is connected with the edge data processing module and used for integrating and splitting the data processed by the edge data processing module according to equipment states, product information, process quality and process specifications;
the data security module is respectively connected with the cache module, the data pushing module, the edge storage module, the failure data processing module, the edge data processing module and the edge data integration module and is used for providing data encryption, decryption, hierarchical control access authority, identity authentication and log management services.
Preferably, the edge storage module may insert data in batches, and is divided into: the method comprises the steps of storing three states of unremitted data, uploading successful data and uploading failure data; the edge storage module can modify the three states in batches.
Preferably, the custom protocol adopts WebSocket communication protocol.
The high-speed data acquisition method based on the edge computing technology is based on the high-speed data acquisition system based on the edge computing technology, and is characterized by comprising the following steps of:
step 1: a high-speed buffer memory module in the edge side upper computer system reads 16000 cigarettes/min of single-cigarette millisecond-level cigarette real-time data of the cigarette making machine from the shared file of the cigarette making machine;
step 2: when the cache reaches a threshold value, the data pushing module triggers a pushing task to push the real-time data of the cigarettes to the edge data processing module, and if the data uploading is successful, the calibration data uploading is successful; if the data uploading fails, calibrating the data uploading failure, triggering a data local storage task, and uploading the data to an edge storage module; the edge storage module can insert data in batches and is divided into: the method comprises the steps of storing three states of unremitted data, uploading successful data and uploading failure data; the edge storage module can modify the three states in batches;
step 3: the failure data processing module judges uploading failure data marked by the edge storage module through polling detection, inquires quantitative uploading failure data, uploads the pushing failure data to the data pushing module again, and transmits the pushing failure data to the edge data processing module through an edge data interface, and if the data uploading is successful, the uploaded data is deleted; if the data uploading fails, writing the error record into an error log, waiting for a set time, polling the query data and uploading again;
step 4: the edge data processing module is connected with the data pushing module through an edge data interface, acquires real-time data pushed by the data pushing module, and performs data specification, data analysis, cleaning and audit trail according to a set processing rule; defining time, numerical value and character of cigarettes in source data and field format constraint, decomposing multi-source heterogeneous data according to service requirements, realizing replacement of invalid data and missing data, marking abnormal data and tracking in real time;
step 5: the edge data integration module integrates and splits the data processed by the edge data processing module according to the equipment state, the product information, the process quality and the process specification, and transmits the processed data to the Internet of things platform through a custom protocol;
step 6: the Internet of things platform instantiates the real-time cigarette data, integrates the information such as the operation data of the cigarette making machine, the quality data of the cigarette making process, the quality detection data, the equipment correction data and the like into a whole, carries out digital modeling around the process parameters by taking the cigarette making machine equipment as a center, sorts the data information according to the process flow and the business logic relation, corresponds to the edge data integration module, and stores the data into a database.
Preferably, the cache module in the edge side upper computer system in step 1 reads 16000 cigarettes/min of real-time data of single cigarettes in millisecond level of the cigarette making machine from the shared file of the cigarette making machine, and the method comprises the following steps: the high-speed buffer memory module builds monitoring on shared files of the cigarette making machine, and starts reading data by opening the latest data file, adding new data in the file into a queue to be uploaded, uploading the whole batch of data and emptying the queue to be uploaded after the accumulated time of the data exceeds one second since the last uploading; when new data is not read from the file, firstly checking and uploading the data accumulated before the queue to be uploaded; and in the uploading process, monitoring generation of new data in the file in real time, and if no new data is generated, waiting for 100 milliseconds to read the data and continuing to perform subsequent loop judgment.
Preferably, in step 1, the edge side upper computer system includes a data security module, where the data security module is respectively connected to the cache module, the data pushing module, the edge storage module, the failure data processing module, the edge data processing module, and the edge data integration module, and is used to provide encryption, decryption, hierarchical control access rights, identity authentication, and log management services for data.
Preferably, the data security module includes:
edge data security: providing an underlying database SHA security management token and SSL transmission encryption by using a TEE trusted execution environment, and combining the security management token with SSL transmission encryption on software and hardware so that the data has security and reliability in terms of storage and transmission;
identity authentication: the edge side equipment comprises a plurality of functional modules, the identity authentication function provides mutual authentication of modules among different trust domains, and the identity authentication function comprises an identity authentication problem in a single domain and is used for solving the identity allocation problem of each functional module;
access control: the module controls the access process of the resource in real time based on the business logic, sets the authority of reading, writing, updating, executing service and the like between the modules, and supports fine granularity division between different modules;
security log: and supporting logs generated by various modules, databases, various application services and the like in the edge side equipment, and carrying out security level classification and management on the system events.
The beneficial effects of the invention are as follows:
the process quality data of specific single cigarettes in the whole batch can be collected by depending on the reading speed of the high-speed cache module; by utilizing the edge computing technology, the full automation of data uploading is realized by integrating the services of real-time data pushing, edge data storage, processing, failure data re-uploading and the like, the problem of failure data uploading of a system is not required to be detected manually, and the data transmission efficiency is improved.
The edge data processing module defines a plurality of data processing rules established aiming at the characteristics of cigarettes, provides a standard data format, analyzes multi-source heterogeneous data, can process data at a millisecond speed, has the advantage of real-time performance in terms of data processing, and can better support real-time processing and execution of local business.
And the edge data integration module integrates and splits the process quality and the business logic of the cigarette product. The edge side integrated data can slow down the pressure of mass data and network traffic, so that the millisecond-level transmission from the equipment to the platform of the Internet of things is achieved.
The problems of data acquisition quality, data authenticity and instantaneity in the intelligent control process are solved, and further the cigarette process quality is improved.
Drawings
FIG. 1 is a block diagram of a high-speed data acquisition system based on edge computing technology of the present invention;
FIG. 2 is a schematic diagram of a data push mechanism according to the present invention;
fig. 3 is a high-speed reading flow chart of the present invention.
Detailed Description
The invention is further described below with reference to the accompanying drawings and examples:
example 1
As shown in fig. 1, 2 and 3, the invention provides a high-speed data acquisition system based on an edge computing technology, which comprises a cigarette making machine shared file 1, an edge side upper computer system 2, an internet of things platform 3 and a database 4; the edge side upper computer system 2 comprises a cache module 5, a data pushing module 6, an edge storage module 8, a failure data processing module 7, an edge data processing module 10, an edge data integration module 11 and a data security module 12;
the cigarette making machine sharing file 1 is used for inputting real-time data of cigarettes of the cigarette making machine; the high-speed buffer module 5 is connected with the cigarette making machine shared file 1 and is used for reading data in the cigarette making machine shared file 1; the data pushing module 6 is connected with the cache module 5, and is used for triggering a pushing task when the cache reaches a threshold value, pushing real-time cigarette data to the edge data processing module 10 by the data pushing module 6, and if the data uploading is successful, calibrating that the data uploading is successful; if the data uploading fails, calibrating the data uploading failure, triggering a data local storage task, and uploading the data to an edge storage module 8;
the edge storage module 8 is connected with the data pushing module 6 and is used for classifying and recording the data uploaded by the data pushing module 6 and storing the data in a local database; the edge storage module 8 can insert data in batches and is divided into: the method comprises the steps of storing three states of unremitted data, uploading successful data and uploading failure data; the edge storage module 8 may modify the three states in batches.
The failure data processing module 7 is connected with the edge storage module 8 and the data pushing module 6, and is used for polling and detecting pushing failure data stored by the edge storage module 8 and uploading the pushing failure data to the data pushing module 6 again;
the edge data processing module 10 is connected with the data pushing module 6 through an edge data interface 9 and is used for carrying out data standardization, data analysis, cleaning and audit trail on real-time data pushed by the data pushing module 6 according to a set processing rule; the edge data processing module 10 is close to the equipment end and can process data at a millisecond speed, so that the module has the advantage of real-time performance in terms of data processing, and the real-time processing and execution of local business can be better supported.
The edge data integration module 11 is connected with the edge data processing module 10, and is used for integrating and splitting the data processed by the edge data processing module 10 according to the equipment state, the product information, the process quality and the process specification; the data model is high in aggregation and low in coupling, the smoke data acquisition is more efficient, and the service logic is clearer; and the edge nodes are used for data integration, so that the pressure of mass data and network traffic is relieved, and the millisecond level transmission from the equipment to the platform of the Internet of things is achieved.
The data security module 12 is respectively connected with the cache module 5, the data pushing module 6, the edge storage module 8, the failure data processing module 7, the edge data processing module 10 and the edge data integration module 11, and is used for providing encryption, decryption, hierarchical control access authority, identity authentication and log management services for data.
The Internet of things platform 3 is connected with the edge data integration module 11 through a custom protocol 13, instantiates the acquired real-time data of cigarettes, builds an imaging model of a cigarette machine, the imaging model of the cigarette machine integrates information such as operation data of the cigarette machine, quality data of cigarette process, quality detection data, equipment correction data and the like, carries out digital modeling around process parameters by taking the equipment of the cigarette machine as a center, sorts the data information according to a process flow and a business logic relationship, and corresponds to the edge data integration module one by one; the imaging model is consistent with the state of the cigarette making machine in actual production, and data support is provided for intelligent control of the cigarette making machine in a digital modeling mode. Meanwhile, the internet of things platform 3 stores the data into the DSE database 4 through DSE persistence provider. The custom protocol 13 adopts WebSocket communication protocol.
The database 4 mainly provides data storage services, in order to ensure continuous availability and linear expandability, and cross multiple commercial servers without occurrence of fault points, a DataStax Enterprise (DSE) version of Cassandra (non-open source code/community version) is adopted, cassandra is an extensible NoSQL database, a distributed architecture consisting of 6 DSE nodes, and cigarette data is stored in segments, can be read or written from any node, and can be read and checked from any two backups during reading, so that flexible and rapid response time can be realized for mass data generated by a cigarette machine, and the system is also associated with Manufacturing Execution System (MES) data.
On the other hand, the invention provides a high-speed data acquisition method based on an edge computing technology, and the high-speed data acquisition system based on the edge computing technology comprises the following steps:
step 1: the high-speed buffer memory module 5 in the edge side upper computer system 2 reads 16000 cigarettes/min of single cigarette millisecond-level cigarette real-time data of the cigarette making machine from the shared file 1 of the cigarette making machine; comprising the following steps: the high-speed buffer memory module 5 establishes monitoring of the shared file 1 of the cigarette making machine, and starts reading data by opening the latest data file, adding new data in the file into a queue to be uploaded, uploading the whole batch of data and emptying the queue to be uploaded after the accumulated time of the data exceeds one second since the last uploading; when new data is not read from the file, firstly checking and uploading the data accumulated before the queue to be uploaded; and in the uploading process, monitoring generation of new data in the file in real time, and if no new data is generated, waiting for 100 milliseconds to read the data and continuing to perform subsequent loop judgment.
Step 2: when the cache reaches a threshold value, the data pushing module 6 triggers a pushing task to push the real-time cigarette data to the edge data processing module 10, and if the data uploading is successful, the calibration data uploading is successful; if the data uploading fails, calibrating the data uploading failure, triggering a data local storage task, and uploading the data to an edge storage module 8; the edge storage module 8 can insert data in batches and is divided into: the method comprises the steps of storing three states of unremitted data, uploading successful data and uploading failure data; the edge storage module can modify the three states in batches;
step 3: the failure data processing module 7 judges the uploading failure data marked by the edge storage module 8 through polling detection, inquires quantitative uploading failure data, uploads the pushing failure data to the data pushing module 6 again, and transmits the pushing failure data to the edge data processing module 10 through the edge data interface 9, and if the data uploading is successful, the uploaded data is deleted; if the data uploading fails, writing the error record into an error log, waiting for a set time, polling the query data and uploading again;
step 4: the edge data processing module 10 is connected with the data pushing module 6 through the edge data interface 9 to acquire real-time data pushed by the data pushing module 6, and the edge data processing module 10 defines a plurality of data processing rules established aiming at the characteristics of cigarettes and automatically performs data specification, data analysis, cleaning and audit trail according to the rules; defining time, numerical value and character of cigarettes in source data and field format constraint, decomposing multi-source heterogeneous data according to service requirements, realizing replacement of invalid data and missing data, marking abnormal data and tracking in real time;
step 5: the edge data integration module 11 integrates and splits the data processed by the edge data processing module 10 according to the equipment state, the product information, the process quality and the process specification, and transmits the processed data to the internet of things platform 3 through the custom protocol 13; the custom protocol 13 adopts WebSocket communication protocol.
Step 6: the Internet of things platform 3 instantiates the real-time data of cigarettes to construct an imaging model of a cigarette making machine, the imaging model of the cigarette making machine integrates the information such as the operation data of the cigarette making machine, the quality data of the cigarette making process, the quality detection data, the equipment correction data and the like, the digital modeling is carried out around the process parameters by taking the equipment of the cigarette making machine as the center, the data information is arranged according to the process flow and the business logic relationship, and the data corresponds to the edge data integration module 10 one by one; the imaging model is consistent with the state of the cigarette making machine in actual production, and data support is provided for intelligent control of the cigarette making machine in a digital modeling mode. Meanwhile, the internet of things platform 3 stores the data into the DSE database 4 through DSE persistence provider.
The edge computing device is usually located close to the user side or on the transmission path, and has a higher possibility of being invaded by a potential attacker, so the edge side upper computer system 2 further comprises a data security module 12, and the data security module 12 is respectively connected with the cache module 5, the data pushing module 6, the edge storage module 8, the failure data processing module 7, the edge data processing module 10 and the edge data integration module 11, and is used for providing encryption, decryption, hierarchical control access authority, identity authentication and log management services for data.
A data security module 12 comprising:
edge data security: providing an underlying database SHA security management token and SSL transmission encryption by using a TEE trusted execution environment, and combining the security management token with SSL transmission encryption on software and hardware so that the data has security and reliability in terms of storage and transmission;
identity authentication: the edge side equipment comprises a plurality of functional modules, the identity authentication function provides mutual authentication of modules among different trust domains, and the identity authentication function comprises an identity authentication problem in a single domain and is used for solving the identity allocation problem of each functional module;
access control: the module controls the access process of the resource in real time based on the business logic, sets the authority of reading, writing, updating, executing service and the like between the modules, and supports fine granularity division between different modules;
security log: and supporting logs generated by various modules, databases, various application services and the like in the edge side equipment, and carrying out security level classification and management on the system events.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the invention in any way, and any person skilled in the art may make modifications or alterations to the disclosed technical content to the equivalent embodiments. However, any simple modification, equivalent variation and variation of the above embodiments according to the technical substance of the present invention still fall within the protection scope of the technical solution of the present invention.
Claims (7)
1. A high-speed data acquisition system based on an edge computing technology comprises a cigarette making machine shared file, an edge side upper computer system, an Internet of things platform and a database; the method is characterized in that: the cigarette making machine sharing file is used for inputting real-time data of cigarettes of the cigarette making machine; the edge side upper computer system is used for acquiring real-time cigarette data in the shared file of the cigarette making machine, automatically processing uploading failure data through data pushing, and uploading the data to the Internet of things platform through a custom protocol after cleaning specification, audit trail, data splitting and integration are carried out on the pushed data; the Internet of things platform instantiates the real-time cigarette data and stores the data in a database; the database is used for providing data storage service;
the edge side upper computer system comprises a cache module, a data pushing module, an edge storage module, a failure data processing module, an edge data integration module and a data security module;
the high-speed buffer module is connected with the shared file of the cigarette making machine and used for reading data in the shared file of the cigarette making machine;
the data pushing module is connected with the high-speed cache module and is used for triggering a pushing task when the high-speed cache reaches a threshold value, pushing real-time cigarette data to the edge data processing module, and if the data uploading is successful, calibrating the data uploading to be successful; if the data uploading fails, calibrating the data uploading failure, triggering a data local storage task, and uploading the data to the edge storage module;
the edge storage module is connected with the data pushing module and is used for classifying and recording the data uploaded by the data pushing module and storing the data in a local database;
the failure data processing module is connected with the edge storage module and the data pushing module and is used for polling and detecting pushing failure data stored by the edge storage module and uploading the pushing failure data to the data pushing module again;
the edge data processing module is connected with the data pushing module through an edge data interface and is used for carrying out data standardization, data analysis, cleaning and audit trail on real-time data pushed by the data pushing module according to a set processing rule; the edge data processing module is close to the equipment end and can process data at a millisecond speed;
the edge data integration module is connected with the edge data processing module and used for integrating and splitting the data processed by the edge data processing module according to equipment states, product information, process quality and process specifications;
the data security module is respectively connected with the cache module, the data pushing module, the edge storage module, the failure data processing module, the edge data processing module and the edge data integration module and is used for providing data encryption, decryption, hierarchical control access authority, identity authentication and log management services.
2. The high-speed data acquisition system based on the edge computing technology of claim 1, wherein: the edge storage module can insert data in batches and is divided into: the method comprises the steps of storing three states of unremitted data, uploading successful data and uploading failure data; the edge storage module can modify the three states in batches.
3. The high-speed data acquisition system based on the edge computing technology of claim 1, wherein: the custom protocol adopts a WebSocket communication protocol.
4. A high-speed data acquisition method based on an edge computing technology and based on the high-speed data acquisition system based on the edge computing technology according to any one of claims 1 to 3, characterized by comprising the following steps:
step 1: a high-speed buffer memory module in the edge side upper computer system reads 16000 cigarettes/min of single-cigarette millisecond-level cigarette real-time data of the cigarette making machine from the shared file of the cigarette making machine;
step 2: when the cache reaches a threshold value, the data pushing module triggers a pushing task to push the real-time data of the cigarettes to the edge data processing module, and if the data uploading is successful, the calibration data uploading is successful; if the data uploading fails, calibrating the data uploading failure, triggering a data local storage task, and uploading the data to an edge storage module; the edge storage module can insert data in batches and is divided into: the method comprises the steps of storing three states of unremitted data, uploading successful data and uploading failure data; the edge storage module can modify the three states in batches;
step 3: the failure data processing module judges uploading failure data marked by the edge storage module through polling detection, inquires quantitative uploading failure data, uploads the pushing failure data to the data pushing module again, and transmits the pushing failure data to the edge data processing module through an edge data interface, and if the data uploading is successful, the uploaded data is deleted; if the data uploading fails, writing the error record into an error log, waiting for a set time, polling the query data and uploading again;
step 4: the edge data processing module is connected with the data pushing module through an edge data interface, acquires real-time data pushed by the data pushing module, and performs data specification, data analysis, cleaning and audit trail according to a set processing rule; defining time, numerical value and character of cigarettes in source data and field format constraint, decomposing multi-source heterogeneous data according to service requirements, realizing replacement of invalid data and missing data, marking abnormal data and tracking in real time;
step 5: the edge data integration module integrates and splits the data processed by the edge data processing module according to the equipment state, the product information, the process quality and the process specification, and transmits the processed data to the Internet of things platform through a custom protocol;
step 6: the Internet of things platform instantiates the real-time cigarette data, integrates the information such as the operation data of the cigarette making machine, the quality data of the cigarette making process, the quality detection data, the equipment correction data and the like into a whole, carries out digital modeling around the process parameters by taking the cigarette making machine equipment as a center, sorts the data information according to the process flow and the business logic relation, corresponds to the edge data integration module, and stores the data into a database.
5. The high-speed data acquisition method based on the edge computing technology as set forth in claim 4, wherein: the cache module in the edge side upper computer system in step 1 reads 16000 cigarettes/min single cigarette millisecond-level cigarette real-time data of the cigarette making machine from the shared file of the cigarette making machine, and the method comprises the following steps: the high-speed buffer memory module builds monitoring on shared files of the cigarette making machine, and starts reading data by opening the latest data file, adding new data in the file into a queue to be uploaded, uploading the whole batch of data and emptying the queue to be uploaded after the accumulated time of the data exceeds one second since the last uploading; when new data is not read from the file, firstly checking and uploading the data accumulated before the queue to be uploaded; and in the uploading process, monitoring generation of new data in the file in real time, and if no new data is generated, waiting for 100 milliseconds to read the data and continuing to perform subsequent loop judgment.
6. The high-speed data acquisition method based on the edge computing technology according to claim 5, wherein the method comprises the following steps: the edge side upper computer system in step 1 includes a data security module, where the data security module is respectively connected with the cache module, the data pushing module, the edge storage module, the failure data processing module, the edge data processing module, and the edge data integration module, and is used for providing encryption, decryption, hierarchical control access authority, identity authentication, and log management services for data.
7. The high-speed data acquisition method based on the edge computing technology as set forth in claim 6, wherein: the data security module comprises:
edge data security: providing an underlying database SHA security management token and SSL transmission encryption by using a TEE trusted execution environment, and combining the security management token with SSL transmission encryption on software and hardware so that the data has security and reliability in terms of storage and transmission;
identity authentication: the edge side equipment comprises a plurality of functional modules, the identity authentication function provides mutual authentication of modules among different trust domains, and the identity authentication function comprises an identity authentication problem in a single domain and is used for solving the identity allocation problem of each functional module;
access control: the module controls the access process of the resource in real time based on the business logic, sets the authority of reading, writing, updating, executing service and the like between the modules, and supports fine granularity division between different modules;
security log: and supporting logs generated by various modules, databases, various application services and the like in the edge side equipment, and carrying out security level classification and management on the system events.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010702443.0A CN114039965B (en) | 2020-07-21 | 2020-07-21 | High-speed data acquisition system and method based on edge computing technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010702443.0A CN114039965B (en) | 2020-07-21 | 2020-07-21 | High-speed data acquisition system and method based on edge computing technology |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114039965A CN114039965A (en) | 2022-02-11 |
| CN114039965B true CN114039965B (en) | 2024-02-13 |
Family
ID=80134065
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010702443.0A Active CN114039965B (en) | 2020-07-21 | 2020-07-21 | High-speed data acquisition system and method based on edge computing technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114039965B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104967651A (en) * | 2015-03-02 | 2015-10-07 | 深圳市腾讯计算机系统有限公司 | Data push, storage and downloading methods and devices based on CDN architecture |
| CN108984433A (en) * | 2017-06-05 | 2018-12-11 | 华为技术有限公司 | Cache data control method and equipment |
| CN109164776A (en) * | 2018-10-10 | 2019-01-08 | 江苏极熵物联科技有限公司 | A kind of distribution type data collection method towards industrial equipment |
| CN110417757A (en) * | 2019-07-10 | 2019-11-05 | 广州博依特智能信息科技有限公司 | A kind of industrial data storage system based on edge calculations gateway |
| CN110766926A (en) * | 2019-09-26 | 2020-02-07 | 重庆溢希恩真节能电力设备有限公司 | Wireless data acquisition and monitoring system for stamping equipment |
| KR102075791B1 (en) * | 2019-04-10 | 2020-03-02 | 주식회사 와이드티엔에스 | System For Prosessing Fast Data Using Linking IoT Device In Edge Computing |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200007409A1 (en) * | 2018-06-29 | 2020-01-02 | Electronics And Telecommunications Research Institute | INTELLIGENT INTERNET OF EVERYTHING (IoE) EDGE COMPUTIING SYSTEM FOR HIGH RELIABLE INTERNET OF THINGS (IoT) SERVICE |
-
2020
- 2020-07-21 CN CN202010702443.0A patent/CN114039965B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104967651A (en) * | 2015-03-02 | 2015-10-07 | 深圳市腾讯计算机系统有限公司 | Data push, storage and downloading methods and devices based on CDN architecture |
| CN108984433A (en) * | 2017-06-05 | 2018-12-11 | 华为技术有限公司 | Cache data control method and equipment |
| CN109164776A (en) * | 2018-10-10 | 2019-01-08 | 江苏极熵物联科技有限公司 | A kind of distribution type data collection method towards industrial equipment |
| KR102075791B1 (en) * | 2019-04-10 | 2020-03-02 | 주식회사 와이드티엔에스 | System For Prosessing Fast Data Using Linking IoT Device In Edge Computing |
| CN110417757A (en) * | 2019-07-10 | 2019-11-05 | 广州博依特智能信息科技有限公司 | A kind of industrial data storage system based on edge calculations gateway |
| CN110766926A (en) * | 2019-09-26 | 2020-02-07 | 重庆溢希恩真节能电力设备有限公司 | Wireless data acquisition and monitoring system for stamping equipment |
Non-Patent Citations (3)
| Title |
|---|
| Edge computing-based real-time passenger counting using a compact convolutional neural network;Biao Yang;《Neural Computing and Applications》;全文 * |
| 边缘计算在智慧制造领域的应用研究;蔡锴;《2019全国边缘计算学术研讨会论文集》;全文 * |
| 边缘计算在烟草工业中的架构与应用实践;曹喆;《工业控制计算机》;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114039965A (en) | 2022-02-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107577805B (en) | Business service system for log big data analysis | |
| CN107818150B (en) | Log auditing method and device | |
| US9679021B2 (en) | Parallel transactional-statistics collection for improving operation of a DBMS optimizer module | |
| CN110795257A (en) | Method, device and equipment for processing multi-cluster operation records and storage medium | |
| CN109308329A (en) | A kind of log collecting method and device based on cloud platform | |
| CN111858251B (en) | Data security audit method and system based on big data computing technology | |
| CN106250287A (en) | A kind of log information processing means | |
| US20120072589A1 (en) | Information Processing Apparatus and Method of Operating the Same | |
| CN114201540A (en) | Industrial multi-source data acquisition and storage system | |
| CN113486095A (en) | Civil aviation air traffic control cross-network safety data exchange management platform | |
| CN111048164A (en) | Medical big data long-term storage system | |
| CN114039965B (en) | High-speed data acquisition system and method based on edge computing technology | |
| CN110309206B (en) | Order information acquisition method and system | |
| CN109522349B (en) | Cross-type data calculation and sharing method, system and equipment | |
| CN116431324A (en) | Edge system based on Kafka high concurrency data acquisition and distribution | |
| CN116506300A (en) | Website traffic data statistics method and system | |
| CN112783920A (en) | Industrial Internet of things data real-time computing method and system based on data arrangement | |
| CN107566187B (en) | SLA violation monitoring method, device and system | |
| CN109684159A (en) | Method for monitoring state, device, equipment and the storage medium of distributed information system | |
| CN117112039B (en) | Transmission optimization system and operation method of data center | |
| CN115357657B (en) | Data processing method and device, computer equipment and storage medium | |
| CN111143280B (en) | Data scheduling method, system, device and storage medium | |
| US20240184801A1 (en) | Data discrepancy detection in a sensitive data replication pipeline | |
| CN114780118A (en) | Virtual equipment updating method and device | |
| CN117370063A (en) | Cloud server memory fault feature extraction method, system and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |