CN114546993A - Industrial sensor data processing method based on streaming processing - Google Patents
Industrial sensor data processing method based on streaming processing Download PDFInfo
- Publication number
- CN114546993A CN114546993A CN202210435330.8A CN202210435330A CN114546993A CN 114546993 A CN114546993 A CN 114546993A CN 202210435330 A CN202210435330 A CN 202210435330A CN 114546993 A CN114546993 A CN 114546993A
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- data processing
- sensor
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
- G06F16/212—Schema design and management with details for data modelling support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Abstract
The invention provides an industrial sensor data processing method based on streaming processing. The invention obtains sensor data to a data acquisition layer; the method comprises the steps that a protocol conversion layer is established on a data acquisition layer to adapt to sensors of various types and protocols; receiving data through a kafka message queue at a data processing layer and forwarding the data to a data processing engine; calculating and returning a result in real time according to a data model of relevant design in a data processing engine, and simultaneously storing original data in a time sequence database; the data processing engine directly sends the data after real-time processing to the time sequence database for storage, and simultaneously sends the data back to the message queue cluster for secondary consumption of other systems; the configuration information is stored in a relational database. When the data flow processing is carried out, the method can continuously and stably support huge data flow, does not reduce the performance of the whole system, improves the operation efficiency, meets the requirement of the real-time performance of the sensor data processing, and reduces the cost.
Description
Technical Field
The invention relates to the field of data processing, in particular to an industrial sensor data processing method based on stream processing.
Background
With the rise of intelligent factories, the rapid development of industrial sensors and internet of things enriches the acquisition types and acquisition means of sensor information and data, so that the data volume and the calculated amount to be processed are increased explosively, and the solution based on the prior internet of things can not meet the increasing demands of people gradually.
In order to adapt to the development change that the data volume and the calculated amount to be processed are increased explosively, the data processing capability is more and more emphasized, but the existing data analysis speed and efficiency are low, various sensor protocols, data frequency, data content and data volume are inconsistent, when the conventional data processing aims at simple logic processing, the conventional data processing adopts hardware operation to solve the problem, the efficiency is high, the cost is high, complex logic cannot be basically realized, the conventional data processing is limited by hardware and environmental problems, the flexibility and the expandability are lacked, the simulation debugging is troublesome, the technical parameters of the system cannot be dynamically adjusted in real time, the judgment is completely carried out by manual experience, and some faults also exist. Therefore, an industrial sensor data processing method based on streaming processing is needed to solve the above problems.
Disclosure of Invention
The invention aims to provide a stream processing-based industrial sensor data processing method, which aims to solve the problems that the existing data analysis is low in speed and efficiency, various sensor protocols, data frequencies, data contents and data quantity are inconsistent, the conventional data processing adopts hardware operation to solve the problem of high efficiency, high cost, difficulty in realizing complex logic basically, limitation of hardware and environmental problems, lack of flexibility and expandability, troublesome simulation and debugging, incapability of dynamically adjusting system technical parameters in real time, full dependence on manual experience judgment and single-point failure.
The invention provides an industrial sensor data processing method based on streaming processing, which comprises the following steps:
acquiring sensor data to a data acquisition layer;
the method comprises the steps that a protocol conversion layer is established on a data acquisition layer to adapt to sensors of various types and protocols;
receiving data through a kafka message queue at a data processing layer and forwarding the data to a data processing engine;
calculating and returning a result in real time according to a data model of relevant design in a data processing engine, and simultaneously storing original data in a time sequence database;
the data processing engine directly sends the data after real-time processing to the time sequence database for storage, and simultaneously sends the data back to the message queue cluster for secondary consumption of other systems;
the configuration information is stored in a relational database.
Further, the data processing engine, the time sequence database and the relational database are controlled by the control configuration cluster.
Further, the sensor includes: acceleration sensor, temperature sensor, humidity transducer, flow sensor, pressure sensor, XYZ three-way sensor.
Further, in the step of acquiring sensor data to the data acquisition layer, the sensor data is transmitted from the various collectors to the data acquisition layer through a wireless or wired network.
Further, in the step that the data processing engine calculates and returns the result in real time according to the data model of the relevant design, and simultaneously, the original data is stored in the time sequence database, the sensor data is processed in real time by introducing calculation and processing rules in the data processing engine, wherein the real-time processing of the sensor data comprises filtering abnormal values and gain amplification factors, and the result is stored in the time sequence database.
Further, in the event that current system resources tend to or are already saturated, the data processing engine may add worker nodes at any time without requiring a shutdown or shutdown.
Further, the step of storing configuration information in a relational database, the configuration information comprising: the method comprises the following steps of self node information required by flow control, upstream and downstream node information used for communication, branch flow information, a data processing model corresponding to a self node and an early warning rule.
The invention has the following beneficial effects: on one hand, the industrial sensor data processing method based on stream processing can continuously and stably support huge data flow during data stream processing, does not reduce the performance of the whole system, improves the operation efficiency, meets the requirement of the real-time property of sensor data processing, reduces the cost, can be seamlessly expanded and flexibly configured, and can simulate data in real time, dynamically adjust the sensor data, and gain or modulate the data in real time. On the other hand, the data processing layer can carry out persistence operation to persist the message to a disk, thereby greatly reducing the probability of data loss; distributed upgradeability in efficient combination with a data processing engine; the form of processing the service data greatly expands the data processing capacity of the system and simultaneously ensures the rapidity of calculation and storage.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a schematic flow chart of a method for processing industrial sensor data based on streaming processing according to the present invention;
fig. 2 is an application architecture diagram of an industrial sensor data processing method based on streaming processing according to the present invention.
Illustration of the drawings: 1-a wired sensor; 2-a wireless sensor; 3-a wired collector; 4-wireless acquisition gateway.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the specific embodiments of the present invention and the accompanying drawings. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. The technical solutions provided by the embodiments of the present invention are described in detail below with reference to the accompanying drawings.
Referring to fig. 1 to 2, a method for processing industrial sensor data based on streaming processing includes:
s101, acquiring sensor data to a data acquisition layer.
In this embodiment, the sensor data is transmitted from the various collectors to the data collection layer through a wireless or wired network. The sensor includes: acceleration sensor, temperature sensor, humidity transducer, flow sensor, pressure sensor, XYZ three-way sensor.
S102, a protocol conversion layer is established on a data acquisition layer to adapt to sensors of various models and protocols.
S103 accepts data at the data processing layer via the kafka message queue and forwards to the data processing engine.
S104, the data processing engine calculates according to the data model of the relevant design and returns the result in real time, and meanwhile, the original data is stored in a time sequence database.
In the embodiment, by introducing calculation and processing rules into the data processing engine, the sensor data is processed in real time, wherein the processing of the sensor data in real time comprises filtering abnormal values and gain amplification factors, and the result is stored in a time sequence database.
The data processing engine can introduce different judgment rules according to different sensor types, and can trigger the upper and lower limit alarm of the sensor in quasi-real time. If the temperature and pressure can be introduced into the system, the system is high in temperature and pressureThe high-speed alarm, acceleration and speed can be given by data processed by a calculation engine, such as an XYZ three-way sensor, after the actual spatial acceleration can be calculated, the alarm is given according to rules (such as the alarm can not be given in any direction of the XYZ three-axis, but the alarm is given in any direction
After the result is calculated, the spatial acceleration exceeds the alarm threshold value, so that an alarm is given).
The data processing engine can also dynamically adjust the judgment rule in real time according to parameters introduced by the peripheral environment, so as to achieve the effect of dynamic alarm. The upper and lower limits of the alarm for the sensor may also be different, depending on the time. In actual production, some devices sensitive to temperature can give a false alarm when the temperature difference changes in the morning and evening, the ambient temperature starts to rise as early as 8 points according to actual time, and the alarm threshold value can be correspondingly reduced. After 8 o' clock later, the ambient temperature begins to drop, and the alarm threshold value can be correspondingly increased. The system can acquire the real-time environment humidity data of a weather forecast website in real time aiming at the environment sensitive to humidity, and adjust corresponding alarm threshold values aiming at scenes with different humidity.
And S105, directly sending the data processed in real time to the time sequence database for storage by the data processing engine, and sending the data back to the message queue cluster for secondary consumption of other systems.
S106, the configuration information is stored in a relational database.
In this embodiment, the configuration information includes: the method comprises the following steps of self node information required by flow control, upstream and downstream node information used for communication, branch flow information, a data processing model corresponding to a self node and an early warning rule.
In this embodiment, the data processing engine, the time sequence database, and the relational database are controlled by a control configuration cluster. In the event that current system resources tend to or are already saturated, the data processing engine can add work nodes at any time without requiring downtime.
The invention provides an industrial sensor data processing method based on stream processing, which has the following working principle: the data are acquired by the wired sensor 1 and the wireless sensor 2 and stored in the data acquisition layer by the wired acquirer 3 and the wireless acquisition gateway 4. When the system is started, the configuration is read from the relational database and is simultaneously loaded into the data processing engine, and according to the configuration, the data processing engine starts corresponding algorithms and flows aiming at different configurations and different flows to process data. And during the operation of the system, if the configuration is changed or the process is changed, stopping the related task process, and starting the task process after the configuration and the process are read again.
An embodiment of the present invention further provides a storage medium, and a storage medium, where a computer program is stored in the storage medium, and when the computer program is executed by a processor, the computer program implements part or all of the steps in each embodiment of the streaming processing-based industrial sensor data processing method provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a Read-only memory (ROM) or a Random Access Memory (RAM).
Those skilled in the art will clearly understand that the method in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
Claims (7)
1. An industrial sensor data processing method based on streaming processing is characterized by comprising the following steps:
acquiring sensor data to a data acquisition layer;
a protocol conversion layer is established on a data acquisition layer to adapt to sensors of various types and protocols;
receiving data through a kafka message queue at a data processing layer and forwarding the data to a data processing engine;
calculating and returning a result in real time according to a data model of relevant design in a data processing engine, and simultaneously storing original data in a time sequence database;
the data processing engine directly sends the data after real-time processing to the time sequence database for storage, and simultaneously sends the data back to the message queue cluster for secondary consumption of other systems;
the configuration information is stored in a relational database.
2. The industrial sensor data processing method based on streaming processing as claimed in claim 1, wherein the data processing engine, the time sequence database and the relational database are controlled by a control configuration cluster.
3. The method for processing industrial sensor data based on stream processing according to claim 1, wherein the sensor comprises: acceleration sensor, temperature sensor, humidity transducer, flow sensor, pressure sensor, XYZ three-way sensor.
4. The industrial sensor data processing method based on stream processing as claimed in claim 1, wherein in the step of acquiring sensor data to the data acquisition layer, the sensor data is transmitted from various collectors to the data acquisition layer through a wireless or wired network.
5. The method as claimed in claim 1, wherein in the step of calculating and returning the result in real time by the data processing engine according to the data model of the relevant design, and storing the original data in the time-series database, the sensor data is processed in real time by introducing calculation and processing rules into the data processing engine, wherein the processing of the sensor data in real time comprises filtering abnormal values and gain amplification factors, and storing the result in the time-series database.
6. The method of claim 1, wherein the data processing engine can add work nodes at any time without shutdown and shutdown in case of current system resource trend or saturation.
7. The method for processing industrial sensor data based on stream processing as claimed in claim 1, wherein the step of storing configuration information in a relational database, the configuration information comprising: the method comprises the following steps of self node information required by flow control, upstream and downstream node information used for communication, branch flow information, a data processing model corresponding to a self node and an early warning rule.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210435330.8A CN114546993A (en) | 2022-04-24 | 2022-04-24 | Industrial sensor data processing method based on streaming processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210435330.8A CN114546993A (en) | 2022-04-24 | 2022-04-24 | Industrial sensor data processing method based on streaming processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114546993A true CN114546993A (en) | 2022-05-27 |
Family
ID=81666638
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210435330.8A Pending CN114546993A (en) | 2022-04-24 | 2022-04-24 | Industrial sensor data processing method based on streaming processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114546993A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115914360A (en) * | 2022-09-15 | 2023-04-04 | 成都飞机工业（集团）有限责任公司 | Time sequence data storage method, device, equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105608758A (en) * | 2015-12-17 | 2016-05-25 | 山东鲁能软件技术有限公司 | Big data analysis platform apparatus and method based on algorithm configuration and distributed stream computing |
| CN109800129A (en) * | 2019-01-17 | 2019-05-24 | 青岛特锐德电气股份有限公司 | A kind of real-time stream calculation monitoring system and method for processing monitoring big data |
| CN111130882A (en) * | 2019-12-25 | 2020-05-08 | 四川省公安科研中心 | Monitoring system and method of network equipment |
-
2022
- 2022-04-24 CN CN202210435330.8A patent/CN114546993A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105608758A (en) * | 2015-12-17 | 2016-05-25 | 山东鲁能软件技术有限公司 | Big data analysis platform apparatus and method based on algorithm configuration and distributed stream computing |
| CN109800129A (en) * | 2019-01-17 | 2019-05-24 | 青岛特锐德电气股份有限公司 | A kind of real-time stream calculation monitoring system and method for processing monitoring big data |
| CN111130882A (en) * | 2019-12-25 | 2020-05-08 | 四川省公安科研中心 | Monitoring system and method of network equipment |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115914360A (en) * | 2022-09-15 | 2023-04-04 | 成都飞机工业（集团）有限责任公司 | Time sequence data storage method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111077870A (en) | Intelligent OPC data real-time acquisition and monitoring system and method based on stream calculation | |
| CN107645483B (en) | Risk identification method, risk identification device, cloud risk identification device and system | |
| CN109408501B (en) | Position data processing method and device, server and storage medium | |
| CN110740160B (en) | Multi-source data map gridding and data state real-time pushing system | |
| CN112003763A (en) | Network link monitoring method, monitoring device, monitoring equipment and storage medium | |
| CN111209310B (en) | Service data processing method and device based on stream computing and computer equipment | |
| CN110377653B (en) | Real-time big data calculation and storage method and system | |
| CN113032157B (en) | Automatic intelligent server capacity expansion and reduction method and system | |
| CN114546993A (en) | Industrial sensor data processing method based on streaming processing | |
| CN109873785A (en) | Multi-source heterogeneous secure data acquisition system based on semantic Agent | |
| Dunne et al. | A comparison of data streaming frameworks for anomaly detection in embedded systems | |
| CN112614002A (en) | Data acquisition system, method, device, electronic equipment and computer storage medium | |
| CN105069029A (en) | Real-time ETL (extraction-transformation-loading) system and method | |
| CN112702219A (en) | Internet of things network monitoring method, device, equipment and storage medium | |
| CN113765777A (en) | Equipment control method, message transfer method, equipment, readable medium and Internet of things | |
| CN111400351A (en) | Method and device for inquiring streaming data based on distributed parallel architecture | |
| CN106990913A (en) | A kind of distributed approach of extensive streaming collective data | |
| AU2022356758A1 (en) | Auxiliary control method and system for wind turbine generator set, and wind turbine generator set | |
| CN115391429A (en) | Time sequence data processing method and device based on big data cloud computing | |
| CN115858672A (en) | Power terminal management method and device, electronic equipment and storage medium | |
| CN115220131A (en) | Meteorological data quality inspection method and system | |
| CN115276228A (en) | Method, system, storage device and computing device for synchronously uploading power data | |
| CN114546671A (en) | Data processing method and device and electronic equipment | |
| CN114090275A (en) | Data processing method and device and electronic equipment | |
| CN113079531B (en) | Automatic detection regulation and control method, system, terminal and medium for wireless communication module |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220527 |