CN111143438B - Workshop field data real-time monitoring and anomaly detection method based on stream processing - Google Patents
Workshop field data real-time monitoring and anomaly detection method based on stream processing Download PDFInfo
- Publication number
- CN111143438B CN111143438B CN201911387816.3A CN201911387816A CN111143438B CN 111143438 B CN111143438 B CN 111143438B CN 201911387816 A CN201911387816 A CN 201911387816A CN 111143438 B CN111143438 B CN 111143438B
- Authority
- CN
- China
- Prior art keywords
- data
- time
- workshop
- monitoring
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 77
- 238000001514 detection method Methods 0.000 title claims abstract description 42
- 238000012544 monitoring process Methods 0.000 title claims abstract description 40
- 238000004519 manufacturing process Methods 0.000 claims abstract description 30
- 230000002159 abnormal effect Effects 0.000 claims abstract description 20
- 238000007781 pre-processing Methods 0.000 claims abstract description 15
- 238000000034 method Methods 0.000 claims description 34
- 238000004364 calculation method Methods 0.000 claims description 26
- 238000004458 analytical method Methods 0.000 claims description 8
- 238000007726 management method Methods 0.000 claims description 7
- 230000005856 abnormality Effects 0.000 claims description 6
- 238000004220 aggregation Methods 0.000 claims description 6
- 230000002776 aggregation Effects 0.000 claims description 6
- 238000004140 cleaning Methods 0.000 claims description 5
- 238000013480 data collection Methods 0.000 claims description 3
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000011160 research Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 18
- 238000012360 testing method Methods 0.000 description 15
- 238000005111 flow chemistry technique Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000003860 storage Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 239000000463 material Substances 0.000 description 3
- 238000003466 welding Methods 0.000 description 3
- 238000011990 functional testing Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000036760 body temperature Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000009413 insulation Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000011056 performance test Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 238000012797 qualification Methods 0.000 description 1
- 238000010223 real-time analysis Methods 0.000 description 1
- 238000011897 real-time detection Methods 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 238000011895 specific detection Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/04—Manufacturing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Business, Economics & Management (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Computing Systems (AREA)
- Manufacturing & Machinery (AREA)
- Quality & Reliability (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- General Factory Administration (AREA)
Abstract
The invention relates to the technical field of workshop field data monitoring, in particular to a workshop field data real-time monitoring and anomaly detection method based on stream processing, which comprises the following steps: firstly, workshop production operation data acquisition equipment acquires data and generates a data stream, then data caching is carried out through a Kafka distributed message queue, the data is transmitted to a next-stage Bolt to be subjected to data preprocessing independently, operation data statistics and anomaly detection based on a sliding time window are carried out, and then identification comparison is carried out on the data corresponding to the abnormal state in the database according to the data corresponding to the judged abnormal state; according to the invention, through data preprocessing on the data stream, obvious errors such as null values, outliers and the like of the operation data of the production site are eliminated, and the operation data statistics and anomaly detection technology based on the sliding time window is designed, so that timeliness of operation data processing is ensured, and the relevance of connected data is also ensured to a certain extent.
Description
Technical Field
The invention relates to the technical field of workshop field data monitoring, in particular to a workshop field data real-time monitoring and anomaly detection method based on stream processing.
Background
Industrial manufacturing is rapidly growing with the development of Chinese economy, and the development of manufacturing capacity directly influences the development of the national economy and the social progress. Due to the complexity of the field environment of the production workshop and the diversity of product information, the resource allocation of enterprises is unreasonable, and the material waste and the economic loss are caused.
In recent years, with the development of sensing detection technology and the popularization of Internet of things equipment, workshop production field operation data is in an exponential-like rise, for the workshop field, the real-time performance of a monitoring system is low, the expandability is low, delay and even running are easy to occur, the total data amount and the value are rapidly increased and lost along with the time, the value is mined to exceed the calculation capacity of traditional data processing, the real-time monitoring management requirement on workshop production cannot be met, and challenges such as real-time processing, mass data storage and real-time visual analysis of data are brought to the existing abnormal state monitoring system based on the workshop field operation data.
Based on the method, the invention designs a workshop field data real-time monitoring and abnormality detection method based on stream processing so as to solve the problems.
Disclosure of Invention
The invention aims to provide a workshop field data real-time monitoring and anomaly detection method based on stream processing, which aims to solve the problems in the background technology.
In order to achieve the above purpose, the present invention provides the following technical solutions: the workshop field data real-time monitoring and abnormality detection method based on stream processing is characterized by comprising the following steps of:
s1: collecting data and generating a data stream by workshop production operation data collection equipment;
s2: respectively registering nodes of Kafka and Storm on a distributed server management system Zookeeper, and uniformly managing server nodes of the Kafka and Storm;
s3: carrying out data caching through a Kafka distributed message queue, taking a real-time big data computing platform for deploying Storms as a data consumption end of the Kafka, adopting Kafka integrated with the Storms as a data source connected with the message queue and the big data platform, and transmitting the data source to a next-stage Bolt in a data carrier mode for analysis and processing;
s4: when the data source Kafkaspout receives operation data, firstly, obtaining data of different operation data on different acquisition channels through data splitting Bolt;
s5: data preprocessing is carried out in a Bolt independently;
s6: performing statistics and anomaly detection on operation data based on sliding time windows, setting the sliding time windows to be 0.5min, setting the basic time windows to be 5s, performing independent calculation tasks under each basic time window, sliding the calculation tasks with the sliding time windows of 5s within 0.5min, merging the basic time windows, and counting the calculation results of the whole calculation window;
s7: after Spout and Bolt are completed, setting the data flow direction and grouping mode of each component in the Topology of Storm;
s8: on-line judgment of workshop production field states is realized in a Storm real-time processing frame through a real-time stream clustering algorithm, and according to the data corresponding to the judged abnormal states, recognition and comparison are carried out on the data and the corresponding abnormal state data in a database, and then abnormal records are output and stored in the database.
Preferably, the step of preprocessing the data in step S5 includes data cleansing, data formatting, and determining whether the data needs to be stored.
Preferably, the statistics of the operation data in step S6 refers to statistics of indexes such as maximum value, minimum value, average value, frequency of occurrence and energy utilization rate of the operation data, and the anomaly detection mainly comprises real-time critical detection and anomaly monitoring based on sliding time window.
Preferably, the specific steps and units for statistics of operation data and anomaly detection based on sliding time window in step S6 include:
step one: sliding window processing unit: firstly, setting time and parameters for transmitting data required by a sliding window, wherein the parameters mainly comprise unit window length and sliding window length;
step two: a field splitting processing unit: splitting the received operation data according to the types of the monitoring signals, and transmitting the operation data of the same type to a next-stage data processing unit;
step three: a data statistics processing unit: the calculation of data indexes such as the maximum value and the average value of data is realized in a unit time window, and the calculation result is sent to a next-stage data processing unit to perform data aggregation of the whole time window;
step four: frequency calculation processing unit: the method mainly aims at realizing operation data service requiring frequency statistics;
step five: threshold value judgment processing unit: according to the types of all the monitoring data, combining the data characteristics and related researches, formulating a strategy for judging the threshold value;
step six: summarizing and calculating processing unit: and merging and counting the data of all basic time windows on the processing unit, and using a global summarized data aggregation mode, namely, transmitting all the data to the same processing unit for final calculation.
Preferably, the specific mode of data cleaning is as follows: firstly, checking whether the data is qualified, if so, continuing to process the subsequent data, and if not, executing data cleaning and filtering operations, and deleting the data with null values, outliers and obvious error information from the monitored data.
Preferably, the data formatting includes data calibration and redundancy value deletion, in the following specific ways: the data is calibrated and formatted to convert analog values to true values while removing redundant information, such as data headers, and other fields, from the data stored by the sensor and acquisition system.
Preferably, the judgment basis for judging whether the data needs to be stored is whether the data needs to be processed first and then stored in the database.
Compared with the prior art, the invention has the beneficial effects that: according to the invention, through carrying out data preprocessing on the data stream, obvious errors such as null values, outliers, redundant data, format errors and the like of the operation data of the production site are eliminated; the running data statistics and anomaly detection technology based on the sliding time window is designed, timeliness of running data processing is guaranteed, relevance of connected data is guaranteed to a certain extent, state information in the field process of a production workshop is fully, accurately and in real time obtained, effective control, analysis and management of production process data are achieved, and production error prevention is carried out in real time.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of the process of the on-site operation data flow of the workshop of the invention;
FIG. 2 is a flow chart of data preprocessing according to the present invention;
FIG. 3 is a diagram showing statistics and detection based on sliding time window according to the present invention;
FIG. 4 is a flow chart of an implementation of the sliding time window based data statistics and detection of the present invention;
FIG. 5 is a business flow diagram of a status monitoring platform software system for production data according to the present invention;
FIG. 6 is a schematic diagram of the model and detection index of the present invention;
FIG. 7 is a diagram of a front end cell open circuit voltage test page of the present invention;
FIG. 8 is a diagram showing a workshop production process detection recording interface.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1-5, the present invention provides a technical solution: a workshop field data real-time monitoring and anomaly detection method based on stream processing comprises the following steps:
s1: collecting data and generating a data stream by workshop production operation data collection equipment;
s2: after the user has different operation authorities according to the authority of the account, successfully verifying the identity, logging in the system, initializing the system, and respectively registering nodes of Kafka and Storm on a distributed server management system Zookeeper and uniformly managing the server nodes of the Kafka and Storm;
s3: carrying out data caching through a Kafka distributed message queue, taking a real-time big data computing platform for deploying Storms as a data consumption end of the Kafka, adopting Kafka integrated with the Storms as a data source connected with the message queue and the big data platform, and transmitting the data source to a next-stage Bolt in a data carrier mode for analysis and processing;
s4: when the data source Kafkaspout receives operation data, firstly, obtaining data of different operation data on different acquisition channels through data splitting Bolt;
s5: the data preprocessing is carried out in the Bolt alone, and the data preprocessing can also be realized by programming in the Bolt alone in combination with specific operation data monitoring requirements; during actual production and operation, the sensors and the data acquisition equipment are affected by external electromagnetic and noise interference or abnormal conditions, and data preprocessing is used for eliminating errors and redundant data generated under the influence;
the data preprocessing step one is to clean the data, firstly check whether the data is qualified, if so, continue the subsequent data processing. If the data is not qualified, performing data cleaning and filtering operations to delete data with null values, outliers and obvious error information from the monitored data;
the second step of data formatting includes data calibration and redundant value deletion, specifically, calibration and formatting of the data, converting the analog value into a true value, and the data stored by the sensor and the acquisition system contains a lot of redundant information, such as data headers, other fields, etc., which are all useless data for the monitoring system and need to be deleted.
And thirdly, judging whether the data need to be stored or not, and distinguishing the data by taking whether the data need to be directly stored in a database or need to be processed after storage as a judging basis.
S6: and then carrying out statistics and anomaly detection on operation data based on a sliding time window, setting the sliding window to be 0.5min, setting the basic time window to be 5s, carrying out independent calculation tasks under each basic time window, sliding the calculation tasks within 0.5min by using the sliding window of 5s, finally merging the basic time windows, and counting the calculation results of the whole calculation window (0.5 min), wherein the statistics of the operation data refers to indexes such as maximum value, minimum value, average value, occurrence frequency, energy utilization rate and the like of the statistics operation data, the anomaly detection mainly comprises real-time critical detection and anomaly detection based on the sliding time window, and the specific steps and units of the statistics and the anomaly detection of the operation data based on the sliding time window comprise:
step one: sliding window processing unit: firstly, setting time and parameters for transmitting data required by a sliding window, wherein the parameters mainly comprise unit window length and sliding window length;
step two: a field splitting processing unit: splitting the received operation data according to the types of the monitoring signals, and transmitting the operation data of the same type to a next-stage data processing unit;
step three: a data statistics processing unit: calculating data indexes such as the maximum value and the average value of data in a unit time window, and sending a calculation result to a next-stage data processing unit to perform data aggregation of the whole time window;
step four: frequency calculation processing unit: the method mainly aims at realizing operation data service requiring frequency statistics;
step five: threshold value judgment processing unit: according to the types of all the monitoring data, combining the data characteristics and related researches, formulating a strategy for judging the threshold value;
step six: summarizing and calculating processing unit: and merging and counting the data of all basic time windows on the processing unit, and using a global summarized data aggregation mode, namely, transmitting all the data to the same processing unit for final calculation.
The stream processing is essentially that operation data is regarded as continuous data stream, single data points are calculated according to the calculation of the data, namely 'one time when one data is processed again', the timeliness of the operation data processing can be guaranteed, but the relevance of the connected data is cut off to a certain extent, the relevance of the operation data is very important for the prediction and analysis of the machine tool state, and therefore, the operation data is processed in a processing mode of introducing a sliding time window is very necessary
S7: after Spout and Bolt are completed, setting the data flow direction and grouping mode of each component in the Topology of Storm;
s8: on-line judgment of workshop production field states is realized in a Storm real-time processing frame through a real-time stream clustering algorithm, and according to the data corresponding to the judged abnormal states, recognition and comparison are carried out on the data and the corresponding abnormal state data in a database, then abnormal records are output, stored in the database, and the results are displayed in a front-end interface.
Taking an assembly test of a power battery as an example, the following specific embodiments are provided:
firstly, after the user has different operation authorities according to the authority of an account, logging in a system after the user successfully verifies the identity, initializing the system, accessing operation data to a large data platform through a Kafka message queue according to the processing requirement of the data, performing data preprocessing operation, performing data stream processing, realizing on-line judgment of the workshop production site state in a Storm real-time processing frame through a real-time stream clustering algorithm, performing recognition comparison with corresponding abnormal state data in a database according to the data corresponding to the judged abnormal state, outputting an abnormal record, and storing the abnormal record in the database.
The specific detection processing procedure after logging in the system is as follows:
the first step: cell input; and a second step of: mapping code combination; and a third step of: laser welding; fourth step: internal resistance/insulation withstand voltage test; fifth step: and (5) functional testing.
The method is characterized in that the method comprises the steps of collecting, testing and analyzing each stage of a product procedure based on a sensing technology and an automatic identification technology, and verifying the effectiveness of monitoring abnormal states of the product of the production line based on workshop field operation data flow. The data acquisition system adopts sensors such as daily-use sensors, agilent sensors and Mitsubishi sensors, the data acquisition system preprocesses the acquired data, and then the processed data result is accessed into a Kafka message queue, and the model and detection index of part of the data acquisition devices are shown in figure 6.
Then, running data acquisition and storage technology testing is conducted. In different working procedures of workshop production lines, a plurality of types of sensors are configured, wherein the daily-use BT3563 judges whether the battery cell is qualified or not by detecting indexes such as open-circuit voltage, internal resistance and the like, and in each working procedure of battery processing production, the monitored and collected indexes comprise mass operation data of products such as low-impedance detected resistance, total voltage, total internal resistance, single body temperature, on-board temperature and the like of functional test products. By setting up a distributed cluster environment, the collected data is calculated under the condition that the distributed cluster has no other tasks, and the running result is analyzed. Wherein kafka is used as a message queue; and using Storm as a distributed operation data stream processing frame to realize real-time and stable monitoring of workshop yield limiting data. By comparing the front-end query result with the operation data of the background database, it can be found that: the designed database can finish the storage of the operation data, and the problems of 'collapse', 'full buffer area' and the like of the database do not occur in the storage process; the running data query result is not lost, repeated and fast, and the data query requirement can be basically met.
And finally, testing and analyzing the real-time stream processing technology. The stream processing technology and the application case of the on-site production monitoring system in the part of the test workshop are characterized in that a distributed database is selected for real-time storage, an Ajax technology is used for real-time refreshing of operation data, and the interface content of a webpage end is updated in real time, so that real-time analysis, processing and uploading of the data are realized. The case tests the application effect and performance of abnormal state based on the stream processing technology. And taking the functional completion degree and the product qualification rate of the system as test standards. Specific tests and analyses were as follows:
firstly, the abnormal state monitoring test flow based on the stream processing technology is that nodes of Kafka and Storm are respectively registered on a distributed server management system Zookeeper, and server nodes of the Kafka and Storm are uniformly managed. In order to ensure that monitoring data can be processed in real time, data caching is carried out through a Kafka distributed message queue, a real-time big data computing platform for deploying Storm is used as a data consumption end of Kafka, kafkaSpout integrated by Kafka and Storm is used as a data source connected with the message queue and the big data platform, and the data is transmitted to a next-stage Bolt in a data carrier mode for analysis processing. When the data source Kafkaspout receives operation data, firstly, data of different operation data on different acquisition channels are obtained through data splitting Bolt, and programming is carried out in the Bolt alone in combination with specific operation data monitoring requirements to realize pretreatment of the operation data, data statistics based on a sliding time window, data anomaly monitoring and the like. After finishing logic writing of each Spout and Bolt, setting data flow direction and grouping mode of each component in the Topology of Storm;
then, carrying out functional test of a flow processing technology, wherein in the actual production process of a workshop, the first procedure is Cell investment, the procedure is mainly to judge whether the current flow processing technology meets the range of a test index by detecting an open-circuit voltage attenuation index and measuring the actual voltage, and if the current flow processing technology is not qualified, the current flow processing technology is required to return to a supplier; real-time, efficient and stable detection of this index is very important, so that a storm-based stream processing technique is introduced to process and analyze the acquired data in real time, so that the processing delay is significantly reduced.
The machine tool operation data real-time processing technology based on flow processing mainly realizes data preprocessing, data statistics, data anomaly detection and real-time operation state judgment in a monitoring system. The data preprocessing and the data statistics mainly finish data processing in the background, and the data anomaly detection and running state judgment results are mainly displayed in the front-end interface and are displayed in the anomaly management module. According to the parameters of the battery cell and the requirements of products, the minimum value and the maximum value of the open-circuit voltage are respectively set to 3.294V and 3.299V. The measured voltage attenuation signal is easy to be interfered by the outside, the instantaneous change characteristic is large, the measured value is accurate, and the like, and the false alarm phenomenon is easy to be generated by directly carrying out threshold monitoring, so that according to a data critical anomaly detection strategy based on a sliding time window, the measured average value and the maximum minimum value of the last 5 seconds per second are counted, if the probability that the actual value of the open-circuit voltage of the battery cell exceeds the alarm threshold value is more than 80%, the output is 'NG', namely the product is unqualified, and the result is displayed in a front-end interface; if the value calculated and analyzed in real time through stream processing calculation is within the process specified range, the value can be judged to be qualified, and the OK is fed back at the front end interface. In the process, by introducing a sliding time window concept based on flow processing, the detection stability and efficiency are improved, the false detection rate of the battery cell quality is prevented, and the detection and production efficiency of a workshop factory are improved. Fig. 7 shows real-time detection information of the front-end detection interface.
Real-time refreshing of running data and database access are achieved through the Ajax technology, and writing of a front-end page is completed through the Html5 and CSS modes. In the test case, the open-circuit voltage of the battery cell and the welding current monitored by welding quality are taken as test objects. The detection processing delay meets the requirement of a production field, the actual processing and production are not affected, the performance test result and the detection index of each link procedure can be loaded by using a web browser, and the dynamic adjustment and modification of the detection index can be completed at the web end. In the production process of each process, the interface of the monitoring record item of each process is shown in fig. 8.
In the description of the present specification, the descriptions of the terms "one embodiment," "example," "specific example," and the like, mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The preferred embodiments of the invention disclosed above are intended only to assist in the explanation of the invention. The preferred embodiments are not intended to be exhaustive or to limit the invention to the precise form disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best understand and utilize the invention. The invention is limited only by the claims and the full scope and equivalents thereof.
Claims (6)
1. The workshop field data real-time monitoring and abnormality detection method based on stream processing is characterized by comprising the following steps of:
s1: collecting data and generating a data stream by workshop production operation data collection equipment;
s2: respectively registering nodes of Kafka and Storm on a distributed server management system Zookeeper, and uniformly managing server nodes of the Kafka and Storm;
s3: carrying out data caching through a Kafka distributed message queue, taking a real-time big data computing platform for deploying Storms as a data consumption end of the Kafka, adopting Kafka integrated with the Storms as a data source connected with the message queue and the big data platform, and transmitting the data source to a next-stage Bolt in a data carrier mode for analysis and processing;
s4: when the data source Kafkaspout receives operation data, firstly, obtaining data of different operation data on different acquisition channels through data splitting Bolt;
s5: data preprocessing is carried out in a Bolt independently;
s6: performing statistics and anomaly detection on operation data based on sliding time windows, setting the sliding time windows to be 0.5min, setting the basic time windows to be 5s, performing independent calculation tasks under each basic time window, sliding the calculation tasks with the sliding time windows of 5s within 0.5min, merging the basic time windows, and counting the calculation results of the whole calculation window;
s7: after Spout and Bolt are completed, setting the data flow direction and grouping mode of each component in the Topology of Storm;
s8: on-line judgment of workshop production field states is realized in a Storm real-time processing frame through a real-time stream clustering algorithm, and according to the data corresponding to the judged abnormal states, recognition comparison is carried out on the data and the corresponding abnormal state data in a database, and then an abnormal record is output and stored in the database;
the specific steps and units of the sliding time window based operation data statistics and anomaly detection in step S6 include:
step one: sliding window processing unit: firstly, setting time and parameters for transmitting data required by a sliding window, wherein the parameters mainly comprise unit window length and sliding window length;
step two: a field splitting processing unit: splitting the received operation data according to the types of the monitoring signals, and transmitting the operation data of the same type to a next-stage data processing unit;
step three: a data statistics processing unit: the calculation of data indexes such as the maximum value and the average value of data is realized in a unit time window, and the calculation result is sent to a next-stage data processing unit to perform data aggregation of the whole time window;
step four: frequency calculation processing unit: the method mainly aims at realizing operation data service requiring frequency statistics;
step five: threshold value judgment processing unit: according to the types of all the monitoring data, combining the data characteristics and related researches, formulating a strategy for judging the threshold value;
step six: summarizing and calculating processing unit: and merging and counting the data of all basic time windows on the processing unit, and using a global summarized data aggregation mode, namely, transmitting all the data to the same processing unit for final calculation.
2. The method for monitoring and detecting anomalies in real time on-site data of a workshop based on stream processing as recited in claim 1, wherein the method comprises the steps of: the step of preprocessing the data in step S5 includes data cleansing, data formatting, and determining whether the data needs to be stored.
3. The method for monitoring and detecting anomalies in real time on-site data of a workshop based on stream processing as recited in claim 1, wherein the method comprises the steps of: the statistics of the operation data in step S6 refers to statistics of indexes such as maximum value, minimum value, average value, frequency of occurrence and energy utilization rate of the operation data, and the anomaly detection mainly comprises real-time critical detection and anomaly monitoring based on a sliding time window.
4. The method for monitoring and detecting the abnormality of the workshop site data in real time based on the stream processing according to claim 2, wherein the method comprises the following steps: the specific mode of data cleaning is as follows: firstly, checking whether the data is qualified, if so, continuing to process the subsequent data, and if not, executing data cleaning and filtering operations, and deleting the data with null values, outliers and obvious error information from the monitored data.
5. The method for monitoring and detecting the abnormality of the workshop site data in real time based on the stream processing according to claim 2, wherein the method comprises the following steps: the data formatting comprises data calibration and redundant value deletion, and the specific modes are as follows: the data is calibrated and formatted to convert analog values to true values while removing redundant information, such as data headers, and other fields, from the data stored by the sensor and acquisition system.
6. The method for monitoring and detecting the abnormality of the workshop site data in real time based on the stream processing according to claim 2, wherein the method comprises the following steps: the judging basis for judging whether the data need to be stored is whether the data need to be processed first and then stored in the database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911387816.3A CN111143438B (en) | 2019-12-30 | 2019-12-30 | Workshop field data real-time monitoring and anomaly detection method based on stream processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911387816.3A CN111143438B (en) | 2019-12-30 | 2019-12-30 | Workshop field data real-time monitoring and anomaly detection method based on stream processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111143438A CN111143438A (en) | 2020-05-12 |
| CN111143438B true CN111143438B (en) | 2023-09-12 |
Family
ID=70521564
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911387816.3A Active CN111143438B (en) | 2019-12-30 | 2019-12-30 | Workshop field data real-time monitoring and anomaly detection method based on stream processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111143438B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000858B (en) * | 2020-08-17 | 2024-04-05 | 浙江鸿城科技有限责任公司 | Workshop data anomaly identification method |
| CN112699113B (en) * | 2021-01-12 | 2022-08-05 | 上海交通大学 | Industrial manufacturing process operation monitoring system driven by time sequence data stream |
| CN113325824B (en) * | 2021-06-02 | 2022-10-25 | 三门核电有限公司 | Regulating valve abnormity identification method and system based on threshold monitoring |
| CN113486063A (en) * | 2021-07-05 | 2021-10-08 | 国网河北省电力有限公司信息通信分公司 | Method and device for processing flow data in power internet of things and terminal equipment |
| CN113993001B (en) * | 2021-09-08 | 2024-04-12 | 四创电子股份有限公司 | Real-time stream analysis alarm method based on sliding data window |
| CN113762163B (en) * | 2021-09-09 | 2022-06-07 | 杭州澳亚生物技术股份有限公司 | GMP workshop intelligent monitoring management method and system |
| WO2023050229A1 (en) * | 2021-09-29 | 2023-04-06 | 西门子股份公司 | Industrial data processing method and apparatus, electronic device, and storage medium |
| CN117331921A (en) * | 2023-09-28 | 2024-01-02 | 石家庄铁道大学 | Bearing monitoring multisource data processing method |
| CN117520011A (en) * | 2023-11-09 | 2024-02-06 | 中国三峡建工(集团)有限公司 | Sharing management method and system for real-time data memory |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107968840A (en) * | 2017-12-15 | 2018-04-27 | 华北电力大学（保定） | A kind of extensive power equipment monitoring, alarming Real-time Data Processing Method and system |
| US10129118B1 (en) * | 2016-03-29 | 2018-11-13 | Amazon Technologies, Inc. | Real time anomaly detection for data streams |
| CN110347116A (en) * | 2019-07-17 | 2019-10-18 | 重庆大学 | A kind of conditions of machine tool monitoring system and monitoring method based on operation data stream |
-
2019
- 2019-12-30 CN CN201911387816.3A patent/CN111143438B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10129118B1 (en) * | 2016-03-29 | 2018-11-13 | Amazon Technologies, Inc. | Real time anomaly detection for data streams |
| CN107968840A (en) * | 2017-12-15 | 2018-04-27 | 华北电力大学（保定） | A kind of extensive power equipment monitoring, alarming Real-time Data Processing Method and system |
| CN110347116A (en) * | 2019-07-17 | 2019-10-18 | 重庆大学 | A kind of conditions of machine tool monitoring system and monitoring method based on operation data stream |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111143438A (en) | 2020-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111143438B (en) | Workshop field data real-time monitoring and anomaly detection method based on stream processing | |
| CN110347116B (en) | Machine tool state monitoring system and monitoring method based on operation data flow | |
| CN104765322B (en) | Production line monitoring system | |
| CN101770219B (en) | Knowledge acquisition method of fault diagnosis knowledge library of turn-milling combined machine tool | |
| CN112284440B (en) | Sensor data deviation self-adaptive correction method | |
| CN108491861A (en) | Power transmission and transformation equipment state abnormal patterns recognition methods based on multi-source multi-parameter fusion and device | |
| WO2022174679A1 (en) | Method and apparatus for predicting voltage inconsistency fault of battery cells, and server | |
| CN115186883A (en) | Industrial equipment health state monitoring system and method based on Bian Yun collaborative computing | |
| CN111027615A (en) | Middleware fault early warning method and system based on machine learning | |
| CN115170000A (en) | Remote monitoring method and system based on electric energy meter communication module | |
| CN116108402A (en) | Method, equipment and storage medium based on electric power multi-source heterogeneous data fusion analysis | |
| CN103529337B (en) | The recognition methods of nonlinear correlation relation between equipment failure and electric quantity information | |
| CN117572837B (en) | Intelligent power plant AI active operation and maintenance method and system | |
| CN106774054A (en) | GIS device analysis system and method based on the identification of complicated unstructured data | |
| CN110336860A (en) | Key node data guard method based on multidimensional data processing in industrial Internet of Things | |
| CN115114124A (en) | Host risk assessment method and device | |
| CN111314110B (en) | Fault early warning method for distributed system | |
| CN117491813A (en) | Insulation abnormality detection method for power battery system of new energy automobile | |
| CN116862109A (en) | Regional carbon emission situation awareness early warning method | |
| Li et al. | Meteorological radar fault diagnosis based on deep learning | |
| Pan et al. | Study on intelligent anti–electricity stealing early-warning technology based on convolutional neural networks | |
| CN115481183A (en) | Real-time data processing method and system | |
| CN203825457U (en) | Cigarette making and tipping machine set control system based on K-BUS bus | |
| CN206505304U (en) | A kind of real-time warm examining system of multiple spot | |
| CN111103487A (en) | Non-invasive PST anomaly monitoring method based on power consumption analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |