CN112559803A - Data anomaly detection method and system based on iteration - Google Patents
Data anomaly detection method and system based on iteration Download PDFInfo
- Publication number
- CN112559803A CN112559803A CN202010650548.6A CN202010650548A CN112559803A CN 112559803 A CN112559803 A CN 112559803A CN 202010650548 A CN202010650548 A CN 202010650548A CN 112559803 A CN112559803 A CN 112559803A
- Authority
- CN
- China
- Prior art keywords
- data
- unit
- module
- anomaly
- anomaly detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
Abstract
The invention discloses a data anomaly detection method and a data anomaly detection system based on iteration, in particular to a data anomaly detection system based on iteration, which comprises a data acquisition module, a data processing module, a data shaping module, an anomaly prediction module, an anomaly detection module and an anomaly alarm module, wherein the data acquisition module, the data processing module, the data shaping module, the anomaly prediction module, the anomaly detection module and the anomaly alarm module are sequentially connected, the data processing module comprises a data cleaning unit, a feature extraction unit and a data analysis unit, the data cleaning unit, the feature extraction unit and the data analysis unit are sequentially connected, and the anomaly detection module comprises a detection unit, a determination unit, a fitting unit, a comparison unit, a judgment unit and a storage unit. The data anomaly detection system has the advantages of high speed and high accuracy when the data anomaly detection is carried out.
Description
Technical Field
The invention relates to the technical field of data detection, in particular to a data anomaly detection method and a data anomaly detection system based on iteration.
Background
In recent years, with the rapid development of urbanization and industrialization, the situation of air pollution in many areas of China is increasingly severe, and the air quality is not optimistic. Common air pollutants are NO2, O3, CO, PM2.5, and the like. These air pollutants easily cause inflammation of human respiratory tract, destroy blood circulation and nervous system of human body, and even cause death of human body. Therefore, how to effectively prevent the harm of human from air pollution is receiving extensive attention. However, it is very difficult to achieve the treatment of air pollution in a short period of time. Therefore, the air pollution can be prevented and treated in time by effectively predicting the air quality in dozens of hours in the future, the serious pollution can be prevented, and the improvement of the public life quality is widely concerned by the society.
At present, two methods are available for data mining and modeling of a real process of a complex nonlinear system: one is a traditional process-based modeling approach; the other is based on data-driven statistical methods. However, the existing air quality detection data is difficult to perform fine anomaly detection.
Disclosure of Invention
Based on the technical problems in the background art, the invention provides a data anomaly detection method and a data anomaly detection system based on iteration.
The invention provides an iteration-based data anomaly detection system which comprises a data acquisition module, a data processing module, a data reshaping module, an anomaly prediction module, an anomaly detection module and an anomaly alarm module, wherein the data acquisition module, the data processing module, the data reshaping module, the anomaly prediction module, the anomaly detection module and the anomaly alarm module are sequentially connected, the data processing module comprises a data cleaning unit, a feature extraction unit and a data analysis unit, the data cleaning unit, the feature extraction unit and the data analysis unit are sequentially connected, the anomaly detection module comprises a detection unit, a determination unit, a fitting unit, a comparison unit, a judgment unit and a storage unit, and the detection unit, the determination unit, the fitting unit, the comparison unit, the judgment unit and the storage unit are sequentially connected.
Preferably, the data acquisition module is used for acquiring real-time data information and transmitting the real-time data information to the data processing module.
Preferably, the data processing module is used for cleaning, extracting and analyzing the data, and transmitting the processed data to the data shaping module.
Preferably, the data shaping module is configured to integrate the received data information, convert the format of the data information into a required data format, and transmit the data information after the format conversion to the anomaly prediction module.
Preferably, the anomaly prediction module is configured to analyze data to obtain predicted data, compare the predicted data with the real data to determine whether the original data is abnormal, and transmit predicted data information to the anomaly detection module.
Preferably, the anomaly detection module is configured to perform anomaly detection on the data to be detected in the current period by using a pre-constructed detection model according to the prediction data, and transmit the anomaly data to the anomaly alarm module.
Preferably, the abnormal alarm module is configured to receive the abnormal data information and perform alarm processing on the abnormal data information.
Preferably, the data cleaning unit is configured to clean the data, remove missing values and abnormal values, and transmit the cleaned data to the feature extraction unit, the feature extraction unit is configured to extract features of the data, the extracted feature data is transmitted to the data analysis unit, and the data analysis unit is configured to analyze the feature data.
Preferably, the detection unit is configured to detect abnormal data, the detected abnormal data is transmitted to the determination unit to determine the abnormal data, the determined abnormal data is transmitted to the fitting unit to perform fitting arrangement on the data, then the data is transmitted to the comparison unit to compare the abnormal data with the normal data, the comparison result is sent to the determination unit to perform abnormality determination, and meanwhile, the data information is stored through the storage unit.
An iteration-based data anomaly detection method comprises the following steps:
the S1 data acquisition module is used for acquiring real-time data information and transmitting the real-time data information to the data processing module;
the S2 data processing module is used for cleaning, extracting and analyzing the data and transmitting the processed data to the data shaping module, and the data processing module comprises a data cleaning unit, a characteristic extraction unit and a data analysis unit;
the data cleaning unit is used for cleaning data, removing missing values and abnormal values, and transmitting the cleaned data to the feature extraction unit, the feature extraction unit is used for extracting data features, the extracted feature data is transmitted to the data analysis unit, and the data analysis unit is used for analyzing the feature data;
the S3 data shaping module is used for integrating the received data information, converting the format of the data information into a required data format and transmitting the data information after being converted into the format to the abnormity prediction module;
the S4 anomaly prediction module is used for analyzing the data to obtain prediction data, comparing the prediction data with the real data to judge whether the original data is abnormal or not, and transmitting the predicted data information to the anomaly detection module;
the S5 anomaly detection module is used for carrying out anomaly detection on the data to be detected in the current period by utilizing a pre-constructed detection model according to the prediction data, and transmitting the anomaly data to the anomaly alarm module;
the S6 abnormity alarm module is used for receiving the abnormity data information and alarming the abnormity data information, and the abnormity detection module comprises a detection unit, a determination unit, a fitting unit, a comparison unit, a judgment unit and a storage unit;
the detection unit is used for detecting abnormal data, the detected abnormal data are transmitted to the determination unit to be determined, the determined abnormal data are transmitted to the fitting unit to be fitted and sorted, then the data are transmitted to the comparison unit to be compared with the abnormal data, the comparison result is sent to the judgment unit to be judged abnormally, and meanwhile, data information is stored through the storage unit.
According to the data anomaly detection method and system based on iteration, the data anomaly detection system has the advantages of high speed and high accuracy when carrying out anomaly detection on data.
Drawings
Fig. 1 is a schematic flow chart of an iteration-based data anomaly detection system according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments.
Referring to fig. 1, an iteration-based data anomaly detection system comprises a data acquisition module, a data processing module, a data reshaping module, an anomaly prediction module, an anomaly detection module and an anomaly alarm module, wherein the data acquisition module, the data processing module, the data reshaping module, the anomaly prediction module, the anomaly detection module and the anomaly alarm module are sequentially connected, the data processing module comprises a data cleaning unit, a feature extraction unit and a data analysis unit, the data cleaning unit, the feature extraction unit and the data analysis unit are sequentially connected, the anomaly detection module comprises a detection unit, a determination unit, a fitting unit, a comparison unit, a judgment unit and a storage unit, and the detection unit, the determination unit, the fitting unit, the comparison unit, the judgment unit and the storage unit are sequentially connected.
In the invention, the data acquisition module is used for acquiring real-time data information and transmitting the real-time data information to the data processing module.
In the invention, the data processing module is used for cleaning, extracting and analyzing data and transmitting the processed data to the data shaping module.
In the invention, the data shaping module is used for integrating the received data information, converting the format of the data information into a required data format and transmitting the data information after the format conversion to the abnormity prediction module.
In the invention, the abnormity prediction module is used for analyzing data to obtain prediction data, the prediction data is compared with real data to judge whether the original data is abnormal, and meanwhile, the predicted data information is transmitted to the abnormity detection module.
According to the invention, the anomaly detection module is used for carrying out anomaly detection on the data to be detected in the current period by utilizing a pre-constructed detection model according to the prediction data, and transmitting the anomaly data to the anomaly alarm module.
In the invention, the abnormal alarm module is used for receiving the abnormal data information and carrying out alarm processing on the abnormal data information.
In the invention, the data cleaning unit is used for cleaning data, removing missing values and abnormal values, and transmitting the cleaned data to the feature extraction unit, the feature extraction unit is used for extracting data features, the extracted feature data is transmitted to the data analysis unit, and the data analysis unit is used for analyzing the feature data.
In the invention, the detection unit is used for detecting abnormal data, the detected abnormal data is transmitted to the determination unit for determining the abnormal data, the determined abnormal data is transmitted to the fitting unit for fitting and sorting the data, then the data is transmitted to the comparison unit for comparing the abnormal data with normal data, the comparison result is transmitted to the judgment unit for abnormal judgment, and meanwhile, the data information is stored through the storage unit.
An iteration-based data anomaly detection method comprises the following steps:
the S1 data acquisition module is used for acquiring real-time data information and transmitting the real-time data information to the data processing module;
the S2 data processing module is used for cleaning, extracting and analyzing the data and transmitting the processed data to the data shaping module, and the data processing module comprises a data cleaning unit, a characteristic extraction unit and a data analysis unit;
the data cleaning unit is used for cleaning data, removing missing values and abnormal values, and transmitting the cleaned data to the feature extraction unit, the feature extraction unit is used for extracting data features, the extracted feature data is transmitted to the data analysis unit, and the data analysis unit is used for analyzing the feature data;
the S3 data shaping module is used for integrating the received data information, converting the format of the data information into a required data format and transmitting the data information after being converted into the format to the abnormity prediction module;
the S4 anomaly prediction module is used for analyzing the data to obtain prediction data, comparing the prediction data with the real data to judge whether the original data is abnormal or not, and transmitting the predicted data information to the anomaly detection module;
the S5 anomaly detection module is used for carrying out anomaly detection on the data to be detected in the current period by utilizing a pre-constructed detection model according to the prediction data, and transmitting the anomaly data to the anomaly alarm module;
the S6 abnormity alarm module is used for receiving the abnormity data information and alarming the abnormity data information, and the abnormity detection module comprises a detection unit, a determination unit, a fitting unit, a comparison unit, a judgment unit and a storage unit;
the detection unit is used for detecting abnormal data, the detected abnormal data are transmitted to the determination unit to be determined, the determined abnormal data are transmitted to the fitting unit to be fitted and sorted, then the data are transmitted to the comparison unit to be compared with the abnormal data, the comparison result is sent to the judgment unit to be judged abnormally, and meanwhile, data information is stored through the storage unit.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art should be considered to be within the technical scope of the present invention, and the technical solutions and the inventive concepts thereof according to the present invention should be equivalent or changed within the scope of the present invention.
Claims (10)
1. The data anomaly detection system based on iteration is characterized by comprising a data acquisition module, a data processing module, a data reshaping module, an anomaly prediction module, an anomaly detection module and an anomaly alarm module, wherein the data acquisition module, the data processing module, the data reshaping module, the anomaly prediction module, the anomaly detection module and the anomaly alarm module are sequentially connected, the data processing module comprises a data cleaning unit, a feature extraction unit and a data analysis unit, the data cleaning unit, the feature extraction unit and the data analysis unit are sequentially connected, the anomaly detection module comprises a detection unit, a determination unit, a fitting unit, a comparison unit, a judgment unit and a storage unit, and the detection unit, the determination unit, the fitting unit, the comparison unit, the judgment unit and the storage unit are sequentially connected.
2. The iteration-based data anomaly detection system of claim 1, wherein said data collection module is configured to collect real-time data information and transmit the real-time data information to the data processing module.
3. The iteration-based data anomaly detection system according to claim 1, wherein the data processing module is configured to clean, extract and analyze data, and transmit the processed data to the data shaping module.
4. The iteration-based data anomaly detection system according to claim 1, wherein the data shaping module is configured to integrate the received data information, convert the format of the data information into a desired data format, and transmit the data information after the format conversion to the anomaly prediction module.
5. The iteration-based data anomaly detection system of claim 1, wherein the anomaly prediction module is configured to analyze data to obtain predicted data, compare the predicted data with real data to determine whether the original data is anomalous, and transmit predicted data information to the anomaly detection module.
6. The iteration-based data anomaly detection system according to claim 1, wherein the anomaly detection module is configured to perform anomaly detection on data to be detected in a current period by using a pre-constructed detection model according to predicted data, and transmit the anomalous data to the anomaly alarm module.
7. The iteration-based data anomaly detection system of claim 1, wherein the anomaly alarm module is configured to receive anomaly data information and perform alarm processing on the anomaly data information.
8. The iteration-based data anomaly detection system according to claim 1, wherein the data cleaning unit is configured to clean data, remove missing values and abnormal values, and transmit the cleaned data to the feature extraction unit, the feature extraction unit is configured to extract features of the data, the extracted feature data is transmitted to the data analysis unit, and the data analysis unit is configured to analyze the feature data.
9. The iteration-based data anomaly detection system according to claim 1, wherein the detection unit is configured to detect anomalous data, transmit the detected anomalous data to the determination unit to determine the anomalous data, transmit the determined anomalous data to the fitting unit to perform fitting and sorting on the data, transmit the data to the comparison unit to compare the anomalous data with normal data, transmit the comparison result to the determination unit to perform anomaly determination, and simultaneously store the data information through the storage unit.
10. An iteration-based data anomaly detection method comprises the following steps:
the S1 data acquisition module is used for acquiring real-time data information and transmitting the real-time data information to the data processing module;
the S2 data processing module is used for cleaning, extracting and analyzing the data and transmitting the processed data to the data shaping module, and the data processing module comprises a data cleaning unit, a characteristic extraction unit and a data analysis unit;
the data cleaning unit is used for cleaning data, removing missing values and abnormal values, and transmitting the cleaned data to the feature extraction unit, the feature extraction unit is used for extracting data features, the extracted feature data is transmitted to the data analysis unit, and the data analysis unit is used for analyzing the feature data;
the S3 data shaping module is used for integrating the received data information, converting the format of the data information into a required data format and transmitting the data information after being converted into the format to the abnormity prediction module;
the S4 anomaly prediction module is used for analyzing the data to obtain prediction data, comparing the prediction data with the real data to judge whether the original data is abnormal or not, and transmitting the predicted data information to the anomaly detection module;
the S5 anomaly detection module is used for carrying out anomaly detection on the data to be detected in the current period by utilizing a pre-constructed detection model according to the prediction data, and transmitting the anomaly data to the anomaly alarm module;
the S6 abnormity alarm module is used for receiving the abnormity data information and alarming the abnormity data information, and the abnormity detection module comprises a detection unit, a determination unit, a fitting unit, a comparison unit, a judgment unit and a storage unit;
the detection unit is used for detecting abnormal data, the detected abnormal data are transmitted to the determination unit to be determined, the determined abnormal data are transmitted to the fitting unit to be fitted and sorted, then the data are transmitted to the comparison unit to be compared with the abnormal data, the comparison result is sent to the judgment unit to be judged abnormally, and meanwhile, data information is stored through the storage unit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010650548.6A CN112559803A (en) | 2020-07-08 | 2020-07-08 | Data anomaly detection method and system based on iteration |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010650548.6A CN112559803A (en) | 2020-07-08 | 2020-07-08 | Data anomaly detection method and system based on iteration |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112559803A true CN112559803A (en) | 2021-03-26 |
Family
ID=75040852
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010650548.6A Pending CN112559803A (en) | 2020-07-08 | 2020-07-08 | Data anomaly detection method and system based on iteration |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112559803A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113377067A (en) * | 2021-05-27 | 2021-09-10 | 意特利（上海）科技有限公司 | Dynamic sensing monitoring acquisition compensation method and device for numerical control five-axis linkage machine tool |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106096789A (en) * | 2016-06-22 | 2016-11-09 | 华东师范大学 | A kind of based on machine learning techniques can be from the abnormal industry control security protection of perception and warning system |
| CN109993222A (en) * | 2019-03-25 | 2019-07-09 | 中国科学院上海高等研究院 | Data exception detection system and method |
| CN110209560A (en) * | 2019-05-09 | 2019-09-06 | 北京百度网讯科技有限公司 | Data exception detection method and detection device |
| CN110990386A (en) * | 2019-11-27 | 2020-04-10 | 工业互联网创新中心（上海）有限公司 | Industrial internet platform monitoring data transmission and exchange method and system |
| CN111275288A (en) * | 2019-12-31 | 2020-06-12 | 华电国际电力股份有限公司十里泉发电厂 | XGboost-based multi-dimensional data anomaly detection method and device |
-
2020
- 2020-07-08 CN CN202010650548.6A patent/CN112559803A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106096789A (en) * | 2016-06-22 | 2016-11-09 | 华东师范大学 | A kind of based on machine learning techniques can be from the abnormal industry control security protection of perception and warning system |
| CN109993222A (en) * | 2019-03-25 | 2019-07-09 | 中国科学院上海高等研究院 | Data exception detection system and method |
| CN110209560A (en) * | 2019-05-09 | 2019-09-06 | 北京百度网讯科技有限公司 | Data exception detection method and detection device |
| CN110990386A (en) * | 2019-11-27 | 2020-04-10 | 工业互联网创新中心（上海）有限公司 | Industrial internet platform monitoring data transmission and exchange method and system |
| CN111275288A (en) * | 2019-12-31 | 2020-06-12 | 华电国际电力股份有限公司十里泉发电厂 | XGboost-based multi-dimensional data anomaly detection method and device |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113377067A (en) * | 2021-05-27 | 2021-09-10 | 意特利（上海）科技有限公司 | Dynamic sensing monitoring acquisition compensation method and device for numerical control five-axis linkage machine tool |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111046582A (en) | Nuclear power station diesel generating set coil temperature early warning method and system | |
| CN108549914B (en) | Abnormal SER/SOE event identification method based on Apriori algorithm | |
| CN112149576A (en) | Elevator safety real-time monitoring management system based on image analysis | |
| CN114509283A (en) | System fault monitoring method and device, electronic equipment and storage medium | |
| CN114371353A (en) | Power equipment abnormity monitoring method and system based on voiceprint recognition | |
| CN116166983A (en) | Train bow net fault detection method based on machine learning | |
| CN108357517A (en) | Motor train unit train operation troubles information detecting method and system | |
| CN111666978B (en) | Intelligent fault early warning system for IT system operation and maintenance big data | |
| CN116204842A (en) | Abnormality monitoring method and system for electrical equipment | |
| CN110908957A (en) | Network security log audit analysis method in power industry | |
| CN112559803A (en) | Data anomaly detection method and system based on iteration | |
| CN112666430B (en) | Intelligent fault detection method and system for voiceprint of transformer | |
| CN117150418B (en) | Transformer operation detection period formulation method and system based on state characteristic fault tree | |
| CN113220799A (en) | Big data early warning management system | |
| CN117391443A (en) | Dust removal equipment state monitoring and early warning method and system | |
| CN111767913A (en) | Motor train unit vehicle fault dynamic image detection method based on deep learning | |
| CN115034094B (en) | Prediction method and system for operation state of metal processing machine tool | |
| CN111058944B (en) | Method and system for detecting temperature abnormity of cylinder of diesel generating set | |
| CN211043977U (en) | Environmental pollution source monitoring system | |
| CN113156529A (en) | Start-stop control method, system, terminal and storage medium of metrological verification assembly line | |
| CN117438083B (en) | Middle-aged and elderly disease prediction system based on artificial intelligence | |
| CN111157812A (en) | Power equipment fault monitoring method based on electromagnetic induction | |
| CN117649123A (en) | Wisdom garden management system | |
| CN117030722A (en) | Method and system for detecting damage of carbon fiber composite core overhead conductor | |
| CN115238121A (en) | Rotating machine health and fault feature identification method based on audio technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |