CN117527552A - Monitoring alarm method and device based on self-encoder, internet of things platform and medium - Google Patents
Monitoring alarm method and device based on self-encoder, internet of things platform and medium Download PDFInfo
- Publication number
- CN117527552A CN117527552A CN202311701646.8A CN202311701646A CN117527552A CN 117527552 A CN117527552 A CN 117527552A CN 202311701646 A CN202311701646 A CN 202311701646A CN 117527552 A CN117527552 A CN 117527552A
- Authority
- CN
- China
- Prior art keywords
- data
- self
- encoder
- target
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 80
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000012549 training Methods 0.000 claims abstract description 68
- 230000002159 abnormal effect Effects 0.000 claims abstract description 57
- 238000012795 verification Methods 0.000 claims abstract description 42
- 238000012360 testing method Methods 0.000 claims abstract description 41
- 238000012423 maintenance Methods 0.000 claims abstract description 32
- 238000007781 pre-processing Methods 0.000 claims abstract description 14
- 238000012545 processing Methods 0.000 claims description 41
- 238000013528 artificial neural network Methods 0.000 claims description 7
- 238000004891 communication Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 238000013480 data collection Methods 0.000 claims description 6
- 230000006870 function Effects 0.000 claims description 6
- 230000005856 abnormality Effects 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000001514 detection method Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0681—Configuration of triggering conditions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0604—Management of faults, events, alarms or notifications using filtering, e.g. reduction of information by using priority, element types, position or time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/142—Network analysis or design using statistical or mathematical methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/16—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
- H04L43/045—Processing captured monitoring data, e.g. for logfile generation for graphical visualisation of monitoring data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Medical Informatics (AREA)
- Data Mining & Analysis (AREA)
- Algebra (AREA)
- Mathematical Physics (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Evolutionary Computation (AREA)
- Probability & Statistics with Applications (AREA)
- Pure & Applied Mathematics (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Alarm Systems (AREA)
Abstract
The application relates to a monitoring alarm method and device based on a self-encoder, an Internet of things platform and a medium, wherein the method comprises the following steps: collecting initial data from the sensors and the devices and storing the initial data in a time series database; performing data preprocessing on initial data in a time sequence database to obtain target data, and dividing the target data into training data and test verification data; constructing a self-encoder model, training and fitting the self-encoder model based on training data and test verification data, and generating a target self-encoder model; acquiring real-time monitored data to be detected, and carrying out abnormal recognition on the data to be detected based on a target self-encoder model to obtain a recognition result; if the identification result is that the data to be tested is abnormal, generating alarm information and sending the alarm information out. The monitoring efficiency is improved, so that the reliability and the safety of the Internet of things system are improved, and meanwhile, the workload of maintenance and monitoring is reduced.
Description
Technical Field
The application relates to the technical field of the internet of things, in particular to a monitoring alarm method and device based on a self-encoder, an internet of things platform and a medium.
Background
In order to discover potential faults in the first time, a high availability and stability of business services are guaranteed, a current mainstream open source monitoring system consists of time sequence data of basic indexes and alarm triggers, corresponding agents are installed on a server to report real-time running state index data, the real-time running state index data is stored and inquired through a Prometaus time sequence database, and related personnel are reminded to pay attention to the alarms through alert rules of alert manager configuration and pushing short messages, mails, nails and the like.
With the rapid development of traffic, the existing monitoring and alarming scheme has a plurality of limitations in use. The alarm rule is seriously dependent on manual experience to set a corresponding threshold value, and because the system index time sequence data has high dimensionality and the fluctuation of a single index is not equal to abnormality, repeated practice is required to be configured for one high-availability rule, the false alarm is extremely serious, and operation and maintenance personnel are easily submerged by massive invalid alarms; under different time periods, the time sequence data of the operation of the service system can be changed continuously, even the time sequence data presents a periodic change rule, the corresponding threshold values can be different, and a reasonable threshold value is difficult to be configured; and the prior art cannot adapt to the service guarantee problem of the same system of multiple sets of different clients in the 2B service scene of the Internet of things. Therefore, a monitoring method is needed to improve the monitoring efficiency, thereby improving the reliability and the safety of the internet of things system and reducing the workload of maintenance and monitoring.
Disclosure of Invention
The embodiment of the application aims to provide a monitoring alarm method and device based on a self-encoder, an Internet of things platform and medium, so that the monitoring efficiency is improved, the reliability and the safety of an Internet of things system are improved, and meanwhile, the workload of maintenance and monitoring is reduced.
In order to solve the above technical problems, an embodiment of the present application provides a monitoring alarm method based on a self-encoder, including:
collecting initial data from the sensors and the devices and storing the initial data in a time series database;
performing data preprocessing on the initial data in the time sequence database to obtain target data, and dividing the target data into training data and test verification data;
constructing a self-encoder model, training and fitting the self-encoder model based on the training data and the test verification data, and generating a target self-encoder model;
acquiring real-time monitored data to be detected, and carrying out abnormal recognition on the data to be detected based on the target self-encoder model to obtain a recognition result;
and if the identification result is that the data to be detected is abnormal, generating alarm information and sending the alarm information out.
Further, the collecting initial data from the sensors and the devices and storing the initial data in a time series database includes:
collecting the initial data from the sensor and the device, wherein the initial data is time-series data;
and transmitting the initial data to a monitoring platform based on a preset communication protocol, and storing the initial data in the time sequence database.
Further, the data preprocessing is performed on the initial data in the time sequence database to obtain target data, and the target data is divided into training data and test verification data, including:
identifying an abnormal value of the initial data in the time sequence database, and removing the abnormal value to obtain first processing data;
performing data missing value processing on the first processing data to obtain second processing data;
performing data standardization processing on the second processing data to obtain the target data;
dividing the target data into the training data and the test verification data, wherein the training data is the target data which does not comprise abnormal data, and the test verification data is the target data which comprises abnormal data and normal data.
Further, the constructing a self-encoder model, and training and fitting the self-encoder model based on the training data and the test verification data, to generate a target self-encoder model, includes:
creating the self-encoder model of the three-layer neural network by calling a Tensorflow framework API interface;
and training and fitting the self-encoder model based on the training data and the test verification data by adopting a mean square error as a model loss function so that the self-encoder model captures a normal operation mode, and generating the target self-encoder model.
Further, the obtaining the data to be monitored in real time, and performing anomaly identification on the data to be monitored based on the target self-encoder model, to obtain an identification result, includes:
acquiring the data to be detected monitored in real time, and inputting the data to be detected into the target self-encoder model;
reconstructing the data to be detected into a normal mode through the target self-encoder model, and calculating a reconstruction error during data reconstruction;
judging whether the reconstruction error exceeds a preset error threshold value or not to obtain a judging result;
and generating the identification result based on the judgment result.
Further, if the identification result is that the data to be detected is abnormal, generating alarm information, and sending the alarm information out, including:
if the identification result is that the data to be detected is abnormal, generating the alarm information based on the data to be detected, wherein the alarm information comprises an abnormal type, a device identifier and a time stamp;
and sending the alarm information to an operation and maintenance end so that the operation and maintenance end can take preset maintenance measures according to the alarm information for maintenance.
Further, if the identification result is that the data to be detected is abnormal, generating alarm information, and after the alarm information is sent out, the method further includes:
acquiring historical monitoring data and historical alarm information according to a preset time interval, and updating the target self-encoder model based on the historical monitoring data and the historical alarm information;
and visually displaying the history monitoring data and the history alarm information.
In order to solve the above technical problem, an embodiment of the present application provides a monitoring alarm device based on a self-encoder, including:
a data collection unit for collecting initial data from the sensor and the device and storing the initial data in a time series database;
the data processing unit is used for carrying out data preprocessing on the initial data in the time sequence database to obtain target data, and dividing the target data into training data and test verification data;
the model training unit is used for constructing a self-encoder model, training and fitting the self-encoder model based on the training data and the test verification data, and generating a target self-encoder model;
the anomaly identification unit is used for acquiring the data to be detected monitored in real time, and carrying out anomaly identification on the data to be detected based on the target self-encoder model to obtain an identification result;
and the alarm generating unit is used for generating alarm information and sending the alarm information outwards if the identification result is that the data to be detected are abnormal.
In order to solve the technical problems, the invention adopts a technical scheme that: the Internet of things platform comprises one or more processors; and the memory is used for storing one or more programs, so that the one or more processors can realize the self-encoder-based monitoring alarm method.
In order to solve the technical problems, the invention adopts a technical scheme that: a computer readable storage medium having stored thereon a computer program which when executed by a processor implements the self-encoder based monitoring alarm method of any of the above.
The embodiment of the invention provides a monitoring alarm method and device based on a self-encoder, an Internet of things platform and a medium. The method comprises the following steps: collecting initial data from the sensors and the devices and storing the initial data in a time series database; performing data preprocessing on the initial data in the time sequence database to obtain target data, and dividing the target data into training data and test verification data; constructing a self-encoder model, training and fitting the self-encoder model based on the training data and the test verification data, and generating a target self-encoder model; acquiring real-time monitored data to be detected, and carrying out abnormal recognition on the data to be detected based on the target self-encoder model to obtain a recognition result; and if the identification result is that the data to be detected is abnormal, generating alarm information and sending the alarm information out. According to the embodiment of the invention, the self-encoder model is trained, and the trained self-encoder model is deployed in the monitoring system, so that the data to be detected in the monitoring system is subjected to abnormal detection, the situations of manually setting a threshold value and misinformation and missing report are avoided, the monitoring efficiency is improved, the reliability and the safety of the Internet of things system are improved, and the workload of maintenance and monitoring is reduced.
Drawings
For a clearer description of the solution in the present application, a brief description will be given below of the drawings that are needed in the description of the embodiments of the present application, it being obvious that the drawings in the following description are some embodiments of the present application, and that other drawings may be obtained from these drawings without inventive effort for a person of ordinary skill in the art.
FIG. 1 is a flowchart of a self-encoder based monitoring and alarming method according to an embodiment of the present application;
FIG. 2 is a flowchart of an implementation of a sub-process of the self-encoder based monitoring and warning method provided in an embodiment of the present application;
FIG. 3 is a flowchart of an implementation of a sub-process of the self-encoder based monitoring and warning method provided in an embodiment of the present application;
FIG. 4 is a flowchart of an implementation of a sub-process of the self-encoder based monitoring and warning method provided in an embodiment of the present application;
fig. 5 is a schematic load forwarding diagram of an available area service example provided by an embodiment of the present application;
fig. 6 is a schematic load forwarding diagram of an available area service example provided by an embodiment of the present application;
FIG. 7 is a schematic diagram of a self-encoder based monitoring alarm provided in an embodiment of the present application;
fig. 8 is a schematic diagram of an internet of things platform provided in an embodiment of the present application.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the applications herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having" and any variations thereof in the description and claims of the present application and in the description of the figures above are intended to cover non-exclusive inclusions. The terms first, second and the like in the description and in the claims or in the above-described figures, are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
In order to better understand the technical solutions of the present application, the following description will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings.
The present invention will be described in detail with reference to the drawings and embodiments.
It should be noted that, the monitoring and alarming method based on the self-encoder provided in the embodiments of the present application is generally executed by the internet of things platform, and accordingly, the monitoring and alarming device based on the self-encoder is generally configured in the internet of things platform.
Referring to fig. 1, fig. 1 illustrates one embodiment of a self-encoder based supervisory alarm method.
It should be noted that, if there are substantially the same results, the method of the present invention is not limited to the flow sequence shown in fig. 1, and the method includes the following steps:
s1: initial data from the sensors and devices is collected and stored in a time series database.
Specifically, the embodiment of the application collects data from various sensors and devices in the environment of the internet of things, and takes the data as initial data; the initial is then stored in a time series database.
Referring to fig. 2, fig. 2 shows a specific embodiment of step S1, which is described in detail as follows:
s11: the initial data from the sensor and the device is collected, wherein the initial data is time series data.
S12: and transmitting the initial data to a monitoring platform based on a preset communication protocol, and storing the initial data in the time sequence database.
Specifically, the initial data acquired in the embodiment of the present application is time-series data, which includes data such as temperature, humidity, pressure, current, and voltage. And then transmitting the initial data to a monitoring platform based on a preset communication protocol, and storing the initial data in a time sequence database, so as to form sample time sequence data for training a neural network algorithm. The preset communication protocol comprises TCP/IP protocol, HTTP protocol, IMAP protocol, SNMP protocol and the like.
S2: and carrying out data preprocessing on the initial data in the time sequence database to obtain target data, and dividing the target data into training data and test verification data.
Specifically, in the embodiment of the present application, data preprocessing is performed on initial data in a time sequence database to obtain target data. The data preprocessing comprises operations such as outlier removal, data missing value processing, data normalization and the like. The target data is then divided into training data and test verification data.
Referring to fig. 3, fig. 3 shows a specific embodiment of step S2, which is described in detail as follows:
s21: and identifying the abnormal value of the initial data in the time sequence database, and removing the abnormal value to obtain first processing data.
S22: and carrying out data missing value processing on the first processing data to obtain second processing data.
S23: and carrying out data standardization processing on the second processing data to obtain the target data.
S24: dividing the target data into the training data and the test verification data, wherein the training data is the target data which does not comprise abnormal data, and the test verification data is the target data which comprises abnormal data and normal data.
Specifically, in the embodiment of the present application, an abnormal value of initial data in a time sequence database needs to be identified first, and the abnormal value is removed, so as to obtain first processing data; and then recognizing the first processing data to recognize the data missing value, wherein the nodes which can be used for recognizing the data missing value comprise nodes such as attribute generation, description data characteristics, data filtering nodes and the like to recognize the data missing value. After the missing data value is identified, the missing data value is subjected to processing such as data filtering and filling by a missing value filling rule. And after the second processing data is obtained, carrying out data standardization processing on the second processing data to obtain target data. Finally, dividing the target data into training data and test verification data, wherein the training data is the target data which does not comprise abnormal data, namely the value of the training data comprises normal data in the target data; the test verification data is target data including abnormal data and normal data. Its test verification data may contain anomaly data that may be used to perform model verification, thereby improving the robustness of the model.
S3: and constructing a self-encoder model, training and fitting the self-encoder model based on the training data and the test verification data, and generating a target self-encoder model.
Referring to fig. 4, fig. 4 shows a specific embodiment of step S3, which is described in detail as follows:
s31: and creating the self-encoder model of the three-layer neural network by calling a Tensorflow framework API interface.
S32: and training and fitting the self-encoder model based on the training data and the test verification data by adopting a mean square error as a model loss function so that the self-encoder model captures a normal operation mode, and generating the target self-encoder model.
Specifically, in the embodiment of the application, the self-encoder model is used for monitoring the internet of things system. The self-encoder is an (Auto-encoding, AZ) neural network that uses a back-propagation algorithm to make the output value equal to the input value, which compresses the input into a potential spatial representation, and then reconstructs the output from this representation. The self-encoder is a self-supervising algorithm that automatically learns the rules from the data samples.
In the embodiment of the application, a Tensorflow framework api interface is called, and a self-encoder model of the three-layer neural network is created. Inputting training data into a self-coding model for training according to batches by a data loader, taking a mean square error as a model loss function in the training process, adjusting model parameters of the self-coding model when a model loss value is calculated to be not up to a preset value, and then re-inputting the model parameters into the self-coding model for re-training by adopting a back propagation algorithm; meanwhile, the nonlinear learning capacity of the model is enhanced by adopting an activation function in the training process, and the self-encoder model is fitted through training data and test verification data, so that the self-encoder model can capture a normal operation mode, and a target self-encoder model is generated.
S4: and acquiring the data to be detected monitored in real time, and carrying out abnormal recognition on the data to be detected based on the target self-encoder model to obtain a recognition result.
Specifically, the target self-encoder model has been trained in the above steps, and in the embodiment of the present application, the target self-encoder model is deployed in a monitoring system of an internet of things system, so that real-time monitoring is achieved.
Referring to fig. 5, fig. 5 shows a specific embodiment of step S4, which is described in detail as follows:
s41: and acquiring the data to be detected monitored in real time, and inputting the data to be detected into the target self-encoder model.
S42: reconstructing the data to be detected into a normal mode through the target self-encoder model, and calculating a reconstruction error during data reconstruction.
S43: and judging whether the reconstruction error exceeds a preset error threshold value or not to obtain a judging result.
S44: and generating the identification result based on the judgment result.
Specifically, acquiring data to be monitored in real time, and inputting the data to be monitored into a target self-encoder model; reconstructing the data to be detected into a normal mode through a target self-encoder model, calculating a reconstruction error during data reconstruction, and judging whether the reconstruction error exceeds a preset error threshold value or not to obtain a judgment result; and generating a recognition result based on the judgment result. In one embodiment, if the reconstruction error exceeds 3 times of root mean square error, the recognition result is that the data to be detected is abnormal; if the reconstruction error does not exceed the 3 times root mean square error, the identification result is that the data to be detected is not abnormal.
S5: and if the identification result is that the data to be detected is abnormal, generating alarm information and sending the alarm information out.
Referring to fig. 6, fig. 6 shows a specific embodiment of step S5, which is described in detail as follows:
s51: if the identification result is that the data to be detected is abnormal, generating the alarm information based on the data to be detected, wherein the alarm information comprises an abnormal type, a device identifier and a time stamp.
S52: and sending the alarm information to an operation and maintenance end so that the operation and maintenance end can take preset maintenance measures according to the alarm information for maintenance.
Specifically, when the identification result is that the data to be detected is abnormal, the alarm information is generated based on the data to be detected, and then the alarm information is sent to the operation and maintenance end, so that the operation and maintenance end takes preset maintenance measures according to the alarm information. The alarm information comprises an anomaly type, a device identifier and a time stamp. The operation and maintenance end can take preset maintenance measures for maintenance, wherein the preset maintenance measures comprise remote restarting of equipment, maintenance or replacement of faulty equipment.
S6: and acquiring historical monitoring data and historical alarm information according to a preset time interval, and updating the target self-encoder model based on the historical monitoring data and the historical alarm information.
Specifically, in the embodiment of the application, the historical monitoring data and the historical alarm information are acquired according to the preset time interval, and the target self-encoder model is updated based on the historical monitoring data and the historical alarm information, so that the model can adapt to the change and evolution of the equipment behavior, and the monitoring accuracy is improved.
S7: and visually displaying the history monitoring data and the history alarm information.
Specifically, the embodiment of the application uses a grafana-like instrument panel and a visualization tool to visually display the history monitoring data and the history warning information. It should be noted that the monitoring data and the alarm information may also be displayed in real time.
Further, the embodiment of the application can be used for various complex business scenes of the internet of things, including industrial automation, intelligent buildings, intelligent cities, agricultural monitoring and the like. The model is used for intelligently detecting various time sequence data of software and hardware and response abnormality, thereby helping to improve the usability of equipment, reduce the maintenance cost and improve the service efficiency.
In the embodiment of the application, initial data from a sensor and equipment are collected, and the initial data are stored in a time sequence database; performing data preprocessing on the initial data in the time sequence database to obtain target data, and dividing the target data into training data and test verification data; constructing a self-encoder model, training and fitting the self-encoder model based on the training data and the test verification data, and generating a target self-encoder model; acquiring real-time monitored data to be detected, and carrying out abnormal recognition on the data to be detected based on the target self-encoder model to obtain a recognition result; and if the identification result is that the data to be detected is abnormal, generating alarm information and sending the alarm information out. According to the embodiment of the invention, the self-encoder model is trained, and the trained self-encoder model is deployed in the monitoring system, so that the data to be detected in the monitoring system is subjected to abnormal detection, the situations of manually setting a threshold value and misinformation and missing report are avoided, the monitoring efficiency is improved, the reliability and the safety of the Internet of things system are improved, and the workload of maintenance and monitoring is reduced.
Further, customization and expandability can be achieved by the embodiment of the application, customization training can be conducted according to different business scenes of the Internet of things by using the self-encoder, different types of equipment and sensors can be provided with different self-encoder models so as to meet specific requirements, and the device and the method have wide expandability.
Further, the embodiment of the application can realize anomaly detection and fault prediction, and the self-encoder of the embodiment of the application is a self-supervision learning algorithm and can be used for detecting anomalies in time series data. In an internet of things business scenario, sensors and devices generate a large amount of time series data. The self-encoder may be used to detect abnormal patterns in the sensor data to predict potential equipment failure or problems in advance.
Further, the embodiment of the application can reduce the false alarm rate. Conventional monitoring systems may generate a large number of false positives, wasting operator time and resources. The self-encoder can reduce false alarm rate by learning normal data mode, and only trigger alarm under the condition of really existing abnormality, thereby improving the actual efficiency of the monitoring system.
Further, the embodiment of the application can realize monitoring and response. The self-encoder in the embodiment of the application can process the sensor data in real time and rapidly detect the abnormality. This means that when problems occur in the internet of things equipment or system, measures can be taken quickly, reducing potential downtime or losses.
Further, the safety of the Internet of things system can be improved, and in the Internet of things, the safety is important. The self-encoder may detect unauthorized device connections or abnormal network traffic situations.
Referring to fig. 7, as an implementation of the method shown in fig. 1, the present application provides an embodiment of a self-encoder-based monitoring alarm device, where the embodiment of the device corresponds to the embodiment of the method shown in fig. 1, and the device may be specifically applied to an internet of things platform.
As shown in fig. 7, the self-encoder based monitoring alarm apparatus of the present embodiment includes: a data collection unit 81, a data processing unit 82, a model training unit 83, an abnormality recognition unit 84, an alarm generation unit 85, a model updating unit 86, and a visual presentation unit 87, wherein:
a data collection unit 81 for collecting initial data from the sensors and devices and storing the initial data in a time series database;
the data processing unit 82 is configured to perform data preprocessing on the initial data in the time sequence database to obtain target data, and divide the target data into training data and test verification data;
a model training unit 83, configured to construct a self-encoder model, and train and fit the self-encoder model based on the training data and the test verification data, to generate a target self-encoder model;
the anomaly identification unit 84 is configured to obtain real-time monitored data to be detected, and perform anomaly identification on the data to be detected based on the target self-encoder model, so as to obtain an identification result;
the alarm generating unit 85 is configured to generate alarm information if the identification result indicates that the data to be tested is abnormal, and send the alarm information out;
a model updating unit 86, configured to acquire historical monitoring data and historical alarm information according to a preset time interval, and update the target self-encoder model based on the historical monitoring data and the historical alarm information;
and the visual display unit 87 is used for visually displaying the history monitoring data and the history alarm information.
Further, the data collection unit 81 includes:
an initial data collection unit for collecting the initial data from the sensor and the device, wherein the initial data is time-series data;
and the initial data transmission unit is used for transmitting the initial data to the monitoring platform based on a preset communication protocol and storing the initial data in the time sequence database.
Further, the data processing unit 82 includes:
the first processing data generation unit is used for identifying the abnormal value of the initial data in the time sequence database and removing the abnormal value to obtain first processing data;
the second processing data generating unit is used for carrying out data missing value processing on the first processing data to obtain second processing data;
the target data generating unit is used for carrying out data standardization processing on the second processing data to obtain target data;
the target data dividing unit is used for dividing the target data into the training data and the test verification data, wherein the training data is the target data which does not comprise abnormal data, and the test verification data is the target data which comprises abnormal data and normal data.
Further, the model training unit 83 includes:
the model creation unit is used for creating the self-encoder model of the three-layer neural network by calling a Tensorflow framework API interface;
and the target self-encoder model generating unit is used for training and fitting the self-encoder model based on the training data and the test verification data by adopting a mean square error as a model loss function so that the self-encoder model captures a normal operation mode and generates the target self-encoder model.
Further, the abnormality identifying unit 84 includes:
the data acquisition unit to be detected is used for acquiring the data to be detected monitored in real time and inputting the data to be detected into the target self-encoder model;
a reconstruction error calculation unit, configured to reconstruct the data to be detected into a normal mode through the target self-encoder model, and calculate a reconstruction error during data reconstruction;
the judging result generating unit is used for judging whether the reconstruction error exceeds a preset error threshold value or not to obtain a judging result;
and the identification result generating unit is used for generating the identification result based on the judgment result.
Further, the alarm generating unit 85 includes:
the alarm information generation unit is used for generating alarm information based on the data to be detected if the identification result is that the data to be detected is abnormal, wherein the alarm information comprises an abnormal type, a device identifier and a time stamp;
and the alarm information sending unit is used for sending the alarm information to the operation and maintenance end so that the operation and maintenance end can take preset maintenance measures according to the alarm information to carry out maintenance.
In the embodiment of the application, initial data from a sensor and equipment are collected, and the initial data are stored in a time sequence database; performing data preprocessing on the initial data in the time sequence database to obtain target data, and dividing the target data into training data and test verification data; constructing a self-encoder model, training and fitting the self-encoder model based on the training data and the test verification data, and generating a target self-encoder model; acquiring real-time monitored data to be detected, and carrying out abnormal recognition on the data to be detected based on the target self-encoder model to obtain a recognition result; and if the identification result is that the data to be detected is abnormal, generating alarm information and sending the alarm information out. According to the embodiment of the invention, the self-encoder model is trained, and the trained self-encoder model is deployed in the monitoring system, so that the data to be detected in the monitoring system is subjected to abnormal detection, the situations of manually setting a threshold value and misinformation and missing report are avoided, the monitoring efficiency is improved, the reliability and the safety of the Internet of things system are improved, and the workload of maintenance and monitoring is reduced.
In order to solve the technical problems, the embodiment of the application also provides an internet of things platform. Referring specifically to fig. 8, fig. 8 is a basic structural block diagram of the platform of the internet of things according to the present embodiment.
The internet of things platform 9 comprises a memory 91, a processor 92, a network interface 93 which are communicatively connected to each other via a system bus. It should be noted that only the internet of things platform 9 having three components memory 91, processor 92, network interface 93 is shown, but it should be understood that not all of the illustrated components are required to be implemented and that more or fewer components may be implemented instead. It will be appreciated by those skilled in the art that the internet of things platform herein is a device capable of automatically performing numerical calculations and/or information processing according to preset or stored instructions, and the hardware thereof includes, but is not limited to, microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, and the like.
The memory 91 includes at least one type of readable storage medium including flash memory, hard disk, multimedia card, card memory (e.g., SD or DX memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), programmable Read Only Memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 91 may be an internal storage unit of the internet of things platform 9, for example, a hard disk or a memory of the internet of things platform 9. In other embodiments, the memory 91 may also be an external storage device of the internet of things platform 9, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the internet of things platform 9. Of course, the memory 91 may also include both an internal storage unit of the internet of things platform 9 and an external storage device thereof. In this embodiment, the memory 91 is generally used for storing an operating system and various application software installed on the internet of things platform 9, such as program codes of a self-encoder based monitoring alarm method. Further, the memory 91 may be used to temporarily store various types of data that have been output or are to be output.
The processor 92 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 92 is typically used to control the overall operation of the internet of things platform 9. In this embodiment, the processor 92 is configured to execute the program code stored in the memory 91 or process data, for example, execute the program code of the self-encoder based monitoring alarm method described above, to implement various embodiments of the self-encoder based monitoring alarm method.
The network interface 93 may comprise a wireless network interface or a wired network interface, which network interface 93 is typically used to establish a communication connection between the internet of things platform 9 and other electronic devices.
The present application also provides another embodiment, namely, a computer readable storage medium, where a computer program is stored, where the computer program is executable by at least one processor, so that the at least one processor performs the steps of a self-encoder based monitoring alarm method as described above.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method of the embodiments of the present application.
It is apparent that the embodiments described above are only some embodiments of the present application, but not all embodiments, the preferred embodiments of the present application are given in the drawings, but not limiting the patent scope of the present application. This application may be embodied in many different forms, but rather, embodiments are provided in order to provide a more thorough understanding of the present disclosure. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that modifications may be made to the embodiments described in the foregoing, or equivalents may be substituted for elements thereof. All equivalent structures made by the specification and the drawings of the application are directly or indirectly applied to other related technical fields, and are also within the protection scope of the application.
Claims (10)
1. A self-encoder based monitoring alarm method, comprising:
collecting initial data from the sensors and the devices and storing the initial data in a time series database;
performing data preprocessing on the initial data in the time sequence database to obtain target data, and dividing the target data into training data and test verification data;
constructing a self-encoder model, training and fitting the self-encoder model based on the training data and the test verification data, and generating a target self-encoder model;
acquiring real-time monitored data to be detected, and carrying out abnormal recognition on the data to be detected based on the target self-encoder model to obtain a recognition result;
and if the identification result is that the data to be detected is abnormal, generating alarm information and sending the alarm information out.
2. The self-encoder based supervisory alert method as recited in claim 1, wherein the collecting initial data from the sensors and devices and storing the initial data in a time series database comprises:
collecting the initial data from the sensor and the device, wherein the initial data is time-series data;
and transmitting the initial data to a monitoring platform based on a preset communication protocol, and storing the initial data in the time sequence database.
3. The method of claim 1, wherein the performing data preprocessing on the initial data in the time-series database to obtain target data, and dividing the target data into training data and test verification data comprises:
identifying an abnormal value of the initial data in the time sequence database, and removing the abnormal value to obtain first processing data;
performing data missing value processing on the first processing data to obtain second processing data;
performing data standardization processing on the second processing data to obtain the target data;
dividing the target data into the training data and the test verification data, wherein the training data is the target data which does not comprise abnormal data, and the test verification data is the target data which comprises abnormal data and normal data.
4. The method of claim 1, wherein the constructing a self-encoder model and training and fitting the self-encoder model based on the training data and the test verification data to generate a target self-encoder model comprises:
creating the self-encoder model of the three-layer neural network by calling a Tensorflow framework API interface;
and training and fitting the self-encoder model based on the training data and the test verification data by adopting a mean square error as a model loss function so that the self-encoder model captures a normal operation mode, and generating the target self-encoder model.
5. The method for monitoring and alarming based on the self-encoder according to claim 1, wherein the steps of obtaining the data to be monitored in real time, and performing anomaly recognition on the data to be monitored based on the target self-encoder model to obtain a recognition result include:
acquiring the data to be detected monitored in real time, and inputting the data to be detected into the target self-encoder model;
reconstructing the data to be detected into a normal mode through the target self-encoder model, and calculating a reconstruction error during data reconstruction;
judging whether the reconstruction error exceeds a preset error threshold value or not to obtain a judging result;
and generating the identification result based on the judgment result.
6. The method for monitoring and alarming based on the self-encoder according to any one of claims 1 to 5, wherein if the identification result is that the data to be tested is abnormal, generating alarm information and sending the alarm information out, includes:
if the identification result is that the data to be detected is abnormal, generating the alarm information based on the data to be detected, wherein the alarm information comprises an abnormal type, a device identifier and a time stamp;
and sending the alarm information to an operation and maintenance end so that the operation and maintenance end can take preset maintenance measures according to the alarm information for maintenance.
7. The method for monitoring and alarming based on a self-encoder according to any one of claims 1 to 5, wherein if the identification result is that the data to be tested is abnormal, generating alarm information, and after the alarm information is sent out, the method further comprises:
acquiring historical monitoring data and historical alarm information according to a preset time interval, and updating the target self-encoder model based on the historical monitoring data and the historical alarm information;
and visually displaying the history monitoring data and the history alarm information.
8. A self-encoder based monitoring alarm apparatus comprising:
a data collection unit for collecting initial data from the sensor and the device and storing the initial data in a time series database;
the data processing unit is used for carrying out data preprocessing on the initial data in the time sequence database to obtain target data, and dividing the target data into training data and test verification data;
the model training unit is used for constructing a self-encoder model, training and fitting the self-encoder model based on the training data and the test verification data, and generating a target self-encoder model;
the anomaly identification unit is used for acquiring the data to be detected monitored in real time, and carrying out anomaly identification on the data to be detected based on the target self-encoder model to obtain an identification result;
and the alarm generating unit is used for generating alarm information and sending the alarm information outwards if the identification result is that the data to be detected are abnormal.
9. An internet of things platform, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the self-encoder based monitoring alarm method according to any one of claims 1 to 7 when executing the computer program.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the self-encoder based monitoring alarm method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311701646.8A CN117527552A (en) | 2023-12-12 | 2023-12-12 | Monitoring alarm method and device based on self-encoder, internet of things platform and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311701646.8A CN117527552A (en) | 2023-12-12 | 2023-12-12 | Monitoring alarm method and device based on self-encoder, internet of things platform and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117527552A true CN117527552A (en) | 2024-02-06 |
Family
ID=89747850
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311701646.8A Pending CN117527552A (en) | 2023-12-12 | 2023-12-12 | Monitoring alarm method and device based on self-encoder, internet of things platform and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117527552A (en) |
-
2023
- 2023-12-12 CN CN202311701646.8A patent/CN117527552A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109001649B (en) | Intelligent power supply diagnosis system and protection method | |
| CN107369303B (en) | Factory's intelligent diagnosing method, apparatus and system | |
| JP7007243B2 (en) | Anomaly detection system | |
| KR20180108446A (en) | System and method for management of ict infra | |
| CN104246636A (en) | Method and system for real-time performance degradation advisory for centrifugal compressors | |
| CN114567538A (en) | Alarm information processing method and device | |
| CN111445034A (en) | System and method for predicting industrial equipment failure | |
| CN116125958A (en) | Intelligent factory fault diagnosis and decision-making system based on digital twinning | |
| CN114200877B (en) | Monitoring method and device for electric equipment | |
| CN117235443A (en) | Electric power operation safety monitoring method and system based on edge AI | |
| Mao et al. | A visual model-based evaluation framework of cloud-based prognostics and health management | |
| CN110888782B (en) | Device processing method, system, electronic device and computer readable storage medium | |
| US11347610B2 (en) | Method and system for assisting troubleshooting of a complex system | |
| CN112506754A (en) | System performance monitoring method and platform | |
| CN117220917A (en) | Network real-time monitoring method based on cloud computing | |
| CN117527552A (en) | Monitoring alarm method and device based on self-encoder, internet of things platform and medium | |
| CN116468423A (en) | Operation and maintenance emergency coordination method, system and terminal equipment | |
| CN113835961B (en) | Alarm information monitoring method, device, server and storage medium | |
| CN116310521A (en) | Abnormality detection method, abnormality detection device, abnormality detection storage medium, abnormality detection electronic device, abnormality detection system, abnormality detection program, and recording medium | |
| CN115964935A (en) | Data center machine room IT equipment management method, device, server and medium | |
| CN110069382A (en) | Software supervision method, server, terminal device, computer equipment and medium | |
| JP2019191880A (en) | Equipment management support system | |
| CN112486773A (en) | Data processing method, device and system | |
| CN113011745A (en) | Abnormity detection method, device, equipment and medium in power grid safety operation and maintenance | |
| CN112804104A (en) | Early warning method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |