CN116628425A - Big data real-time monitoring system and method - Google Patents
Big data real-time monitoring system and method Download PDFInfo
- Publication number
- CN116628425A CN116628425A CN202310642151.6A CN202310642151A CN116628425A CN 116628425 A CN116628425 A CN 116628425A CN 202310642151 A CN202310642151 A CN 202310642151A CN 116628425 A CN116628425 A CN 116628425A
- Authority
- CN
- China
- Prior art keywords
- big data
- dragonfly
- iteration
- fnn
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 40
- 238000000034 method Methods 0.000 title claims abstract description 18
- 241000238633 Odonata Species 0.000 claims abstract description 87
- 238000013528 artificial neural network Methods 0.000 claims abstract description 40
- 238000001514 detection method Methods 0.000 claims abstract description 37
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 32
- 238000005457 optimization Methods 0.000 claims abstract description 31
- 238000012545 processing Methods 0.000 claims abstract description 27
- 230000002159 abnormal effect Effects 0.000 claims abstract description 11
- 230000005856 abnormality Effects 0.000 claims abstract description 7
- 230000006399 behavior Effects 0.000 claims description 9
- 238000004140 cleaning Methods 0.000 claims description 4
- 230000002431 foraging effect Effects 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 4
- 238000000926 separation method Methods 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000005295 random walk Methods 0.000 claims description 3
- 238000007405 data analysis Methods 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 16
- 239000013598 vector Substances 0.000 description 4
- 230000006978 adaptation Effects 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000002068 genetic effect Effects 0.000 description 1
- 210000005036 nerve Anatomy 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000010415 tropism Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/004—Artificial life, i.e. computing arrangements simulating life
- G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computational Linguistics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to the technical field of data analysis, in particular to a big data real-time monitoring system and method, comprising the following steps: the monitoring acquisition processing module: the system is used for monitoring and collecting big data in real time and processing the collected big data; and (3) a processing model building module: the FNN is used for building a big data operation system and carrying out parameter optimization; big data detection module: the method comprises the steps of detecting big data characteristic data through a FNN neural network after parameter optimization; an abnormality alarm module: and the alarm is performed according to the big data characteristic data which are detected to be abnormal by the big data detection module. According to the invention, through building the FNN of the big data, optimizing the parameters of the FNN by optimizing the dragonfly algorithm, and dynamically adjusting partial weight factors of the dragonfly algorithm, the parameter optimizing efficiency is improved, the best overhaul time is obtained for an overhaul worker, and the damage degree of a big data operation system is reduced.
Description
Technical Field
The invention relates to the technical field of data analysis, in particular to a big data real-time monitoring system and method.
Background
Big data requires new processing modes to have stronger insight discoveries, decision making forces and process optimization capabilities to accommodate high growth rate, massive and diverse information assets. Big data has four big characteristics of huge data scale, various data types, quick data transfer and low value density. There are many algorithms already available in big data monitoring analysis. When the big data monitoring system is used at present, the influence of subjectivity on an evaluation result can be better overcome by using the FNN neural network algorithm, because the FNN neural network has nonlinear transformation and self-regulation learning capabilities, but the FNN neural network algorithm is easy to have a problem of local minimum value, and the FNN neural network parameter is optimized by using a Genetic Algorithm (GA), so that the accuracy of the FNN neural network can be improved, but the phenomenon of unstable swing near an optimal solution can occur; the particle swarm is used for optimizing the weight and the threshold of the FNN neural network, so that the global optimal value can be found more accurately, but the requirement of high-efficiency operation is difficult to meet under the condition of huge data quantity.
Disclosure of Invention
The invention aims to solve the defects in the background technology by providing a big data real-time monitoring system and a big data real-time monitoring method. Therefore, the big data monitoring system is combined with the neural network, and parameters of the FNN neural network are optimized through optimization of the dragonfly algorithm, so that a large amount of data can be rapidly processed, global and local optimization is met, the big data real-time monitoring result is more accurate, and the system performance is further improved.
The technical scheme adopted by the invention is as follows:
provided is a big data real-time monitoring system, comprising:
the monitoring acquisition processing module: the system is used for monitoring and collecting big data in real time and processing the collected big data;
and (3) a processing model building module: the FNN is used for building a big data operation system and carrying out parameter optimization;
big data detection module: the method comprises the steps of detecting big data characteristic data through a FNN neural network after parameter optimization;
an abnormality alarm module: and the alarm is performed according to the big data characteristic data which are detected to be abnormal by the big data detection module.
As a preferred technical scheme of the invention: the model building module builds the FNN based on the obtained big data characteristic data and optimizes weights and thresholds among an input layer, a hidden layer and an output layer in the FNN based on an optimization dragonfly algorithm.
As a preferred technical scheme of the invention: the optimization dragonfly algorithm is as follows:
X i (t+1)＝X i (t)+ΔX i (t+1)
ΔX i (t+1)＝[aA i (t)+bB i (t)+cC i (t)+dD i (t)+eE i (t)]+αΔX i (t)
wherein X is i (t+1) represents the position of the t+1st iteration dragonfly i, X i (t) represents the position of the t-th iteration dragonfly i, deltaX i (t+1) represents the moving step length of the t+1st iteration dragonfly i, alpha represents the inertial weight coefficient, and DeltaX i (t) represents the moving step length of the t-th iteration dragonfly i, A i (t) represents the separation degree of the t-th iteration dragonfly i, B i (t) represents the alignment speed of the ith iteration dragonfly, C i (t) represents the aggregation degree of the t-th iteration dragonfly i, D i (t) represents the foraging ability of the t-th iteration dragonfly i, E i (t) represents the enemy-avoiding capacity of the t iteration dragonfly i, and a, b, c, d, e is the weight corresponding to the five behaviors respectively;
A i (t)、B i (t)、C i (t)、D i (t)、E i (t) satisfies:
D i (t)＝X + (t)-X i (t)
E i (t)＝X-(t)-X i (t)
wherein X is j (t) represents the positions of adjacent dragonflies j of the t-th iteration dragonfly i, and N represents the total number of dragonflies; v (V) j (t) represents the velocity of adjacent dragonflies j of the t-th iteration dragonfly i; x is X + (t) represents the position of the food item at the t-th iteration; x is X - (t) represents the location of the natural enemy of the t-th iteration;
when X is i (t) in the absence of nearby individuals around X i (t) updating with random walk behavior:
wherein r is 1 、r 2 Is a random number between (0, 1), f is a constant, σ is a factorial function, and σ (f) =f-! ,
as a preferred technical scheme of the invention: the monitoring acquisition processing module is used for carrying out data cleaning and data normalization processing on the acquired big data and carrying out feature extraction on the processed big data.
As a preferred technical scheme of the invention: in the optimization dragonfly algorithm, a weight b=c is set, and the weights b and c are dynamically adjusted as follows:
wherein b 'and c' represent weights b and c corresponding to weights before dynamic adjustment, β 1 、β 2 And beta 3 Representing the control coefficient, b max And b min Representing weights b' maximum and minimum, c max And c min The weights c' maximum and minimum are shown, T representing the maximum number of iterations.
As a preferred technical scheme of the invention: in the processing model building module, a FNN neural network structure is built for each node in the big data operation system, and the FNN neural network structure of each node is optimized by adopting an optimization dragonfly algorithm.
As a preferred technical scheme of the invention: the big data detection module detects the input big data characteristic nerve based on a detection function g (x) of the FNN neural network:
wherein Q is I Represents a set of k-1 layer nodes connected to a k-layer node, J represents a k-1 layer node connected to a k-layer node,big data feature data of the J-th input representing the I-th point of the input layer,/->The mean and variance of the k-th Gaussian membership function are shown.
As a preferred technical scheme of the invention: the abnormal alarm module divides a first-level alarm threshold value, a second-level alarm threshold value and a third-level alarm threshold value according to the detection value of the detection function of the big data detection module.
As a preferred technical scheme of the invention: the abnormal alarm module also sends out alarms or different levels of alarms according to the magnitude relation between the detection value of the actual detection function in the big data detection module and the primary alarm threshold value, the secondary alarm threshold value and the tertiary alarm threshold value.
The real-time monitoring method for big data comprises the following steps:
s1: real-time monitoring and collecting big data;
s2: preprocessing the collected big data;
s3: extracting features of the preprocessed big data;
s4: building a FNN neural network of a big data operation system and carrying out parameter optimization;
s5: detecting big data characteristic data through the FNN after parameter optimization;
s6: and alarming according to the abnormal big data characteristic data.
Compared with the prior art, the big data real-time monitoring system and method provided by the invention have the beneficial effects that:
according to the invention, the parameters of the FNN neural network are optimized by building the FNN neural network with big data and optimizing the dragonfly algorithm, so that the requirements of global and local simultaneous optimization can be met, the convergence performance of the algorithm can be improved based on the dynamic adjustment of partial weight factors of the dragonfly algorithm, the parameter optimizing efficiency can be improved, a large amount of data can be rapidly processed, the operation speed can be improved, the optimal overhaul time is striven for maintainers, and the damage degree of a big data operation system is reduced.
Drawings
FIG. 1 is a system block diagram of a preferred embodiment of the present invention;
FIG. 2 is a flow chart of a method in a preferred embodiment of the invention.
The meaning of each label in the figure is: 100. the monitoring acquisition processing module; 200. a processing model building module; 300. a big data detection module; 400. and an abnormality alarm module.
Detailed Description
It should be noted that, under the condition of no conflict, the embodiments of the present embodiments and features in the embodiments may be combined with each other, and the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and obviously, the described embodiments are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, a preferred embodiment of the present invention provides a big data real-time monitoring system, comprising:
monitoring acquisition processing module 100: the system is used for monitoring and collecting big data in real time and processing the collected big data;
process model construction module 200: the FNN is used for building a big data operation system and carrying out parameter optimization;
big data detection module 300: the method comprises the steps of detecting big data characteristic data through a FNN neural network after parameter optimization;
abnormality alert module 400: and the alarm is performed according to the big data characteristic data which is detected to be abnormal by the big data detection module 300.
The monitoring and collecting processing module 100 also performs data cleaning and data normalization processing on the collected big data, and performs feature extraction on the processed big data.
The model building module 400 builds a FNN neural network based on the obtained big data characteristic data and optimizes weights and thresholds among an input layer, a hidden layer and an output layer in the FNN neural network based on an optimization dragonfly algorithm.
The optimization dragonfly algorithm is as follows:
X i (t+1)＝X i (t)+ΔX i (t+1)
ΔX i (t+1)＝[aA i (t)+bB i (t)+cC i (t)+dD i (t)+eE i (t)]+αΔX i (t)
wherein X is i (t+1) represents the position of the t+1st iteration dragonfly i, X i (t) represents the position of the t-th iteration dragonfly i, deltaX i (t+1) represents the moving step length of the t+1st iteration dragonfly i, alpha represents the inertial weight coefficient, and DeltaX i (t) represents the moving step length of the t-th iteration dragonfly i, A i (t) represents the separation degree of the t-th iteration dragonfly i, B i (t) represents the alignment speed of the ith iteration dragonfly, C i (t) represents the aggregation degree of the t-th iteration dragonfly i, D i (t) represents the foraging ability of the t-th iteration dragonfly i, E i (t) represents the enemy-avoiding capacity of the t iteration dragonfly i, and a, b, c, d, e is the weight corresponding to the five behaviors respectively;
A i (t)、B i (t)、C i (t)、D i (t)、E i (t) satisfies:
D i (t)＝X + (t)-X i (t)
E i (t)＝X-(t)-X i (t)
wherein X is j (t) represents the positions of adjacent dragonflies j of the t-th iteration dragonfly i, and N represents the total number of dragonflies; v (V) j (t) represents the velocity of adjacent dragonflies j of the t-th iteration dragonfly i; x is X + (t) represents the t th iterationThe location of the food; x is X - (t) represents the location of the natural enemy of the t-th iteration;
when X is i (t) in the absence of nearby individuals around X i (t) updating with random walk behavior:
wherein r is 1 、r 2 Is a random number between (0, 1), f is a constant, σ is a factorial function, and σ (f) =f-! ,
in the optimization dragonfly algorithm, a weight b=c is set, and the weights b and c are dynamically adjusted as follows:
wherein b 'and c' represent weights b and c corresponding to weights before dynamic adjustment, β 1 、β 2 And beta 3 Representing the control coefficient, b max And b min Representing weights b' maximum and minimum, c max And c min The weights c' maximum and minimum are shown, T representing the maximum number of iterations.
In the processing model building module 200, a FNN neural network structure is built for each node in the big data operation system, and the FNN neural network structure of each node is optimized by adopting an optimization dragonfly algorithm.
The big data detection module 300 detects the input big data feature data based on a detection function g (x) of the FNN neural network:
wherein Q is I Represents a set of k-1 layer nodes connected to a k-layer node, J represents a k-1 layer node connected to a k-layer node,big data feature data of the J-th input representing the I-th point of the input layer,/->The mean and variance of the k-th Gaussian membership function are shown.
The abnormality alert module 400 divides the first-level alert threshold, the second-level alert threshold, and the third-level alert threshold according to the detection value of the detection function of the big data detection module 300.
The abnormal alarm module 400 also sends out alarms or different levels of alarms according to the magnitude relation between the detection value of the actual detection function in the big data detection module 300 and the primary alarm threshold value, the secondary alarm threshold value and the tertiary alarm threshold value.
Referring to fig. 2, a real-time big data monitoring method is provided, which includes the following steps:
s1: real-time monitoring and collecting big data;
s2: preprocessing the collected big data;
s3: extracting features of the preprocessed big data;
s4: building a FNN neural network of a big data operation system and carrying out parameter optimization;
s5: detecting big data characteristic data through the FNN after parameter optimization;
s6: and alarming according to the abnormal big data characteristic data.
In this embodiment, the monitoring and collecting processing module 100 monitors the operation data of the big data operation system, collects updated big data in real time, and performs data cleaning and data normalization processing on the big data, so that the processing of the big data by the subsequent module is facilitated. And extracting the preprocessed big data features, uploading the big data features to the processing model building module 200, building the FNN neural network by the processing model building module 200 based on the obtained big data features, wherein the FNN neural network comprises an input layer, a hidden layer and an output layer, the output is controlled according to weights and thresholds among the layers, and errors are approximated to expectations by continuously adjusting the weights and the thresholds. The initialization quality of the weights and thresholds will determine the computational efficiency of the FNN neural network. In addition, due to the influence of gradient descent learning, the optimal weight is very easy to fall into a local minimum value when updated, so that the algorithm cannot be performed, and the calculation accuracy is influenced. It is therefore necessary to optimize the weights and thresholds of the FNN neural network. The dragonfly algorithm has higher global and local searching capacity, can help the FNN neural network algorithm to obtain global optimization on the weight and the threshold value, and can avoid sinking into local minima.
Initializing parameters of the FNN, including weights and thresholds; the behavior weight a, b, c, d, e in the dragonfly algorithm, the total number N of dragonflies, the maximum iteration number t=1000, the inertia weight coefficient alpha,
the weights and the threshold values are orderly ordered to form row vectors which are used as the position X in the dragonfly algorithm i The method comprises the steps of carrying out a first treatment on the surface of the Then randomly initializing the position X according to the weight and the value range of the threshold value, and randomly generating an initialization step delta X i 。
And setting an adaptation function of the dragonflies, and calculating the adaptation value of each dragonfly. And (3) solving the optimal individual position and the worst position of the dragonfly by using the adaptation value of the dragonfly. The current worst position is regarded as a natural enemy (position), and the current optimal individual position is regarded as a food source position of the dragonfly and is also a value of a row vector of the FNN neural network.
Updating using Euclidean distance formulaOptimal solution is food position X + And the current worst solution is the natural enemy position X - The behavior weight a, b, c, d, e and the inertia weight coefficient α are updated simultaneously.
Judging whether adjacent dragonflies exist in the field range: if so, the position vector X is updated by applying the following formula i And step vector DeltaX i : (taking the 5 th iteration as an example):
X i (6)＝X i (5)+ΔX i (6)
ΔX i (6)＝[aA i (5)+bB i (5)+cC i (5)+dD i (5)+eE i (5)]+αΔX i (5)
X i (6) Represents the position of the dragonfly i at the 6 th iteration, X i (5) Represents the position of dragonfly i at the 5 th iteration, deltaX i (6) Represents the moving step length, deltaX, of the dragonfly i at the 6 th iteration i (5) Represents the moving step length of the 5 th iteration dragonfly i, alpha represents the inertia weight coefficient, A i (5) Represents the separation degree of the dragonfly i at the 5 th iteration, B i (5) Represents the alignment speed of the 5 th iteration dragonfly i, C i (5) Represents the gathering degree of the 5 th iteration dragonfly i, D i (5) Representing the foraging capacity of the 5 th iteration dragonfly i, E i (5) Representing the enemy-avoiding capability of the 5 th iteration dragonfly i, wherein a, b, c, d, e is the weight corresponding to the five behaviors respectively;
wherein:
D i (5)＝X + (5)-X i (5)
E i (5)＝X-(5)-X i (5)
wherein X is j (5) Representing the positions of adjacent dragonflies j of the 5 th iteration dragonfly i, and N represents the total number of dragonflies; v (V) j (5) Representing the speed of adjacent dragonflies j of the 5 th iteration dragonfly i; x is X + (5) Representing the position of the food at iteration 5; x is X - (5) Representing the position of the natural enemy of the 5 th iteration;
if X i (5) X in the absence of nearby individuals i (5) The update is performed using the following:
wherein r is 1 、r 2 Is a random number between (0, 1), f is a constant, σ is a factorial function, and σ (f) =f-! ,
however, in the initial operation stage of the dragonfly algorithm, the sample gap between individuals of the population is larger, the local search weight should be increased, the iterative evolution is carried out continuously, the population tropism is increased, and the global search weight should be increased to improve the convergence efficiency, so the weight factors b and c are dynamically adjusted:
setting a weight b=c, and dynamically adjusting the weights b and c as follows:
wherein b 'and c' represent weights b and c corresponding to weights before dynamic adjustment, β 1 、β 2 And beta 3 Representing the control coefficient, b max And b min Representing weights b' maximum and minimum, c max And c min The weights c' maximum and minimum are shown.
The dynamic adjustment of the weight factors is beneficial to improving the convergence performance of the algorithm, optimizing the performance of the dragonfly algorithm, and optimizing the weight and the threshold value of the obtained FNN neural network. And building a FNN neural network for each node of the big data operation system, and optimizing the weight and the threshold of each node based on an optimization dragonfly algorithm.
Judging whether the maximum iteration number and the allowable error range are met, and if so, outputting an optimal weight and an optimal threshold; if not, initializing the parameters again and performing iterative optimization of the parameters.
Substituting the optimal weight and the optimal threshold value obtained by the optimized dragonfly algorithm into the FNN neural network, and detecting the input big data characteristic data by the big data detection module 300 based on a detection function g (x) of the FNN neural network:
wherein Q is I Represents a set of k-1 layer nodes connected to a k-layer node, j represents a k-1 layer node connected to a k-layer node,big data feature data of the J-th input representing the i-th point of the input layer,/>The mean and variance of the k-th Gaussian membership function are shown. The anomaly alarm module 400 divides the first level alarm threshold, the second level alarm threshold and the third level alarm threshold respectively, and starts the first level alarm/the second level alarm when the detection function value reaches the first level alarm threshold/the second level alarm threshold/the third level alarm thresholdThree-level alarming; if the detection function value does not reach the alarm threshold value, no alarm is sent out.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Furthermore, it should be understood that although the present disclosure describes embodiments, not every embodiment is provided with a separate embodiment, and that this description is provided for clarity only, and that the disclosure is not limited to the embodiments described in detail below, and that the embodiments described in the examples may be combined as appropriate to form other embodiments that will be apparent to those skilled in the art.
Claims (8)
1. A big data real-time monitoring system is characterized in that: comprising the following steps:
monitoring acquisition processing module (100): the system is used for monitoring and collecting big data in real time and processing the collected big data;
process model building module (200): the FNN is used for building a big data operation system and carrying out parameter optimization;
big data detection module (300): the method comprises the steps of detecting big data characteristic data through a FNN neural network after parameter optimization;
an abnormality alert module (400): the system is used for alarming according to the big data characteristic data of the abnormal occurrence detected by the big data detection module (300);
the model building module (400) builds a FNN neural network based on the acquired big data characteristic data and optimizes weights and thresholds among an input layer, a hidden layer and an output layer in the FNN neural network based on an optimization dragonfly algorithm;
the optimized dragonfly algorithm is as follows:
X i (t+1)＝X i (t)+ΔX i (t+1)
ΔX i (t+1)＝[aA i (t)+bB i (t)+cC i (t)+dD i (t)+eE i (t)]+αΔX i (t)
wherein X is i (t+1) represents the position of the t+1st iteration dragonfly i, X i (t) represents the position of the t-th iteration dragonfly i, ΔX i (t+1) represents the moving step length of the t+1st iteration dragonfly i, alpha represents the inertia weight coefficient, and DeltaX i (t) represents the moving step length of the t-th iteration dragonfly i, A i (t) represents the separation degree of the t-th iteration dragonfly i, B i (t) represents the alignment speed of the t-th iteration dragonfly i, ci (t) represents the gathering degree of the t-th iteration dragonfly i, and D i (t) represents the foraging capacity of the t-th iteration dragonfly i, ei (t) represents the enemy-avoiding capacity of the t-th iteration dragonfly i, and a, b, c, d, e are weights corresponding to five behaviors respectively;
A i (t)、B i (t)、C i (t)、D i (t)、E i (t) satisfies:
D i (t)＝X + (t)-X i (t)
E i (t)＝X - (t)-X i (t)
wherein X is j (t) represents the position of the adjacent dragonfly j of the t-th iteration dragonfly i, N tableShowing the total number of dragonflies; v (V) j (t) represents the velocity of adjacent dragonfly j of the t-th iteration dragonfly i; x is X + (t) represents the position of the food item at the t-th iteration; x is X - (t) represents the location of the natural enemy of the t-th iteration;
when X is i (t) in the absence of nearby individuals around X i (t) updating with random walk behavior:
wherein r is 1 、r 2 Is a random number between (0, 1), f is a constant, σ is a factorial function, and σ (f) =f-! ,
2. the big data real-time monitoring system of claim 1, wherein: the monitoring acquisition processing module (100) also carries out data cleaning and data normalization processing on the acquired big data, and carries out feature extraction on the processed big data.
3. The big data real-time monitoring system of claim 1, wherein: in the optimized dragonfly algorithm, a weight b=c is set, and the weights b and c are dynamically adjusted as follows:
wherein b 'and c' represent weights b and c corresponding to weights before dynamic adjustment, β 1 、β 2 And beta 3 Representing the control coefficient, b max And b mmin Representing weights b' maximum and minimum, c max And c min The weights c' maximum and minimum are shown, T representing the maximum number of iterations.
4. A big data real time monitoring system according to claim 3, characterized in that: in the processing model building module (200), a FNN neural network structure is built for each node in the big data operation system, and the FNN neural network structure of each node is optimized by adopting an optimization dragonfly algorithm.
5. The big data real-time monitoring system of claim 1, wherein: the big data detection module (300) detects the input big data characteristic data based on a detection function g (x) of the FNN neural network:
wherein Q is I Represents a set of k-1 layer nodes connected to a k-layer node, J represents a k-1 layer node connected to a k-layer node,big data feature data of the J-th input representing the I-th point of the input layer,/->The mean and variance of the k-th Gaussian membership function are shown.
6. The big data real-time monitoring system of claim 5, wherein: the abnormality warning module (400) divides a first-level warning threshold, a second-level warning threshold and a third-level warning threshold according to the detection value of the detection function of the big data detection module (300).
7. The big data real-time monitoring system of claim 6, wherein: the abnormal alarm module (400) also sends out alarms or alarms of different levels according to the magnitude relation between the detection value of the actual detection function in the big data detection module (300) and the primary alarm threshold value, the secondary alarm threshold value and the tertiary alarm threshold value.
8. A big data real-time monitoring method, based on the big data real-time monitoring system of any one of claims 1-7, characterized in that: the method comprises the following steps:
s1: real-time monitoring and collecting big data;
s2: preprocessing the collected big data;
s3: extracting features of the preprocessed big data;
s4: building a FNN neural network of a big data operation system and carrying out parameter optimization;
s5: detecting big data characteristic data through the FNN after parameter optimization;
s6: and alarming according to the abnormal big data characteristic data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310642151.6A CN116628425A (en) | 2023-06-01 | 2023-06-01 | Big data real-time monitoring system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310642151.6A CN116628425A (en) | 2023-06-01 | 2023-06-01 | Big data real-time monitoring system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116628425A true CN116628425A (en) | 2023-08-22 |
Family
ID=87641571
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310642151.6A Pending CN116628425A (en) | 2023-06-01 | 2023-06-01 | Big data real-time monitoring system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116628425A (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202836A (en) * | 2016-08-24 | 2016-12-07 | 江苏大学 | A kind of Optimization Design of piecemeal rotor switched reluctance motor |
| CN107490760A (en) * | 2017-08-22 | 2017-12-19 | 西安工程大学 | The circuit breaker failure diagnostic method of fuzzy neural network is improved based on genetic algorithm |
| CN109507887A (en) * | 2018-12-28 | 2019-03-22 | 浙江大学 | The industrial melt index soft measurement method of fuzzy neural network combination colony intelligence optimizing |
| CN110633791A (en) * | 2019-09-26 | 2019-12-31 | 北航（四川）西部国际创新港科技有限公司 | Convolutional neural network-based unmanned aerial vehicle abnormal behavior identification method |
| CN110929464A (en) * | 2019-11-20 | 2020-03-27 | 燕山大学 | Storage battery parameter identification method based on improved dragonfly algorithm |
| CN112215278A (en) * | 2020-10-09 | 2021-01-12 | 吉林大学 | Multi-dimensional data feature selection method combining genetic algorithm and dragonfly algorithm |
| CN113239638A (en) * | 2021-06-29 | 2021-08-10 | 百维金科(上海)信息科技有限公司 | Overdue risk prediction method for optimizing multi-core support vector machine based on dragonfly algorithm |
| CN114386322A (en) * | 2021-12-24 | 2022-04-22 | 国网辽宁省电力有限公司电力科学研究院 | Dragonfly algorithm-based reactive power optimization method for power distribution network with distributed power supply |
| CN114504330A (en) * | 2022-01-30 | 2022-05-17 | 天津大学 | Fatigue state monitoring system based on portable electroencephalogram acquisition head ring |
| CN114580763A (en) * | 2022-03-10 | 2022-06-03 | 南京工业大学 | Power load prediction method based on improved dragonfly and lightweight gradient lifting tree model |
-
2023
- 2023-06-01 CN CN202310642151.6A patent/CN116628425A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202836A (en) * | 2016-08-24 | 2016-12-07 | 江苏大学 | A kind of Optimization Design of piecemeal rotor switched reluctance motor |
| CN107490760A (en) * | 2017-08-22 | 2017-12-19 | 西安工程大学 | The circuit breaker failure diagnostic method of fuzzy neural network is improved based on genetic algorithm |
| CN109507887A (en) * | 2018-12-28 | 2019-03-22 | 浙江大学 | The industrial melt index soft measurement method of fuzzy neural network combination colony intelligence optimizing |
| CN110633791A (en) * | 2019-09-26 | 2019-12-31 | 北航（四川）西部国际创新港科技有限公司 | Convolutional neural network-based unmanned aerial vehicle abnormal behavior identification method |
| CN110929464A (en) * | 2019-11-20 | 2020-03-27 | 燕山大学 | Storage battery parameter identification method based on improved dragonfly algorithm |
| CN112215278A (en) * | 2020-10-09 | 2021-01-12 | 吉林大学 | Multi-dimensional data feature selection method combining genetic algorithm and dragonfly algorithm |
| CN113239638A (en) * | 2021-06-29 | 2021-08-10 | 百维金科(上海)信息科技有限公司 | Overdue risk prediction method for optimizing multi-core support vector machine based on dragonfly algorithm |
| CN114386322A (en) * | 2021-12-24 | 2022-04-22 | 国网辽宁省电力有限公司电力科学研究院 | Dragonfly algorithm-based reactive power optimization method for power distribution network with distributed power supply |
| CN114504330A (en) * | 2022-01-30 | 2022-05-17 | 天津大学 | Fatigue state monitoring system based on portable electroencephalogram acquisition head ring |
| CN114580763A (en) * | 2022-03-10 | 2022-06-03 | 南京工业大学 | Power load prediction method based on improved dragonfly and lightweight gradient lifting tree model |
Non-Patent Citations (2)
| Title |
|---|
| 吴伟民;吴汪洋;林志毅;李泽熊;方典禹;: "基于增强个体信息交流的蜻蜓算法", 计算机工程与应用, no. 04 * |
| 胡小平 等: "基于蜻蜓算法的改进研究", 《计算机科学与应用》, pages 1377 - 1386 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022121289A1 (en) | Methods and systems for mining minority-class data samples for training neural network | |
| CN111211994B (en) | Network traffic classification method based on SOM and K-means fusion algorithm | |
| CN113256066B (en) | PCA-XGboost-IRF-based job shop real-time scheduling method | |
| Prajwala | A comparative study on decision tree and random forest using R tool | |
| CN110929848B (en) | Training and tracking method based on multi-challenge perception learning model | |
| CN110708318A (en) | Network abnormal flow prediction method based on improved radial basis function neural network algorithm | |
| CN110826617A (en) | Situation element classification method and training method and device of model thereof, and server | |
| CN115514581B (en) | Data analysis method and equipment for industrial internet data security platform | |
| CN116227786A (en) | Unmanned aerial vehicle comprehensive efficiency evaluation system | |
| CN116545651A (en) | Network intrusion detection method based on machine learning | |
| CN116993548A (en) | Incremental learning-based education training institution credit assessment method and system for LightGBM-SVM | |
| CN113609480B (en) | Multipath learning intrusion detection method based on large-scale network flow | |
| Thirimanne et al. | Comparative algorithm analysis for machine learning based intrusion detection system | |
| US20170293608A1 (en) | Unusual score generators for a neuro-linguistic behavioral recognition system | |
| CN114416423A (en) | Root cause positioning method and system based on machine learning | |
| CN117574213A (en) | APSO-CNN-based network traffic classification method | |
| CN111767273B (en) | Data intelligent detection method and device based on improved SOM algorithm | |
| CN117117850A (en) | Short-term electricity load prediction method and system | |
| CN116628425A (en) | Big data real-time monitoring system and method | |
| CN115114951A (en) | Genetic algorithm and clustering algorithm composite nested abnormal sound detection algorithm | |
| CN116841718A (en) | Physical resource scheduling method and scheduler based on Kubernetes | |
| CN112069876A (en) | Handwriting recognition method based on adaptive differential gradient optimization | |
| CN117333726B (en) | Quartz crystal cutting abnormality monitoring method, system and device based on deep learning | |
| Sathyaraj et al. | A hybrid approach to improve the quality of software fault prediction using Naïve Bayes and k-NN classification algorithm with ensemble method | |
| CN118133146B (en) | Artificial intelligence-based risk intrusion recognition method for Internet of things |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20230822 |
|
| RJ01 | Rejection of invention patent application after publication |