CN110781494A - Data abnormity early warning method, device, equipment and storage medium - Google Patents
Data abnormity early warning method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN110781494A CN110781494A CN201911010978.5A CN201911010978A CN110781494A CN 110781494 A CN110781494 A CN 110781494A CN 201911010978 A CN201911010978 A CN 201911010978A CN 110781494 A CN110781494 A CN 110781494A
- Authority
- CN
- China
- Prior art keywords
- data
- abnormal
- early warning
- standardized
- verification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/52—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow
- G06F21/53—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow by executing in a restricted environment, e.g. sandbox or secure virtual machine
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Testing And Monitoring For Control Systems (AREA)
Abstract
The invention relates to the technical field of data processing, and discloses a data abnormity early warning method, a device, equipment and a storage medium, wherein the method comprises the following steps: the method comprises the steps of obtaining verification data in a preset time period, carrying out fuzzy clustering analysis on the verification data, generating a dynamic clustering chart according to fuzzy clustering analysis results, selecting abnormal data from the dynamic clustering chart, matching abnormal data grades corresponding to the abnormal data from a preset database, and outputting the abnormal data and the abnormal data grades to carry out early warning, so that the abnormal data is obtained by carrying out fuzzy clustering analysis on the verification data, then calculating abnormal data grades, and carrying out early warning according to the abnormal data and the abnormal data grades, thereby solving the technical problem of how to carry out early warning when the verification data is abnormal.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for early warning of data exception.
Background
The core of the existing behavior verification system is to create a unique behavior interaction sandbox, and black and white samples are marked by collecting behavior data in the sandbox based on multiple dimensions and methods, so that data characteristics are constructed, different types of man-machine distinguishing models are established, corresponding behavior verification abnormal data are always generated in the behavior verification process, and the existing behavior verification has no effective early warning mode for dealing with the abnormal data and only simply eliminates the abnormal data.
Therefore, the technical problem of how to perform early warning when the verification data is abnormal exists essentially.
The above is only for the purpose of assisting understanding of the technical aspects of the present invention, and does not represent an admission that the above is prior art.
Disclosure of Invention
The invention mainly aims to provide a data abnormity early warning method, a device, equipment and a storage medium, and aims to solve the technical problem of early warning when data are verified to be abnormal.
In order to achieve the above object, the present invention provides a data anomaly early warning method, which comprises the following steps:
acquiring verification data in a preset time period;
carrying out fuzzy clustering analysis on the verification data, and generating a dynamic clustering chart according to the result of the fuzzy clustering analysis;
selecting anomalous data from the dynamic cluster map;
matching abnormal data grades corresponding to the abnormal data from a preset database;
and outputting the abnormal data and the abnormal data grade to perform early warning.
Preferably, the performing fuzzy clustering analysis on the verification data and generating a dynamic clustering chart according to a result of the fuzzy clustering analysis specifically includes:
standardizing the verification data to obtain standardized data;
and constructing a fuzzy similar matrix according to the standardized data, and generating a dynamic cluster map according to the fuzzy similar matrix.
Preferably, the normalizing the verification data to obtain normalized data specifically includes:
carrying out translation standard deviation transformation on the verification data to obtain standard deviation data;
and taking the standard deviation data within a preset standardized threshold value range as standardized data.
Preferably, after the step of taking the standard deviation data within the preset standardized threshold as the standardized data, the method further comprises the following steps:
taking standard deviation data which is not within the range of a preset standardized threshold value as data to be transformed;
carrying out translation range transformation on the data to be transformed to obtain range data;
and taking the range of the range difference data within the preset standardized threshold value as standardized data.
Preferably, the constructing a fuzzy similar matrix according to the normalized data, and generating a dynamic clustering chart according to the fuzzy similar matrix specifically include:
calculating a similarity coefficient of the standardized data by adopting an Euclidean distance method;
constructing a fuzzy similarity matrix according to the standardized data and the similarity coefficient;
and generating a dynamic cluster map according to the fuzzy similar matrix.
Preferably, the selecting abnormal data from the dynamic cluster map specifically includes:
acquiring data information and data distribution information from the dynamic cluster map;
and determining abnormal data according to the data information and the data distribution information.
Preferably, the matching of the abnormal data grade corresponding to the abnormal data from the preset database specifically includes:
acquiring an abnormal data type corresponding to the abnormal data and a data normal value of the abnormal data type;
calculating a data abnormal value of the abnormal data according to the abnormal data and the data normal value;
and matching abnormal data grades corresponding to the data abnormal values from a preset database.
In addition, in order to achieve the above object, the present invention further provides a data abnormality warning device, including:
the data acquisition module is used for acquiring verification data in a preset time period;
the fuzzy clustering module is used for carrying out fuzzy clustering analysis on the verification data and generating a dynamic clustering chart according to the result of the fuzzy clustering analysis;
the data selection module is used for selecting abnormal data from the dynamic cluster map;
the grade matching module is used for matching the abnormal data grade corresponding to the abnormal data from a preset database;
and the output early warning module is used for outputting the abnormal data and the abnormal data grade to carry out early warning.
In addition, in order to achieve the above object, the present invention further provides a data anomaly early warning device, including: the data anomaly early warning method comprises a memory, a processor and a data anomaly early warning program which is stored on the memory and can run on the processor, wherein the data anomaly early warning program is configured with steps for realizing the data anomaly early warning method.
In addition, in order to achieve the above object, the present invention further provides a storage medium, where a data abnormality warning program is stored, and when the data abnormality warning program is executed by a processor, the data abnormality warning program implements the steps of the data abnormality warning method as described above.
According to the data anomaly early warning method, the verification data in the preset time period are obtained, fuzzy clustering analysis is conducted on the verification data, the dynamic clustering chart is generated according to the fuzzy clustering analysis result, abnormal data are selected from the dynamic clustering chart, the abnormal data grades corresponding to the abnormal data are matched from the preset database, the abnormal data and the abnormal data grades are output to conduct early warning, therefore, the abnormal data are obtained through the fuzzy clustering analysis on the verification data, the abnormal data grades are calculated, early warning is conducted according to the abnormal data and the abnormal data grades, and the technical problem of how to conduct early warning when the verification data are abnormal is solved.
Drawings
Fig. 1 is a schematic structural diagram of a data anomaly early warning device in a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a data anomaly warning method according to a first embodiment of the present invention;
FIG. 3 is an illustration of a first embodiment of a data anomaly warning method according to the present invention;
FIG. 4 is a flowchart illustrating a data anomaly warning method according to a second embodiment of the present invention;
FIG. 5 is a flowchart illustrating a data anomaly warning method according to a third embodiment of the present invention;
fig. 6 is a functional block diagram of the data anomaly warning device according to the first embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a data anomaly early warning device in a hardware operating environment according to an embodiment of the present invention.
As shown in fig. 1, the data abnormality warning apparatus may include: a processor 1001, such as a Central Processing Unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may comprise a Display screen (Display), an input unit such as keys, and the optional user interface 1003 may also comprise a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The Memory 1005 may be a Random Access Memory (RAM) Memory or a non-volatile Memory (e.g., a magnetic disk Memory). The memory 1005 may alternatively be a storage device separate from the processor 1001.
Those skilled in the art will appreciate that the configuration of the apparatus shown in fig. 1 does not constitute a limitation of the data anomaly warning apparatus, and may include more or fewer components than those shown, or some components in combination, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a storage medium, may include therein an operating system, a network communication module, a user interface module, and a data abnormality warning program.
In the data anomaly early warning device shown in fig. 1, the network interface 1004 is mainly used for connecting an external network and performing data communication with other network devices; the user interface 1003 is mainly used for connecting to a user equipment and performing data communication with the user equipment; the apparatus of the present invention calls a data abnormality warning program stored in the memory 1005 through the processor 1001, and performs the following operations:
acquiring verification data in a preset time period;
carrying out fuzzy clustering analysis on the verification data, and generating a dynamic clustering chart according to the result of the fuzzy clustering analysis;
selecting anomalous data from the dynamic cluster map;
matching abnormal data grades corresponding to the abnormal data from a preset database;
and outputting the abnormal data and the abnormal data grade to perform early warning.
Further, the processor 1001 may call the data anomaly early warning program stored in the memory 1005, and further perform the following operations:
standardizing the verification data to obtain standardized data;
and constructing a fuzzy similar matrix according to the standardized data, and generating a dynamic cluster map according to the fuzzy similar matrix.
Further, the processor 1001 may call the data anomaly early warning program stored in the memory 1005, and further perform the following operations:
carrying out translation standard deviation transformation on the verification data to obtain standard deviation data;
and taking the standard deviation data within a preset standardized threshold value range as standardized data.
Further, the processor 1001 may call the data anomaly early warning program stored in the memory 1005, and further perform the following operations:
taking standard deviation data which is not within the range of a preset standardized threshold value as data to be transformed;
carrying out translation range transformation on the data to be transformed to obtain range data;
and taking the range of the range difference data within the preset standardized threshold value as standardized data.
Further, the processor 1001 may call the data anomaly early warning program stored in the memory 1005, and further perform the following operations:
calculating a similarity coefficient of the standardized data by adopting an Euclidean distance method;
constructing a fuzzy similarity matrix according to the standardized data and the similarity coefficient;
and generating a dynamic cluster map according to the fuzzy similar matrix.
Further, the processor 1001 may call the data anomaly early warning program stored in the memory 1005, and further perform the following operations:
acquiring data information and data distribution information from the dynamic cluster map;
and determining abnormal data according to the data information and the data distribution information.
Further, the processor 1001 may call the data anomaly early warning program stored in the memory 1005, and further perform the following operations:
acquiring an abnormal data type corresponding to the abnormal data and a data normal value of the abnormal data type;
calculating a data abnormal value of the abnormal data according to the abnormal data and the data normal value;
and matching abnormal data grades corresponding to the data abnormal values from a preset database.
In the embodiment, the technical problem of how to perform early warning when the verification data is abnormal is solved by acquiring the verification data in a preset time period, performing fuzzy clustering analysis on the verification data, generating a dynamic clustering chart according to a fuzzy clustering analysis result, selecting abnormal data from the dynamic clustering chart, matching an abnormal data grade corresponding to the abnormal data from a preset database, and outputting the abnormal data and the abnormal data grade to perform early warning.
Based on the hardware structure, the embodiment of the data anomaly early warning method is provided.
Referring to fig. 2, fig. 2 is a flowchart illustrating a data anomaly early-warning method according to a first embodiment of the present invention.
In a first embodiment, the data anomaly early warning method includes the following steps:
in step S10, verification data within a preset time period is acquired.
It should be noted that the execution subject of the present embodiment may be a data anomaly early warning device, and may also be other devices that can implement the same or similar functions. The obtaining of the verification data in the preset time period may be performed from the background server, and may also be performed in other manners.
It should be understood that the preset time period may be 1 hour, or may be other time values, which is not limited in this embodiment, taking the preset time period as 1 hour as an example for description, the obtaining of the verification data in the preset time period is to obtain all the verification data in 1 hour.
It can be understood that the data abnormality early warning device may acquire the verification data once every 1 hour, acquire all the verification data within the last 1 hour, detect the acquired verification data, and determine whether there is abnormal data therein.
And step S20, performing fuzzy clustering analysis on the verification data, and generating a dynamic clustering chart according to the result of the fuzzy clustering analysis.
It should be noted that, the obtained verification data is standardized to obtain standardized data, the standardized data is between [0 and 1], then a fuzzy similar matrix is constructed according to the standardized data, and a dynamic clustering map is generated according to the fuzzy similar matrix, and the dynamic clustering map includes data information and data distribution information.
Step S30, selecting abnormal data from the dynamic cluster map.
It should be understood that data information and data distribution information are obtained from the dynamic cluster map, and abnormal data is determined according to the data information and the data distribution information.
It can be understood that the data classification and the data classification condition in the dynamic cluster map can be determined according to the distribution information, whether the data information is normal or abnormal can be known according to the data classification condition, and it should be understood that the data information which is obviously deviated from other data classifications in the dynamic cluster map is abnormal and is judged as abnormal data.
In a specific implementation, for example, there are 7 authentication data such as A, B, C, D, E, F, G, performing fuzzy clustering analysis on the 7 verification data, and generating a dynamic clustering chart according to the fuzzy clustering analysis result, as shown in FIG. 3, wherein 2 data are classified according to the dynamic clustering chart, wherein A, B, C is classified as a region and is more closely located at 1 in the figure, and E, F, G is classified as a region and is more closely located at 2 in the figure, and D is neither in the 1-data class nor the 2-data class, and is located further away from other data, it can be determined as abnormal data, it should be noted that the number of data classification classes can be set according to the actual situation, for example, 2 data classifications may be set, 4 data classifications may also be set, or n data classifications may also be set, which is not limited in this embodiment.
And step S40, matching the abnormal data grade corresponding to the abnormal data from a preset database.
It should be noted that each abnormal data corresponds to a data abnormal value, and the abnormal data level is set in advance according to the data abnormal value, for example, the abnormal data level of the abnormal data with the data abnormal value of 0 to 1 is set as a 1-level data abnormal, the abnormal data level of the abnormal data with the data abnormal value of 1 to 2 is set as a 2-level data abnormal, the abnormal data level of the abnormal data with the data abnormal value of 2 to 3 is set as a 3-level data abnormal, the abnormal data level can be set according to specific situations, the range of the data abnormal value and the abnormal data level can be other values, which is not limited in this embodiment, and in this embodiment, three abnormal data levels of the 1-level data abnormal, the 2-level data abnormal and the 3-level data abnormal are taken as examples for explanation.
And step S50, outputting the abnormal data and the abnormal data grade to perform early warning.
It should be noted that when the verification data in the verification system is abnormal, the user is notified in the form of mail or telephone or other forms, so as to help the user to find out that the verification data in the verification system is abnormal in time and to know the abnormal data level, so as to reduce the security risk and make the user use the verification system better.
It should be understood that the switch of the early warning notification function may be turned on and off according to the setting of the user, the early warning notification function is set through the early warning notification function, after the switch is turned on, the early warning notification function becomes effective, if the switch is turned off or set to 0, the early warning notification is not turned on, the user does not receive the early warning notification, and the switch of the early warning notification function is set by the user according to the actual situation, which is not limited in this embodiment.
In the embodiment, the technical problem of how to perform early warning when the verification data is abnormal is solved by acquiring the verification data in a preset time period, performing fuzzy clustering analysis on the verification data, generating a dynamic clustering chart according to a fuzzy clustering analysis result, selecting abnormal data from the dynamic clustering chart, matching an abnormal data grade corresponding to the abnormal data from a preset database, and outputting the abnormal data and the abnormal data grade to perform early warning.
In an embodiment, as shown in fig. 4, a second embodiment of the data anomaly warning method according to the present invention is proposed based on the first embodiment, where the step S20 includes:
step S201, standardizing the verification data to obtain standardized data.
It should be noted that, the verification data is subjected to translation standard deviation transformation to obtain standard deviation data, the standard deviation data within a preset standardized threshold range is used as standardized data, the standard deviation data not within the preset standardized threshold range is used as data to be transformed, the data to be transformed is subjected to translation range transformation to obtain range data, and the range data within the preset standardized threshold range is used as standardized data.
It should be understood that the preset normalized threshold may be set to [0,1], or may be set to other ranges, which is not limited in this embodiment, and in this embodiment, the preset normalized threshold is set to [0,1] as an example for explanation.
It should be noted that the domain of the verification data may be expressed as X ═ { X ═ X
1，x
2，…,x
n}; each verification datum represents the behavior of the verification datum by m indexes, and the expression of the verification datum is as follows:
x
i＝{x
i1,x
i2,…,x
im},i＝1,2,…n；
the original matrix can be obtained according to the above formula:
A＝(x
ij)
m×n；
in the formula, x
jSample data representing verification data.
The calculation formula for performing translation standard deviation transformation on the verification data is as follows:
in the formula, x
ijThe data representing the verification of the data is,
representing mean of verification data samples, s
jAnd (3) representing the standard deviation of the sample of the verification data, and the calculation formulas are respectively as follows:
after the calculation of the steps, standard deviation data is obtained, the standard deviation data within the range of [0,1] is used as standardized data, the standard deviation data not within the range of [0,1] is used as data to be transformed, the data to be transformed is subjected to translational range transformation, and the translational range transformation calculation formula is as follows:
max { x 'in formula'
ijDenotes the maximum value of the verification data samples, min { x'
ijDenotes the minimum value of the validation data samples.
After the translation standard deviation transformation and the translation range transformation, the final standardized data are both in the range of [0,1 ].
And S202, constructing a fuzzy similar matrix according to the standardized data, and generating a dynamic clustering chart according to the fuzzy similar matrix.
It should be noted that, the formula for calculating the similarity coefficient of the normalized data by the euclidean distance method is as follows:
r
ij＝1-c·d(x
i,x
j)；
wherein c represents a constant, 0. ltoreq. r
ij≤1，d(x
i,x
j) Denotes x
iAnd x
jThe euclidean distance between them.
And constructing a fuzzy similarity matrix according to the standardized data and the similarity coefficient, thereby obtaining different classification results and forming a dynamic clustering chart.
In this embodiment, the normalized data is obtained by normalizing the verification data, a fuzzy similar matrix is constructed according to the normalized data, and a dynamic cluster map is generated according to the fuzzy similar matrix, so that the dynamic cluster map is generated by processing the verification data.
In an embodiment, as shown in fig. 5, a third embodiment of the data anomaly early-warning method according to the present invention is provided based on the first embodiment or the second embodiment, in this embodiment, the description is made based on the first embodiment, and the step S40 includes:
step S401, obtaining an abnormal data type corresponding to the abnormal data and a data normal value of the abnormal data type.
It should be noted that after the abnormal data are screened out through the foregoing steps, the abnormal data type corresponding to the abnormal data and the data normal value of the abnormal data type are obtained, it can be understood that each data type in the verification system has its corresponding data normal value, and the data normal value may be an average value of all data samples in the data type.
In a specific implementation, for example, the abnormal data screened in the foregoing steps is D, the abnormal data type of the obtained D is data type X, and an average value of all data samples of the data type X is obtained and is used as a data normal value.
Step S402, calculating a data abnormal value of the abnormal data according to the abnormal data and the data normal value.
It should be noted that after obtaining the data normal value of the abnormal data type corresponding to the abnormal data, a data abnormal value of the abnormal data may be calculated according to the abnormal data and the data normal value, where the data abnormal value is used to assess the abnormal data grade of the abnormal data, and the data abnormal value may be calculated by calculating a percentage of the abnormal data and the data normal value, and taking the calculation result as the data abnormal value, or calculating the data abnormal value in another manner, which is not limited in this embodiment.
And step S403, matching abnormal data grades corresponding to the data abnormal values from a preset database.
It should be noted that, an abnormal data grade is set in advance according to the data abnormal value, the abnormal degree of the abnormal data is distinguished according to the data grade, the sample data abnormal value and the abnormal data grade corresponding to each sample data abnormal value are stored in the preset database, after the data abnormal value of the abnormal data is calculated through the above steps, the calculated data abnormal value can be matched with the sample data abnormal value in the preset database, and the abnormal data grade corresponding to the sample data abnormal value which is successfully matched is used as the abnormal data grade of the abnormal data.
It should be understood that when abnormal data occurs in the verification data, a data abnormal value is calculated according to the steps, the abnormal data grade of the abnormal data is judged according to the data abnormal value, the abnormal data and the abnormal data grade are output, a user is prompted to give an early warning in the form of a mail and the like, and after the user receives the abnormal data and the abnormal data grade, the influence degree of the abnormal data can be known.
In this embodiment, an abnormal data type corresponding to the abnormal data and a data normal value of the abnormal data type are obtained, a data abnormal value of the abnormal data is calculated according to the abnormal data and the data normal value, an abnormal data grade corresponding to the data abnormal value is matched from a preset database, so that the abnormal data grade of the abnormal data is calculated, and the abnormal degree of the abnormal data is distinguished according to the abnormal data grade.
In addition, an embodiment of the present invention further provides a storage medium, where a data anomaly early warning program is stored on the storage medium, and when executed by a processor, the data anomaly early warning program implements the following operations:
acquiring verification data in a preset time period;
carrying out fuzzy clustering analysis on the verification data, and generating a dynamic clustering chart according to the result of the fuzzy clustering analysis;
selecting anomalous data from the dynamic cluster map;
matching abnormal data grades corresponding to the abnormal data from a preset database;
and outputting the abnormal data and the abnormal data grade to perform early warning.
Further, the data anomaly early warning program when executed by the processor further realizes the following operations:
standardizing the verification data to obtain standardized data;
and constructing a fuzzy similar matrix according to the standardized data, and generating a dynamic cluster map according to the fuzzy similar matrix.
Further, the data anomaly early warning program when executed by the processor further realizes the following operations:
carrying out translation standard deviation transformation on the verification data to obtain standard deviation data;
and taking the standard deviation data within a preset standardized threshold value range as standardized data.
Further, the data anomaly early warning program when executed by the processor further realizes the following operations:
taking standard deviation data which is not within the range of a preset standardized threshold value as data to be transformed;
carrying out translation range transformation on the data to be transformed to obtain range data;
and taking the range of the range difference data within the preset standardized threshold value as standardized data.
Further, the data anomaly early warning program when executed by the processor further realizes the following operations:
calculating a similarity coefficient of the standardized data by adopting an Euclidean distance method;
constructing a fuzzy similarity matrix according to the standardized data and the similarity coefficient;
and generating a dynamic cluster map according to the fuzzy similar matrix.
Further, the data anomaly early warning program when executed by the processor further realizes the following operations:
acquiring data information and data distribution information from the dynamic cluster map;
and determining abnormal data according to the data information and the data distribution information.
Further, the data anomaly early warning program when executed by the processor further realizes the following operations:
acquiring an abnormal data type corresponding to the abnormal data and a data normal value of the abnormal data type;
calculating a data abnormal value of the abnormal data according to the abnormal data and the data normal value;
and matching abnormal data grades corresponding to the data abnormal values from a preset database.
In the embodiment, the technical problem of how to perform early warning when the verification data is abnormal is solved by acquiring the verification data in a preset time period, performing fuzzy clustering analysis on the verification data, generating a dynamic clustering chart according to a fuzzy clustering analysis result, selecting abnormal data from the dynamic clustering chart, matching an abnormal data grade corresponding to the abnormal data from a preset database, and outputting the abnormal data and the abnormal data grade to perform early warning.
In addition, referring to fig. 6, an embodiment of the present invention further provides a data anomaly early-warning device, where the data anomaly early-warning device includes:
the data obtaining module 10 is configured to obtain verification data within a preset time period.
It should be understood that the preset time period may be 1 hour, or may be other time values, which is not limited in this embodiment, taking the preset time period as 1 hour as an example for description, the obtaining of the verification data in the preset time period is to obtain all the verification data in 1 hour.
It can be understood that the data abnormality early warning device may acquire the verification data once every 1 hour, acquire all the verification data within the last 1 hour, detect the acquired verification data, and determine whether there is abnormal data therein.
And the fuzzy clustering module 20 is configured to perform fuzzy clustering analysis on the verification data, and generate a dynamic clustering chart according to a result of the fuzzy clustering analysis.
It should be noted that, the obtained verification data is standardized to obtain standardized data, the standardized data is between [0 and 1], then a fuzzy similar matrix is constructed according to the standardized data, and a dynamic clustering map is generated according to the fuzzy similar matrix, and the dynamic clustering map includes data information and data distribution information.
And the data selecting module 30 is used for selecting abnormal data from the dynamic cluster map.
It can be understood that the data classification and the data classification condition in the dynamic cluster map can be determined according to the distribution information, whether the data information is normal or abnormal can be known according to the data classification condition, and it should be understood that the data information which is obviously deviated from other data classifications in the dynamic cluster map is abnormal and is judged as abnormal data.
In a specific implementation, for example, there are 7 authentication data such as A, B, C, D, E, F, G, performing fuzzy clustering analysis on the 7 verification data, and generating a dynamic clustering chart according to the fuzzy clustering analysis result, as shown in FIG. 3, wherein 2 data are classified according to the dynamic clustering chart, wherein A, B, C is classified as a region and is more closely located at 1 in the figure, and E, F, G is classified as a region and is more closely located at 2 in the figure, and D is neither in the 1-data class nor the 2-data class, and is located further away from other data, it can be determined as abnormal data, it should be noted that the number of data classification classes can be set according to the actual situation, for example, 2 data classifications may be set, 4 data classifications may also be set, or n data classifications may also be set, which is not limited in this embodiment.
And the grade matching module 40 is used for matching the abnormal data grade corresponding to the abnormal data from a preset database.
It should be noted that each abnormal data corresponds to a data abnormal value, and the abnormal data level is set in advance according to the data abnormal value, for example, the abnormal data level of the abnormal data with the data abnormal value of 0 to 1 is set as a 1-level data abnormal, the abnormal data level of the abnormal data with the data abnormal value of 1 to 2 is set as a 2-level data abnormal, the abnormal data level of the abnormal data with the data abnormal value of 2 to 3 is set as a 3-level data abnormal, the abnormal data level can be set according to specific situations, the range of the data abnormal value and the abnormal data level can be other values, which is not limited in this embodiment, and in this embodiment, three abnormal data levels of the 1-level data abnormal, the 2-level data abnormal and the 3-level data abnormal are taken as examples for explanation.
And the output early warning module 50 is used for outputting the abnormal data and the abnormal data grade to carry out early warning.
It should be noted that when the verification data in the verification system is abnormal, the user is notified in the form of mail or telephone or other forms, so as to help the user to find out that the verification data in the verification system is abnormal in time and to know the abnormal data level, so as to reduce the security risk and make the user use the verification system better.
It should be understood that the switch of the early warning notification function may be turned on and off according to the setting of the user, the early warning notification function is set through the early warning notification function, after the switch is turned on, the early warning notification function becomes effective, if the switch is turned off or set to 0, the early warning notification is not turned on, the user does not receive the early warning notification, and the switch of the early warning notification function is set by the user according to the actual situation, which is not limited in this embodiment.
In the embodiment, the technical problem of how to perform early warning when the verification data is abnormal is solved by acquiring the verification data in a preset time period, performing fuzzy clustering analysis on the verification data, generating a dynamic clustering chart according to a fuzzy clustering analysis result, selecting abnormal data from the dynamic clustering chart, matching an abnormal data grade corresponding to the abnormal data from a preset database, and outputting the abnormal data and the abnormal data grade to perform early warning.
In an embodiment, the fuzzy clustering module 20 is further configured to perform normalization processing on the verification data to obtain normalized data, construct a fuzzy similar matrix according to the normalized data, and generate a dynamic clustering graph according to the fuzzy similar matrix.
In an embodiment, the fuzzy clustering module 20 is further configured to perform translation standard deviation transformation on the verification data to obtain standard deviation data, and use the standard deviation data within a preset standardized threshold range as standardized data.
In an embodiment, the fuzzy clustering module 20 is further configured to use standard deviation data that is not within a preset standardized threshold range as data to be transformed; carrying out translation range transformation on the data to be transformed to obtain range data; and taking the range of the range difference data within the preset standardized threshold value as standardized data.
In an embodiment, the fuzzy clustering module 20 is further configured to calculate a similarity coefficient of the normalized data by using a euclidean distance method; constructing a fuzzy similarity matrix according to the standardized data and the similarity coefficient; and generating a dynamic cluster map according to the fuzzy similar matrix.
In an embodiment, the data selecting module 30 is further configured to obtain data information and data distribution information from the dynamic cluster map, and determine abnormal data according to the data information and the data distribution information.
In an embodiment, the level matching module 40 is further configured to obtain an abnormal data type corresponding to the abnormal data and a data normal value of the abnormal data type; calculating a data abnormal value of the abnormal data according to the abnormal data and the data normal value; and matching abnormal data grades corresponding to the data abnormal values from a preset database.
Other embodiments or specific implementation methods of the data anomaly early warning device according to the present invention may refer to the above embodiments, and are not described herein again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM/RAM, magnetic disk, optical disk) as described above, and includes several instructions for enabling an intelligent terminal (which may be a mobile phone, a computer, a terminal, an air conditioner, or a network terminal) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A data abnormity early warning method is characterized by comprising the following steps:
acquiring verification data in a preset time period;
carrying out fuzzy clustering analysis on the verification data, and generating a dynamic clustering chart according to the result of the fuzzy clustering analysis;
selecting anomalous data from the dynamic cluster map;
matching abnormal data grades corresponding to the abnormal data from a preset database;
and outputting the abnormal data and the abnormal data grade to perform early warning.
2. The data anomaly early warning method according to claim 1, wherein the fuzzy clustering analysis is performed on the verification data, and a dynamic clustering chart is generated according to the fuzzy clustering analysis result, and the method specifically comprises the following steps:
standardizing the verification data to obtain standardized data;
and constructing a fuzzy similar matrix according to the standardized data, and generating a dynamic cluster map according to the fuzzy similar matrix.
3. The data anomaly early warning method according to claim 2, wherein the step of standardizing the verification data to obtain standardized data specifically comprises:
carrying out translation standard deviation transformation on the verification data to obtain standard deviation data;
and taking the standard deviation data within a preset standardized threshold value range as standardized data.
4. The data anomaly warning method as claimed in claim 3, wherein after the step of using the standard deviation data within the preset standardized threshold as the standardized data, the method further comprises the following steps:
taking standard deviation data which is not within the range of a preset standardized threshold value as data to be transformed;
carrying out translation range transformation on the data to be transformed to obtain range data;
and taking the range of the range difference data within the preset standardized threshold value as standardized data.
5. The data anomaly early warning method as set forth in claim 2. The method is characterized in that a fuzzy similar matrix is constructed according to the standardized data, and a dynamic clustering chart is generated according to the fuzzy similar matrix, and specifically comprises the following steps:
calculating a similarity coefficient of the standardized data by adopting an Euclidean distance method;
constructing a fuzzy similarity matrix according to the standardized data and the similarity coefficient;
and generating a dynamic cluster map according to the fuzzy similar matrix.
6. The data anomaly early warning method according to claim 1, wherein the selecting anomaly data from the dynamic cluster map specifically comprises:
acquiring data information and data distribution information from the dynamic cluster map;
and determining abnormal data according to the data information and the data distribution information.
7. The data abnormality early warning method according to any one of claims 1 to 6, wherein the matching of the abnormal data grade corresponding to the abnormal data from a preset database specifically includes:
acquiring an abnormal data type corresponding to the abnormal data and a data normal value of the abnormal data type;
calculating a data abnormal value of the abnormal data according to the abnormal data and the data normal value;
and matching abnormal data grades corresponding to the data abnormal values from a preset database.
8. A data abnormality warning device, characterized in that the data abnormality warning device includes:
the data acquisition module is used for acquiring verification data in a preset time period;
the fuzzy clustering module is used for carrying out fuzzy clustering analysis on the verification data and generating a dynamic clustering chart according to the result of the fuzzy clustering analysis;
the data selection module is used for selecting abnormal data from the dynamic cluster map;
the grade matching module is used for matching the abnormal data grade corresponding to the abnormal data from a preset database;
and the output early warning module is used for outputting the abnormal data and the abnormal data grade to carry out early warning.
9. A data abnormality warning apparatus, characterized in that the data abnormality warning apparatus includes: a memory, a processor, and a data anomaly early warning program stored on the memory and executable on the processor, the data anomaly early warning program configured to implement the steps of the data anomaly early warning method as claimed in any one of claims 1 to 7.
10. A storage medium having stored thereon a data anomaly early warning program, which when executed by a processor implements the steps of the data anomaly early warning method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911010978.5A CN110781494A (en) | 2019-10-22 | 2019-10-22 | Data abnormity early warning method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911010978.5A CN110781494A (en) | 2019-10-22 | 2019-10-22 | Data abnormity early warning method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110781494A true CN110781494A (en) | 2020-02-11 |
Family
ID=69386370
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911010978.5A Pending CN110781494A (en) | 2019-10-22 | 2019-10-22 | Data abnormity early warning method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110781494A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112597263A (en) * | 2021-03-03 | 2021-04-02 | 浙江和达科技股份有限公司 | Pipe network detection data abnormity judgment method and system |
| CN113870516A (en) * | 2021-10-20 | 2021-12-31 | 利天万世新能源有限公司 | Communication base station fire detection system |
| CN114088388A (en) * | 2021-12-10 | 2022-02-25 | 华润电力技术研究院有限公司 | Fault diagnosis method and fault diagnosis device for gearbox |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103198175A (en) * | 2013-03-04 | 2013-07-10 | 辽宁省电力有限公司鞍山供电公司 | Transformer fault diagnosis method based on fuzzy cluster |
| US20130329973A1 (en) * | 2012-06-06 | 2013-12-12 | The Regents Of The University Of Michigan | Subvolume identification for prediction of treatment outcome |
| CN108595655A (en) * | 2018-04-27 | 2018-09-28 | 福建师范大学 | A kind of abnormal user detection method of dialogue-based characteristic similarity fuzzy clustering |
| CN108765194A (en) * | 2018-05-29 | 2018-11-06 | 深圳源广安智能科技有限公司 | A kind of effective residential electricity consumption behavior analysis system |
| CN109472159A (en) * | 2018-11-15 | 2019-03-15 | 泰康保险集团股份有限公司 | Access control method, device, medium and electronic equipment |
| CN109600345A (en) * | 2017-09-30 | 2019-04-09 | 北京国双科技有限公司 | Abnormal data flow rate testing methods and device |
| CN109829571A (en) * | 2019-01-11 | 2019-05-31 | 国网浙江省电力有限公司 | A kind of short-term electricity demand forecasting method of user based on multilist fused data |
-
2019
- 2019-10-22 CN CN201911010978.5A patent/CN110781494A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130329973A1 (en) * | 2012-06-06 | 2013-12-12 | The Regents Of The University Of Michigan | Subvolume identification for prediction of treatment outcome |
| CN103198175A (en) * | 2013-03-04 | 2013-07-10 | 辽宁省电力有限公司鞍山供电公司 | Transformer fault diagnosis method based on fuzzy cluster |
| CN109600345A (en) * | 2017-09-30 | 2019-04-09 | 北京国双科技有限公司 | Abnormal data flow rate testing methods and device |
| CN108595655A (en) * | 2018-04-27 | 2018-09-28 | 福建师范大学 | A kind of abnormal user detection method of dialogue-based characteristic similarity fuzzy clustering |
| CN108765194A (en) * | 2018-05-29 | 2018-11-06 | 深圳源广安智能科技有限公司 | A kind of effective residential electricity consumption behavior analysis system |
| CN109472159A (en) * | 2018-11-15 | 2019-03-15 | 泰康保险集团股份有限公司 | Access control method, device, medium and electronic equipment |
| CN109829571A (en) * | 2019-01-11 | 2019-05-31 | 国网浙江省电力有限公司 | A kind of short-term electricity demand forecasting method of user based on multilist fused data |
Non-Patent Citations (2)
| Title |
|---|
| 卢验锋等: "考虑数据聚类特性的电能质量分级预警阈值设置方法", 《电力电容器与无功补偿》 * |
| 张悦等: "关于大型数据传输信息异常快速预警仿真研究", 《计算机仿真》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112597263A (en) * | 2021-03-03 | 2021-04-02 | 浙江和达科技股份有限公司 | Pipe network detection data abnormity judgment method and system |
| CN112597263B (en) * | 2021-03-03 | 2021-05-11 | 浙江和达科技股份有限公司 | Pipe network detection data abnormity judgment method and system |
| CN113870516A (en) * | 2021-10-20 | 2021-12-31 | 利天万世新能源有限公司 | Communication base station fire detection system |
| CN114088388A (en) * | 2021-12-10 | 2022-02-25 | 华润电力技术研究院有限公司 | Fault diagnosis method and fault diagnosis device for gearbox |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3622402B1 (en) | Real time detection of cyber threats using behavioral analytics | |
| CN108512827B (en) | Method, device, equipment and storage medium for establishing abnormal login identification and supervised learning model | |
| CN110781494A (en) | Data abnormity early warning method, device, equipment and storage medium | |
| CN109918279B (en) | Electronic device, method for identifying abnormal operation of user based on log data and storage medium | |
| CN108270723B (en) | Method for acquiring predicted attack path of power network | |
| CN112884016B (en) | Cloud platform credibility assessment model training method and cloud platform credibility assessment method | |
| CN111859371A (en) | Privacy risk assessment method and device of application program and storage medium | |
| CN109670931B (en) | Loan user behavior detection method, loan user behavior detection device, loan user behavior detection equipment and loan user behavior detection storage medium | |
| CN111553241A (en) | Method, device and equipment for rejecting mismatching points of palm print and storage medium | |
| CN111626346A (en) | Data classification method, device, storage medium and device | |
| CN114817933A (en) | Method and device for evaluating robustness of business prediction model and computing equipment | |
| CN112115507B (en) | Cloud service interaction method and big data platform based on cloud computing and information digitization | |
| CN111275453A (en) | Industry identification method and system of Internet of things equipment | |
| CN111404835B (en) | Flow control method, device, equipment and storage medium | |
| CN111090242B (en) | Method, device and equipment for verifying precision of automatic driving test system and storage medium | |
| CN111061637B (en) | Interface testing method, interface testing device and storage medium | |
| CN113011961A (en) | Method, device and equipment for monitoring risk of company associated information and storage medium | |
| CN110233848B (en) | Asset situation analysis method and device | |
| CN108446907B (en) | Safety verification method and device | |
| EP4207006A1 (en) | Model generation program, model generation method, and model generation device | |
| CN112597209A (en) | Data verification method, device and system and computer readable storage medium | |
| CN114266643A (en) | Enterprise mining method, device, equipment and storage medium based on fusion algorithm | |
| EP3748549B1 (en) | Learning device and learning method | |
| CN111241277A (en) | Sparse graph-based user identity identification method and device | |
| CN112652399A (en) | Old man mental state detection terminal and old man mental state detection system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200211 |
|
| RJ01 | Rejection of invention patent application after publication |