CN110677271A - Big data alarm method, device, equipment and storage medium based on ELK - Google Patents
Big data alarm method, device, equipment and storage medium based on ELK Download PDFInfo
- Publication number
- CN110677271A CN110677271A CN201910757196.1A CN201910757196A CN110677271A CN 110677271 A CN110677271 A CN 110677271A CN 201910757196 A CN201910757196 A CN 201910757196A CN 110677271 A CN110677271 A CN 110677271A
- Authority
- CN
- China
- Prior art keywords
- data
- alarm
- characteristic
- elk
- generating
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Alarm Systems (AREA)
Abstract
The embodiment of the application discloses a big data alarm method, a device, equipment and a storage medium based on ELK, and relates to the technical field of big data processing. A big data alarm method based on ELK comprises the following steps: collecting original result data of a service; cleaning and processing the original result data to obtain effective data; performing multidimensional data extraction on the effective data; acquiring the proportion of each data feature in the total data, and distributing feature weights; and generating characteristic values of the various alarm characteristics, and generating and pushing alarm messages. The method can go deep into the analysis of finer granularity by extracting the effective data acquired by the original data in multiple dimensions, thereby better finding the abnormity in the service logic or the function point, and adjusting the alarm strategy by combining the two parameters which influence the alarm strategy, such as the characteristic weight and the characteristic value, thereby realizing the dynamic adjustment of the alarm strategy and avoiding the problems of false alarm, disturbance and the like of the alarm.
Description
Technical Field
The embodiment of the application relates to the technical field of big data processing, in particular to a big data alarm method, a big data alarm device, big data alarm equipment and a storage medium based on ELK.
Background
Logs, within a computer system, are an extremely important component. The logs mainly comprise system logs, service logs and safety logs. The system operation and development personnel can know the software and hardware information of the server through the log, consult the log information, analyze the reason of the error and the like. However, most of the existing computer systems are relatively complex, and many of the systems are not in one place and even cross-country. With the increasing complexity of computer systems and the advent of the big data era, it is easy to refer to dozens or even hundreds of servers which continuously generate various log data. In the face of such huge amount of log data, which is distributed in different places, if we need to search some important information, it would be very clumsy and inefficient to use the traditional tools and methods. Therefore, there is an urgent need for a centralized management method for logs, which integrates data from different sources into one place, and the concept of a centralized log system is generated. The service system can monitor the performance and key technical indexes of the service system in real time through a centralized log system during operation, log-type alarm information can be generated when abnormality occurs, and when an alarm occurs, operation and maintenance personnel check the alarm information and find whether a corresponding emergency plan exists or not.
However, at present, service monitoring alarm and analysis performed through a centralized log system are commonly performed, mainly aiming at a machine or a service interface response code, the analysis of interface response content is not deep, the problem of abnormal service logic or function points is difficult to find, the fine granularity of the alarm is not enough, the alarm strategy is single, the problem cannot be found quickly, and false alarm and harassment are avoided as much as possible.
Disclosure of Invention
The technical problem to be solved in the embodiments of the present application is to provide a big data alarm method, device, equipment and storage medium based on ELK, which can go deep into finer-grained analysis based on a centralized log system, can dynamically adjust an alarm policy, and solve the defect that the alarm policy is too single.
In order to solve the above technical problem, the big data alarm method based on the ELK according to the embodiment of the present application adopts the following technical solutions:
a big data alarm method based on ELK comprises the following steps:
collecting original result data of a service;
cleaning and processing the original result data through Logstash to obtain effective data;
performing multidimensional data extraction on the effective data through an elastic search to generate a plurality of multidimensional data features;
acquiring a plurality of data characteristics of each dimension and the proportion of each data characteristic in the total data of the data characteristics, and distributing characteristic weight for each data characteristic based on the proportion;
generating characteristic values of multiple types of alarm characteristics based on characteristic weights corresponding to various data characteristics, aggregating the multiple types of alarm characteristics, generating alarm messages based on the aggregated multiple types of alarm characteristics for pushing, or generating alarm messages for each type of alarm characteristics for pushing respectively.
According to the ELK-based big data alarm method, multi-dimensional data extraction is carried out on effective data obtained by original data, analysis with finer granularity can be carried out deeply, and therefore abnormity in service logic or function points can be found better; and by calculating the proportion of each data feature of each dimension in the total data of a plurality of data features, reasonably distributing the feature weight of each data feature, and adjusting the alarm strategy by combining the feature weight and the feature value which affect the parameter of the alarm strategy, the alarm strategy can be dynamically adjusted, and the problems of false alarm, disturbance and the like of the alarm are avoided.
Further, in the ELK-based big data alarm method, before the step of collecting the raw result data of the service, the method further includes:
monitoring logs of at least one service data source; the raw result data is recorded in the log.
By monitoring the log, the change of the original result data from the service data source can be acquired in real time, so that the original result data required by the user can be acquired in time.
Further, in the ELK-based big data alarm method, the step of performing multidimensional data extraction on the valid data through an elastic search to generate multidimensional data features includes:
performing data extraction on the dimension of the HTTP request message through an ElasticSearch to obtain three data characteristics of a request line, a request head and a request body;
and performing data extraction on the dimension of the HTTP response message through the ElasticSearch to obtain three data characteristics of a response line, a response head and a response body.
Further, after the step of generating a plurality of data features of multiple dimensions, the ELK-based big data alarm method further includes:
performing visual analysis and display of the multidimensional data features by Kibana.
And the multidimensional data characteristics of the service are visually analyzed and displayed, so that a user can conveniently locate problems and analyze reasons for the problems.
Further, after the step of generating a plurality of data features of multiple dimensions, the ELK-based big data alarm method further includes:
aggregating the multidimensional data features to generate an integral data set through elastic search;
and sending the data set to the ElasticSearch for storage.
The extracted technical features are aggregated to form an integral data set, so that the data can be conveniently stored and transmitted, and relevant logics and functions are realized after the data are read from the aggregated total data set in the subsequent steps. And the data set is stored through the elastic search, so that an index can be established for the data set, and the structured search and analysis of data information in the data set are facilitated.
Further, in the ELK-based big data alarm method, the step of generating the feature values of multiple types of alarm features based on the feature weights corresponding to the various data features includes:
setting a plurality of weight thresholds for each data characteristic;
determining an alarm grade corresponding to each data characteristic based on the characteristic weight corresponding to each data characteristic and a plurality of weight thresholds set for each data characteristic, and further generating characteristic values of multiple types of alarm characteristics based on the data characteristic and the alarm grade.
The alarm strategy can be adjusted dynamically by setting the weight threshold value, and the content and the strategy of the alarm information required by the current service data can be judged by comparing the characteristic weight of the data characteristic with the weight threshold value, so that the alarm strategy can be adjusted by combining the two parameters which influence the alarm strategy, namely the characteristic weight and the weight threshold value, so that the problems existing in the data can be analyzed in a more detailed and deeper manner, and the alarm can be performed more reasonably.
In order to solve the above technical problem, an embodiment of the present application further provides an ELK-based big data alarm device, which adopts the following technical solutions:
an ELK-based big data alarm device comprises:
the acquisition module is used for acquiring original result data of the service;
the processing module is used for cleaning and processing the original result data through the Logstash so as to obtain effective data;
the data extraction module is used for performing multi-dimensional data extraction on the effective data through the ElasticSearch to generate a plurality of multi-dimensional data characteristics;
the weight distribution module is used for acquiring a plurality of data characteristics of each dimension and the proportion of each data characteristic in the total data of the data characteristics, and distributing characteristic weight for each data characteristic based on the proportion;
and the pushing module is used for generating characteristic values of multiple types of alarm characteristics based on the characteristic weights corresponding to the various data characteristics, aggregating the multiple types of alarm characteristics, generating alarm messages based on the aggregated multiple types of alarm characteristics and pushing the alarm messages, or generating alarm messages for each type of alarm characteristics and pushing the alarm messages.
Further, the ELK-based big data alarm device further includes: a monitoring module; the monitoring module is used for monitoring logs of at least one service data source; the raw result data is recorded in the log.
According to the ELK-based big data alarm device, multi-dimensional data extraction is carried out on effective data obtained by original data, analysis with finer granularity can be carried out deeply, and therefore abnormity in service logic or function points can be found better; and by calculating the proportion of each data feature of each dimension in the total data of the plurality of data features, reasonably distributing the feature weight of each data feature, and adjusting the alarm strategy by combining the feature weight and the feature value which affect the parameter of the alarm strategy, the alarm strategy can be dynamically adjusted, and the problems of false alarm, disturbance and the like of the alarm are avoided.
In order to solve the above technical problem, an embodiment of the present application further provides a computer device, which adopts the following technical solutions:
a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the ELK-based big data alarm method according to any one of the above technical solutions when executing the computer program.
In order to solve the above technical problem, an embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solutions:
a computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and when being executed by a processor, the computer program implements the steps of the ELK-based big data alarm method according to any one of the above technical solutions.
Compared with the prior art, the embodiment of the application mainly has the following beneficial effects:
the embodiment of the application discloses a big data alarm method, a device, equipment and a storage medium based on ELK, wherein the big data alarm method based on ELK can deeply reach analysis of finer granularity by carrying out multi-dimensional data extraction on effective data acquired by original data, thereby better discovering the abnormity in service logic or function points; and by calculating the proportion of each data feature of each dimension in the total data of the plurality of data features, reasonably distributing the feature weight of each data feature, and adjusting the alarm strategy by combining two parameters which influence the alarm strategy, namely the feature weight and the feature value, the alarm strategy can be dynamically adjusted, and the problems of false alarm, disturbance and the like of the alarm are avoided.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Other figures can be derived from these figures.
FIG. 1 is a diagram of an exemplary system architecture to which embodiments of the present application may be applied;
FIG. 2 is a flowchart illustrating an embodiment of a big data alarm method based on ELK according to the present disclosure;
FIG. 3 is a schematic structural diagram of an embodiment of an ELK-based big data alarm device according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an embodiment of a computer device in an embodiment of the present application.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the description of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.
It is noted that the terms "comprises," "comprising," and "having" and any variations thereof in the description and claims of this application and the drawings described above are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus. In the claims, the description and the drawings of the specification of the present application, relational terms such as "first" and "second", and the like, may be used solely to distinguish one entity/action/object from another entity/action/object without necessarily requiring or implying any actual such relationship or order between such entities/actions/objects.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
In order to make the technical solutions of the present application better understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the relevant drawings in the embodiments of the present application.
As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have various communication client applications installed thereon, such as a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture experts Group Audio Layer III, mpeg compression standard Audio Layer 3), MP4 players (Moving Picture experts Group Audio Layer IV, mpeg compression standard Audio Layer 4), laptop portable computers, desktop computers, and the like.
The server 105 may be a server providing various services, such as a background server providing support for pages displayed on the terminal devices 101, 102, 103.
It should be noted that, the ELK-based big data alarm method provided in the embodiment of the present application is generally executed by a server/terminal device, and accordingly, the ELK-based big data alarm apparatus is generally disposed in the server/terminal device.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
ELK (ElasticSearch, logstack, Kibana) belongs to one of centralized log systems, and is a solution for centralized log data management of an open source, including three software products, namely ElasticSearch, logstack and Kibana.
Among them, the ElasticSearch is a real-time distributed search and analysis engine, which can be used for full-text search, structured search and analysis. The method is a search engine established on the basis of a full-text search engine Apache Lucene and written by using Java language.
The Logstash is an open-source distributed search engine, and provides three functions of searching, analyzing and storing data. Almost any type of log is supported, including system logs, business logs, and security logs. It can collect and filter logs from many sources and can output data in a variety of ways.
Kibana is an open-source data analysis and visualization platform, provides a log analysis friendly Web interface for Logstash and elastic search, and can help to summarize, analyze and search important data logs. It can search and interact data in the index of the ElasticSearch, and generate table graphs of various dimensions for showing.
With continuing reference to fig. 2, a flowchart of an embodiment of the ELK-based big data alarm method in the embodiment of the present application is shown. The ELK-based big data alarm method comprises the following steps:
step 201: and collecting original result data of the service.
In order to implement the adjustment of the alarm policy, we first need to obtain the original result data about the service, so as to adjust the alarm policy according to various features of the original result data. In this embodiment of the present application, a logstack tool in the ELK component may be used to perform the acquisition of the raw result data, so as to obtain the required raw result data.
In the testing stage of adjusting the alarm strategy, the original result data can be the result data of the interface test case, and can also be the result data of the test case of other test objects and contents. In the alarm instance, the original result data is the result data of the actual use case of the actual alarm object.
A test case refers to a set of test inputs, execution conditions, and expected results tailored for a particular purpose in order to test a certain program path or verify that a certain requirement is met. And the alarm example refers to real data input in practical application.
In this embodiment of the present application, before the step 201, the ELK-based big data alarm method further includes the steps of: monitoring logs of at least one service data source; the raw result data is recorded in the log.
The original result data of the service is generally recorded in the form of logs, and the change of the original result data from a service data source can be acquired in real time by monitoring the logs, so that the original result data required by the user can be acquired in time.
In the implementation manner of the embodiment of the application, in order to obtain a better monitoring effect, the original result data of the service can be monitored in real time through the monitoring platform. And when monitoring the log and collecting the original result data from the log, the log can be realized by a Logstash tool in the ELK component.
In a specific implementation manner of the embodiment of the present application, the log at least records the following field information: a requester, a responder and a status code.
The raw result data collected must include explicit field information so that basic information of fields such as request body, response body, status code, etc. can be extracted from the raw result data. Therefore, the log at least includes the above-mentioned several kinds of field information.
The "request body" represents a component in an HTTP (HyperText Transfer Protocol) request message. The HTTP request message includes: a request line, a request header, and a request body. The content of the request body is a message body, which encodes the component values in a page table into a formatted string, which carries data of a plurality of request parameters, and can transmit the request parameters.
The "response body" represents a component in the HTTP response message. The HTTP response message includes: a response row, a response header, and a response body. The content of the response body is text information returned to the client by the server, and can be a webpage or character string codes and the like.
The "status code" is a component in the response line of the HTTP response message, and is generally expressed as a three-digit number, and is used to feed back the processing result of the request to the client.
The format of the log is not limited, and the log can be in json or text and other formats.
In this embodiment of the application, an electronic device (for example, the server/terminal device shown in fig. 1) on which the ELK-based big data alarm method operates may receive, through a wired connection manner or a wireless connection manner, an instruction sent by a user to collect original result data and a monitoring log. It should be noted that the wireless connection means may include, but is not limited to, a 3G/4G connection, a WiFi connection, a bluetooth connection, a WiMAX connection, a Zigbee connection, a uwb (ultra wideband) connection, and other wireless connection means now known or developed in the future.
Step 202: and cleaning and processing the original result data through the Logstash to obtain effective data.
In the embodiment of the present application, the valid data is characterized by only containing data characteristics required in processing, that is, the above-mentioned request body, response body, status code, and characteristic (belonging to a service module, an operation type, etc.) data related to a service scenario, and there is no unnecessary redundant data. The purpose of processing the original data by using a Logstash tool is to clean redundant data such as useless data or dirty data in the original data, and after the redundant data is cleaned, the data input to the ElasticSearch can be ensured to only contain required data characteristics by performing processing such as conversion, cutting, calculation and the like on field contents in the original data, so that the processing efficiency of a subsequent flow can be improved, and the time can be saved.
The dirty data refers to data in the source system that is not in a given range or meaningless for actual service, or data format is illegal, and there is irregular coding and ambiguous service logic in the source system.
Step 203: performing multidimensional data extraction on the effective data through an elastic search to generate a plurality of multidimensional data characteristics.
In the embodiment of the present application, when data extraction is performed on the valid data from multiple dimensions, the data extraction is performed according to an actual scene of a service. The multidimensional data features generated by the multidimensional data extraction may include: the system comprises a service function module, an interface request mode, an http state code, a service state code, a request body, a response body and the like.
In order to understand the concept of multiple dimensions, we exemplify specific data features, and if we need to obtain complete data of an HTTP request message and an HTTP response message, we regard the HTTP request message and the HTTP response message as one different dimension respectively, and a request line, a request header, and a request body belong to the data features of the HTTP request message at the dimension, and a response line, a response header, and a response body belong to the data features of the HTTP response message at the dimension.
The multidimensional data characteristics may be different for different service types, and if some services are related to HTTP and some are unrelated, the unrelated services must not have data characteristics in two dimensions, namely HTTP request messages and HTTP response messages.
The effective data obtained by the original data is subjected to multi-dimensional data extraction, and fine-grained analysis can be further achieved, so that the abnormity in the service logic or the function point can be better found.
In this embodiment of the present application, after the step 203, the ELK-based big data alarm method further includes: performing visual analysis and display of the multidimensional data features by Kibana.
The visual analysis aims to clearly and effectively convey and communicate data information by means of graphical means by performing correlation analysis on the data and making a complete analysis chart. In the present application, the Kibana tool is utilized to perform visual analysis and display.
And the multi-dimensional data characteristics of the service are visually analyzed and displayed, so that a user can conveniently locate problems and analyze reasons for the problems.
Step 204: acquiring a plurality of data characteristics of each dimension and the proportion of each data characteristic in the total data of the plurality of data characteristics, and distributing characteristic weight for each data characteristic based on the proportion.
To better understand the concept of an alarm strategy, the following example is illustrated:
when the service data is alarmed for the user, the following problems may occur: problems occur in different instances of the same cluster, repeated alarm can be caused, and resources are wasted; the alarm messages are frequent, the user is swiped by the alarm messages, and the numbness can be caused after the problem of alarm inundation occurs; alarm messages for one fault occur simultaneously on different alarm receiving devices in a short time, resulting in great stress on the user.
Therefore, different alarm policies should be set for different services, different time periods and different employee levels. E.g. alarms for more important services should be set with higher priority; in a busy time period, the reminding frequency of the alarm message is reduced, and the excessive alarm messages are avoided; and a higher threshold is set for the alarm message of the staff with higher position level, so that only the more important alarm message can be sent to the staff with higher level for reminding.
In the embodiment of the present application, the importance of the data features to the generated alarm message is represented by using the feature weights, and the change of the importance of each data feature to the alarm message can be obtained in time according to the change of each data feature and the feature weight corresponding to each data feature, so that when the alarm message is adjusted, the reference index of the data feature with the increased feature weight is increased, and the reference index of the data feature with the decreased feature weight is decreased, so as to dynamically adjust the alarm policy.
In this embodiment of the application, after the step 203, the ELK-based big data alarm method further includes: aggregating the multidimensional data features to generate an integral data set through elastic search;
and sending the data set to the ElasticSearch for storage.
The extracted technical features are aggregated to form an integral data set, so that the data can be conveniently stored and transmitted, and relevant logics and functions are realized after the data are read from the aggregated total data set in the subsequent steps.
And the data set is stored through the elastic search, so that an index can be established for the data set, and the structured search and analysis of data information in the data set are facilitated.
Step 205: generating characteristic values of multiple types of alarm characteristics based on characteristic weights corresponding to various data characteristics, aggregating the multiple types of alarm characteristics, generating alarm messages based on the aggregated multiple types of alarm characteristics for pushing, or generating alarm messages for each type of alarm characteristics for pushing respectively.
In the embodiment of the present application, the types of the alarm features at least include the following: alarm mode, alarm module, alarm frequency, alarm interval, notifying personnel, alarm mode and important priority. The characteristic value refers to the specific alarm content of the alarm characteristic.
In a specific implementation manner of the embodiment of the present application, if the status codes are regarded as a data feature, an appropriate feature weight is assigned to each status code by calculating a percentage ratio of the status code in the whole data. In a certain alarm, the proportion of the state code is more than 10%, which means that the proportion of the state code is very high, and the characteristic weight of the state code needs to be increased, so that when the characteristic weight is increased, the data characteristic can be used for alarming as soon as possible, shortening the alarm interval, improving the important priority, sending alarm messages to more notification personnel and the like, and further the adjustment of the alarm strategy is realized.
When the warning message is pushed, the warning message is mainly pushed in the form of characters and sound, for example, the warning message can be pushed in the form of short messages, mails or voices to remind a user.
According to the alarm characteristic pushing mode, multiple types of alarm characteristics can be aggregated into one alarm message to be pushed, and multiple alarm messages can be generated for each type of alarm characteristics to be pushed.
In an implementation manner of the embodiment of the present application, the step of generating feature values of multiple types of alarm features based on feature weights corresponding to the various data features in step 205 includes:
setting a plurality of weight thresholds for each data characteristic;
determining an alarm grade corresponding to each data characteristic based on the characteristic weight corresponding to each data characteristic and a plurality of weight thresholds set for each data characteristic, and further generating characteristic values of multiple types of alarm characteristics based on the data characteristic and the alarm grade.
Wherein different weight thresholds set for the same item of data feature represent different alarm levels to be taken therefor, and each alarm level represents a degree to which a feature value of an alarm feature is adjusted in accordance with the item of data feature. The weight threshold is preset and can be adjusted according to actual needs, and the range of the weight thresholds is between 0% and 100%.
By following the data characteristics of each dimension and the proportion of each data characteristic, and performing logic processing on each data characteristic as required to obtain and set a weight threshold, the alarm strategy can be dynamically adjusted, the problems of false alarm, disturbance and the like of the alarm are avoided, and the purpose of setting the weight threshold is to better dynamically adjust the alarm strategy.
In this step, after setting a plurality of weight thresholds, first of all, based on the feature weight corresponding to each data feature, a plurality of weight thresholds set for each data feature are matched, and by comparing the feature weight with the plurality of weight thresholds, it is confirmed that the feature weight is closest to the range represented by which weight threshold, thereby determining the alarm level corresponding to each data feature, and further, based on the data feature and the alarm level matched with each data feature, generating the feature values of a plurality of types of alarm features.
When the alarm strategy is adjusted, the dynamic change of each alarm characteristic is explained by taking an alarm mode, an alarm frequency and an alarm interval as examples: if the abnormal data is used as the data characteristics when the abnormal data in the service is detected, firstly calculating the proportion of the abnormal data in the total data, and if the proportion of the abnormal data is larger, judging whether the alarm frequency needs to be increased and the alarm interval needs to be shortened by matching and comparing the preset weight threshold according to the characteristic weight set by the proportion; if the proportion of the abnormal data is smaller, the emergency degree and the importance of the alarm are relatively lower, and whether the alarm frequency is required to be reduced and the alarm interval is required to be lengthened is judged according to the comparison result of the characteristic weight and the weight threshold; the alarm frequency is increased/decreased, and the degree of shortening/lengthening of the alarm interval can be judged according to the alarm levels represented by different weight thresholds. The alarm mode may use text content generated according to the specific content of the abnormal data as a feature value. The above steps generate the characteristic values of multiple types of alarm characteristics by combining the data characteristics and the weight threshold value matched with the characteristic weight of the data characteristics, thereby realizing the dynamic adjustment of the alarm strategy.
The characteristic weight of the data characteristic is compared with the weight threshold value, and the content and the strategy of the alarm information required by the current service data can be judged, so that the alarm strategy is adjusted by combining the characteristic weight and the characteristic value which influence the alarm strategy, the problems in the data can be analyzed in a more detailed and deeper manner, and the alarm can be performed more reasonably.
According to the ELK-based big data alarm method, multi-dimensional data extraction is carried out on effective data obtained by original data, analysis with finer granularity can be carried out deeply, and therefore abnormity in service logic or function points can be found better; and by calculating the proportion of each data feature of each dimension in the total data of the plurality of data features, reasonably distributing the feature weight of each data feature, and adjusting the alarm strategy by combining two parameters which influence the alarm strategy, namely the feature weight and the feature value, the alarm strategy can be dynamically adjusted, and the problems of false alarm, disturbance and the like of the alarm are avoided.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the computer program is executed. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
With further reference to fig. 3, fig. 3 is a schematic structural diagram illustrating an embodiment of an ELK-based big data alarm device according to an embodiment of the present application. As an implementation of the method shown in fig. 2, the present application provides an embodiment of an ELK-based big data alarm device, where the embodiment of the device corresponds to the embodiment of the method shown in fig. 2, and the device may be applied to various electronic devices.
As shown in fig. 3, the ELK-based big data alarm device according to this embodiment includes:
an acquisition module 301; for collecting raw result data of the service.
A processing module 302; and the system is used for executing cleaning processing treatment on the original result data through the Logstash so as to obtain effective data.
A data extraction module 303; the data extraction device is used for performing multidimensional data extraction on the effective data through the ElasticSearch to generate a plurality of multidimensional data characteristics.
A weight assignment module 304; the method is used for acquiring the proportion of the data characteristics of each dimension and each data characteristic in the overall data of the data characteristics, and allocating characteristic weight to each data characteristic based on the proportion.
A push module 305; the method and the device are used for generating characteristic values of multiple types of alarm characteristics based on characteristic weights corresponding to various data characteristics, aggregating the multiple types of alarm characteristics, generating alarm messages based on the aggregated multiple types of alarm characteristics for pushing, or generating alarm messages for each type of alarm characteristics for pushing respectively.
In this embodiment of the present application, the ELK-based big data alarm device further includes: and a monitoring module. The monitoring module is used for monitoring logs of at least one service data source; the raw result data is recorded in the log.
In this embodiment of the present application, the ELK-based big data alarm device further includes: and a display module. The display module is used for performing visual analysis and display on the multidimensional data features through Kibana.
In this embodiment of the present application, the ELK-based big data alarm device further includes: and (4) a polymerization module. The aggregation module is used for aggregating the multidimensional data features to generate an integral data set through elastic search; and sending the data set to the ElasticSearch for storage.
In a specific implementation manner of the embodiment of the present application, the pushing module 305 is configured to set a plurality of weight thresholds for each data feature, determine an alarm level corresponding to each data feature based on a feature weight corresponding to each data feature and the plurality of weight thresholds set for each data feature, and further generate feature values of multiple types of alarm features based on the data features and the alarm levels.
According to the ELK-based big data alarm device, multi-dimensional data extraction is carried out on effective data obtained by original data, analysis with finer granularity can be carried out deeply, and therefore abnormity in service logic or function points can be found better; and by calculating the proportion of each data feature of each dimension in the total data of the plurality of data features, reasonably distributing the feature weight of each data feature, and adjusting the alarm strategy by combining two parameters which influence the alarm strategy, namely the feature weight and the feature value, the alarm strategy can be dynamically adjusted, and the problems of false alarm, disturbance and the like of the alarm are avoided.
In order to solve the technical problem, an embodiment of the present application further provides a computer device. Referring to fig. 4, fig. 4 is a block diagram of a basic structure of a computer device according to the present embodiment.
The computer device 6 comprises a memory 61, a processor 62, a network interface 63 communicatively connected to each other via a system bus. It is noted that only a computer device 6 having components 61-63 is shown, but it is understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead. As will be understood by those skilled in the art, the computer device is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable gate array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The computer device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer equipment can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.
The memory 61 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as a hard disk or a memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a flash Card (FlashCard), and the like, which are provided on the computer device 6. Of course, the memory 61 may also comprise both an internal storage unit of the computer device 6 and an external storage device thereof. In this embodiment, the memory 61 is generally used for storing an operating system installed in the computer device 6 and various types of application software, such as program codes of an ELK-based big data alarm method. Further, the memory 61 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 62 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 62 is typically used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is configured to execute the program code stored in the memory 61 or process data, for example, execute the program code of the ELK-based big data alarm method.
The network interface 63 may comprise a wireless network interface or a wired network interface, and the network interface 63 is typically used for establishing a communication connection between the computer device 6 and other electronic devices.
The present application further provides another embodiment, which is to provide a computer-readable storage medium storing an ELK-based big data alert program, where the ELK-based big data alert program is executable by at least one processor to cause the at least one processor to perform the steps of the ELK-based big data alert method as described above.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.
In the above embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical division, and other divisions may be realized in practice, for example, a plurality of modules or components may be combined or integrated into another system, or some features may be omitted, or not executed.
The modules or components may or may not be physically separate, and the components shown as modules or components may or may not be physical modules, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules or components can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The present application is not limited to the above-mentioned embodiments, the above-mentioned embodiments are preferred embodiments of the present application, and the present application is only used for illustrating the present application and not for limiting the scope of the present application, it should be noted that, for a person skilled in the art, it is still possible to make several improvements and modifications to the technical solutions described in the foregoing embodiments or to make equivalent substitutions for some technical features without departing from the principle of the present application. All equivalent structures made by using the contents of the specification and the drawings of the present application can be directly or indirectly applied to other related technical fields, and the same should be considered to be included in the protection scope of the present application.
It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and is provided for the purpose of enabling a thorough understanding of the disclosure of the application. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All other embodiments that can be obtained by a person skilled in the art based on the embodiments in this application without any creative effort and all equivalent structures made by using the contents of the specification and the drawings of this application can be directly or indirectly applied to other related technical fields and are within the scope of protection of the present application.
Claims (10)
1. A big data alarm method based on ELK is characterized by comprising the following steps:
collecting original result data of a service;
cleaning and processing the original result data through Logstash to obtain effective data;
performing multidimensional data extraction on the effective data through an elastic search to generate a plurality of multidimensional data features;
acquiring a plurality of data characteristics of each dimension and the proportion of each data characteristic in the total data of the data characteristics, and distributing characteristic weight for each data characteristic based on the proportion;
generating characteristic values of multiple types of alarm characteristics based on characteristic weights corresponding to various data characteristics, aggregating the multiple types of alarm characteristics, generating alarm messages based on the aggregated multiple types of alarm characteristics for pushing, or generating alarm messages for each type of alarm characteristics for pushing respectively.
2. The ELK-based big data alert method of claim 1, wherein before the step of collecting the raw result data of the service, the method further comprises:
monitoring logs of at least one service data source; the raw result data is recorded in the log.
3. The ELK-based big data alert method according to claim 1, wherein the step of performing multidimensional data extraction on the valid data through elastic search to generate multidimensional data features comprises:
performing data extraction on the dimension of the HTTP request message through an ElasticSearch to obtain three data characteristics of a request line, a request head and a request body;
and performing data extraction on the dimension of the HTTP response message through the ElasticSearch to obtain three data characteristics of a response line, a response head and a response body.
4. The ELK-based big data alert method of claim 1, wherein after the step of generating the multidimensional items of data features, the method further comprises:
performing visual analysis and display of the multidimensional data features by Kibana.
5. The ELK-based big data alert method of claim 1, wherein after the step of generating the multidimensional items of data features, the method further comprises:
aggregating the multidimensional data features to generate an integral data set through elastic search;
and sending the data set to the ElasticSearch for storage.
6. The ELK-based big data alarm method according to claim 1, wherein the step of generating feature values of multiple classes of alarm features based on the feature weights corresponding to the data features comprises:
setting a plurality of weight thresholds for each data characteristic;
determining an alarm grade corresponding to each data characteristic based on the characteristic weight corresponding to each data characteristic and a plurality of weight thresholds set for each data characteristic, and further generating characteristic values of multiple types of alarm characteristics based on the data characteristic and the alarm grade.
7. An ELK-based big data alarm device is characterized by comprising:
the acquisition module is used for acquiring original result data of the service;
the processing module is used for cleaning and processing the original result data through the Logstash so as to obtain effective data;
the data extraction module is used for performing multi-dimensional data extraction on the effective data through the ElasticSearch to generate a plurality of multi-dimensional data characteristics;
the weight distribution module is used for acquiring a plurality of data characteristics of each dimension and the proportion of each data characteristic in the total data of the data characteristics, and distributing characteristic weight for each data characteristic based on the proportion;
and the pushing module is used for generating characteristic values of multiple types of alarm characteristics based on the characteristic weights corresponding to the various data characteristics, aggregating the multiple types of alarm characteristics, generating alarm messages based on the aggregated multiple types of alarm characteristics and pushing the alarm messages, or generating alarm messages for each type of alarm characteristics and pushing the alarm messages.
8. The ELK-based big data alarm device of claim 7, further comprising: a monitoring module; the monitoring module is used for monitoring logs of at least one service data source; the raw result data is recorded in the log.
9. A computer device comprising a memory in which a computer program is stored and a processor which, when executing the computer program, implements the steps of the ELK-based big data alert method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program, which when executed by a processor implements the steps of the ELK-based big data alert method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910757196.1A CN110677271B (en) | 2019-08-16 | 2019-08-16 | Big data alarm method, device, equipment and storage medium based on ELK |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910757196.1A CN110677271B (en) | 2019-08-16 | 2019-08-16 | Big data alarm method, device, equipment and storage medium based on ELK |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110677271A true CN110677271A (en) | 2020-01-10 |
| CN110677271B CN110677271B (en) | 2022-06-24 |
Family
ID=69075340
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910757196.1A Active CN110677271B (en) | 2019-08-16 | 2019-08-16 | Big data alarm method, device, equipment and storage medium based on ELK |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110677271B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111276231A (en) * | 2020-02-27 | 2020-06-12 | 平安医疗健康管理股份有限公司 | Medical data monitoring method and device, computer equipment and storage medium |
| CN115913896A (en) * | 2022-11-09 | 2023-04-04 | 中国联合网络通信集团有限公司 | Device detection method, server and medium |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101325520A (en) * | 2008-06-17 | 2008-12-17 | 南京邮电大学 | Method for locating and analyzing fault of intelligent self-adapting network based on log |
| CN102034148A (en) * | 2010-12-08 | 2011-04-27 | 山东浪潮齐鲁软件产业股份有限公司 | Method for realizing event alert and storm-proof policy of monitoring system |
| CN106371986A (en) * | 2016-09-08 | 2017-02-01 | 上海新炬网络技术有限公司 | Log treatment operation and maintenance monitoring system |
| CN106940677A (en) * | 2017-02-13 | 2017-07-11 | 咪咕音乐有限公司 | One kind application daily record data alarm method and device |
| CN107566163A (en) * | 2017-08-10 | 2018-01-09 | 北京奇安信科技有限公司 | A kind of alarm method and device of user behavior analysis association |
| WO2018010176A1 (en) * | 2016-07-15 | 2018-01-18 | 华为技术有限公司 | Method and device for acquiring fault information |
| US20180373527A1 (en) * | 2017-04-21 | 2018-12-27 | Semmle Limited | Weighting static analysis alerts |
| CN109460341A (en) * | 2018-10-23 | 2019-03-12 | 郑州云海信息技术有限公司 | A kind of alarm method and device of daily record data |
| CN109471846A (en) * | 2018-11-02 | 2019-03-15 | 中国电子科技网络信息安全有限公司 | User behavior auditing system and method on a kind of cloud based on cloud log analysis |
| CN109492073A (en) * | 2018-10-31 | 2019-03-19 | 北京达佳互联信息技术有限公司 | Blog search method, blog search device and computer readable storage medium |
| CN109634818A (en) * | 2018-10-24 | 2019-04-16 | 中国平安人寿保险股份有限公司 | Log analysis method, system, terminal and computer readable storage medium |
| CN109684159A (en) * | 2018-09-07 | 2019-04-26 | 平安普惠企业管理有限公司 | Method for monitoring state, device, equipment and the storage medium of distributed information system |
| CN109688188A (en) * | 2018-09-07 | 2019-04-26 | 平安科技（深圳）有限公司 | Monitoring alarm method, apparatus, equipment and computer readable storage medium |
| CN109783322A (en) * | 2018-11-22 | 2019-05-21 | 远光软件股份有限公司 | A kind of monitoring analysis system and its method of enterprise information system operating status |
| CN110021150A (en) * | 2019-03-27 | 2019-07-16 | 阿里巴巴集团控股有限公司 | A kind of data processing method, device and equipment |
-
2019
- 2019-08-16 CN CN201910757196.1A patent/CN110677271B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101325520A (en) * | 2008-06-17 | 2008-12-17 | 南京邮电大学 | Method for locating and analyzing fault of intelligent self-adapting network based on log |
| CN102034148A (en) * | 2010-12-08 | 2011-04-27 | 山东浪潮齐鲁软件产业股份有限公司 | Method for realizing event alert and storm-proof policy of monitoring system |
| WO2018010176A1 (en) * | 2016-07-15 | 2018-01-18 | 华为技术有限公司 | Method and device for acquiring fault information |
| CN106371986A (en) * | 2016-09-08 | 2017-02-01 | 上海新炬网络技术有限公司 | Log treatment operation and maintenance monitoring system |
| CN106940677A (en) * | 2017-02-13 | 2017-07-11 | 咪咕音乐有限公司 | One kind application daily record data alarm method and device |
| US20180373527A1 (en) * | 2017-04-21 | 2018-12-27 | Semmle Limited | Weighting static analysis alerts |
| CN107566163A (en) * | 2017-08-10 | 2018-01-09 | 北京奇安信科技有限公司 | A kind of alarm method and device of user behavior analysis association |
| CN109684159A (en) * | 2018-09-07 | 2019-04-26 | 平安普惠企业管理有限公司 | Method for monitoring state, device, equipment and the storage medium of distributed information system |
| CN109688188A (en) * | 2018-09-07 | 2019-04-26 | 平安科技（深圳）有限公司 | Monitoring alarm method, apparatus, equipment and computer readable storage medium |
| CN109460341A (en) * | 2018-10-23 | 2019-03-12 | 郑州云海信息技术有限公司 | A kind of alarm method and device of daily record data |
| CN109634818A (en) * | 2018-10-24 | 2019-04-16 | 中国平安人寿保险股份有限公司 | Log analysis method, system, terminal and computer readable storage medium |
| CN109492073A (en) * | 2018-10-31 | 2019-03-19 | 北京达佳互联信息技术有限公司 | Blog search method, blog search device and computer readable storage medium |
| CN109471846A (en) * | 2018-11-02 | 2019-03-15 | 中国电子科技网络信息安全有限公司 | User behavior auditing system and method on a kind of cloud based on cloud log analysis |
| CN109783322A (en) * | 2018-11-22 | 2019-05-21 | 远光软件股份有限公司 | A kind of monitoring analysis system and its method of enterprise information system operating status |
| CN110021150A (en) * | 2019-03-27 | 2019-07-16 | 阿里巴巴集团控股有限公司 | A kind of data processing method, device and equipment |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111276231A (en) * | 2020-02-27 | 2020-06-12 | 平安医疗健康管理股份有限公司 | Medical data monitoring method and device, computer equipment and storage medium |
| CN115913896A (en) * | 2022-11-09 | 2023-04-04 | 中国联合网络通信集团有限公司 | Device detection method, server and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110677271B (en) | 2022-06-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111984499B (en) | Fault detection method and device for big data cluster | |
| CN110708204B (en) | Abnormity processing method, system, terminal and medium based on operation and maintenance knowledge base | |
| EP4099170B1 (en) | Method and apparatus of auditing log, electronic device, and medium | |
| CN112052111B (en) | Processing method, device and equipment for server abnormity early warning and storage medium | |
| CN111339175B (en) | Data processing method, device, electronic equipment and readable storage medium | |
| CN113157545A (en) | Method, device and equipment for processing service log and storage medium | |
| CN110830438A (en) | Abnormal log warning method and device and electronic equipment | |
| CN111585785B (en) | Method and device for shielding alarm information, computer equipment and storage medium | |
| CN113704065A (en) | Monitoring method, device, equipment and computer storage medium | |
| CN110677271B (en) | Big data alarm method, device, equipment and storage medium based on ELK | |
| CN111800292B (en) | Early warning method and device based on historical flow, computer equipment and storage medium | |
| CN112306700A (en) | Abnormal RPC request diagnosis method and device | |
| CN110807050B (en) | Performance analysis method, device, computer equipment and storage medium | |
| CN115237857A (en) | Log processing method and device, computer equipment and storage medium | |
| CN114091704A (en) | Alarm suppression method and device | |
| CN114157679A (en) | Cloud-native-based distributed application monitoring method, device, equipment and medium | |
| CN112948223A (en) | Method and device for monitoring operation condition | |
| CN114398343A (en) | Database abnormal key processing method, device, equipment and medium | |
| CN114443437A (en) | Alarm root cause output method, apparatus, device, medium, and program product | |
| CN111782488B (en) | Message queue monitoring method, device, electronic equipment and medium | |
| CN116843395A (en) | Alarm classification method, device, equipment and storage medium of service system | |
| CN112910733A (en) | Full link monitoring system and method based on big data | |
| CN116991675A (en) | Abnormal access monitoring method and device, computer equipment and storage medium | |
| CN111950623A (en) | Data stability monitoring method and device, computer equipment and medium | |
| Lee et al. | Detecting anomaly teletraffic using stochastic self-similarity based on Hadoop |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |