CN117828515A - Intelligent log abnormality diagnosis system and method based on low-code platform - Google Patents
Intelligent log abnormality diagnosis system and method based on low-code platform Download PDFInfo
- Publication number
- CN117828515A CN117828515A CN202410244864.1A CN202410244864A CN117828515A CN 117828515 A CN117828515 A CN 117828515A CN 202410244864 A CN202410244864 A CN 202410244864A CN 117828515 A CN117828515 A CN 117828515A
- Authority
- CN
- China
- Prior art keywords
- data
- user
- unit
- model
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005856 abnormality Effects 0.000 title claims abstract description 34
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000003745 diagnosis Methods 0.000 title claims abstract description 15
- 238000000605 extraction Methods 0.000 claims abstract description 44
- 238000005457 optimization Methods 0.000 claims abstract description 41
- 238000001514 detection method Methods 0.000 claims abstract description 36
- 238000012549 training Methods 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 28
- 238000013468 resource allocation Methods 0.000 claims abstract description 9
- 238000004422 calculation algorithm Methods 0.000 claims description 50
- 238000004458 analytical method Methods 0.000 claims description 27
- 238000012544 monitoring process Methods 0.000 claims description 23
- 230000002159 abnormal effect Effects 0.000 claims description 22
- 230000008569 process Effects 0.000 claims description 15
- 238000004140 cleaning Methods 0.000 claims description 12
- 238000010801 machine learning Methods 0.000 claims description 11
- 238000013500 data storage Methods 0.000 claims description 10
- 238000001914 filtration Methods 0.000 claims description 9
- 238000007781 pre-processing Methods 0.000 claims description 9
- 230000003993 interaction Effects 0.000 claims description 7
- 238000012800 visualization Methods 0.000 claims description 7
- 238000007637 random forest analysis Methods 0.000 claims description 4
- 238000009826 distribution Methods 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 claims description 3
- 238000002790 cross-validation Methods 0.000 claims description 2
- 238000012423 maintenance Methods 0.000 abstract description 15
- 230000006870 function Effects 0.000 abstract description 6
- 238000011161 development Methods 0.000 description 10
- 230000009471 action Effects 0.000 description 7
- 206010000117 Abnormal behaviour Diseases 0.000 description 3
- 230000006399 behavior Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 238000013480 data collection Methods 0.000 description 2
- 238000013524 data verification Methods 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 230000007115 recruitment Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention provides an intelligent log abnormity diagnosis system and method based on a low-code platform, which belong to the technical field of computers and comprise the following steps: the system comprises a data acquisition and processing module, a characteristic extraction and learning module, an abnormality detection and alarm module, a user feedback and optimization module and a resource optimization configuration module, wherein the data acquisition and processing module, the characteristic extraction and learning module, the abnormality detection and alarm module, the user feedback and optimization module and the resource optimization configuration module are used for acquiring log data, performing characteristic extraction on the processed log data, inputting the processed log data into a model for training, and obtaining a trained model; embedding the trained model into the model to acquire abnormality detection information and carrying out alarm notification when an abnormality access mode exists; after receiving the alarm notification, the user receives and records feedback information of the user; the load condition of the server is monitored, computing resources are reasonably distributed, the technical threshold is reduced, and the system maintenance efficiency is improved; meanwhile, the functions of user feedback, optimization, resource allocation optimization and the like enable the system to be more intelligent, can adapt to different operation and maintenance requirements, improve the stability and maintainability of the system, and reduce maintenance cost.
Description
Technical Field
The invention belongs to the field of computers, and particularly relates to an intelligent log abnormality diagnosis system and method based on a low-code platform.
Background
As modern enterprises increasingly rely on software applications to drive business growth and improve efficiency, traditional software development methods are cumbersome, time consuming, and costly. This has led to a need for more flexible and efficient development. Conventional development methods typically require extensive source code writing, which requires employment of highly skilled developers, long development cycles, and difficulty in accommodating rapidly changing market demands. In addition, recruitment and conservation of talents becomes increasingly challenging due to the rapid development of the technology field. These factors together have driven the rise of low code platforms. The low code platform provides visualization tools and pre-build components for developers to enable them to create applications at a faster rate without requiring extensive programming experience. This not only speeds up the development and delivery of the application, but also reduces development costs. In addition, the appearance of a low-code platform also reduces the skill threshold, so that more people can participate in the development of the application program, and the scale of a development team is enlarged. The background of low code platforms is the need for digital transformation, as well as the bottleneck of traditional development methods. It represents a more modern and flexible way of application development, hopefully continuing to influence future software development areas.
Notably, in modern system operation and maintenance, managing and maintaining log data for large-scale systems is a cumbersome and critical task. Manual analysis of log data is not only time consuming and laborious, but also prone to error. Therefore, it is necessary to introduce automated and intelligent solutions to increase efficiency and reduce operation and maintenance costs. Current system log management field current problems:
manual data acquisition: traditionally, system administrators have been required to manually configure and perform data collection tasks to obtain log data from the various system components and devices, which is time-consuming and labor-intensive.
Data processing is difficult: unstructured log data requires complex processing and cleaning for further analysis.
Manual anomaly detection: the manual detection of the exception log mode requires a significant amount of time and resources and is prone to missing potential problems.
Model optimization is complex: in terms of machine learning models, extensive expertise and a great deal of manual work are required to perform model training and optimization.
Disclosure of Invention
The invention aims at the problems in the prior art, and in a first aspect, the invention provides an intelligent log abnormality diagnosis system based on a low-code platform, which comprises the following components: the system comprises a data acquisition and processing module, a feature extraction and learning module, an abnormality detection and alarm module, a user feedback and optimization module and a resource optimization configuration module;
the data acquisition and processing module is used for acquiring log data and removing repeated logs;
the feature extraction and learning module is used for extracting features of the processed log data and inputting the features into the model for training to obtain a trained model;
the abnormality detection and alarm module is used for configuring an abnormality judgment algorithm, embedding the trained model into the abnormality detection and alarm module to acquire abnormality detection information and carry out alarm notification when an abnormality access mode exists;
the user feedback and optimization module is used for receiving and recording user feedback information after the user receives the alarm notification;
the optimizing module and the resource optimizing configuration module are used for monitoring the load condition of the server and reasonably distributing the computing resources so as to ensure the analysis performance.
Further, the data acquisition and processing module comprises an automatic data acquisition unit, a log data preprocessing unit, a data cleaning and filtering unit and a log data storage unit;
the automatic data acquisition unit is used for automatically acquiring log data of various types from different system components and devices, and a user can define a data source and acquisition rules through a configuration interface of the low-code platform;
the log data preprocessing unit is used for converting unstructured data into structured data, and the low-code platform provides a data preprocessing component for the unstructured log data so that a user can easily define rules;
the data cleaning and filtering unit is used for removing invalid log data, ensuring the quality of the analyzed data, utilizing a data cleaning component of the low-code platform, and defining data cleaning rules by a user;
the log data storage unit is used for storing and managing data, the processed log data can be automatically stored in the database, the low-code platform provides a database operation component, and a user configures the data storage mode and period.
Further, the feature extraction and learning module comprises a feature extraction algorithm unit, a machine learning model configuration unit and a model training and optimizing unit;
the feature extraction algorithm unit is used for extracting features, a user selects a feature extraction component, and different feature extraction algorithms are configured according to different log data types; these components are already packaged so that the user does not have to know the details of the algorithm;
the machine learning model configuration unit is used for generating a model training and evaluating flow by taking the characteristic extraction result as input; on a low-code platform, a user selects a machine learning model, configures parameters and takes a result of feature extraction as input by using a visual interface; the platform automatically generates a model training and evaluating flow;
the model training and optimizing unit is used for carrying out model training and optimizing, and the user automatically carries out model training and optimizing through the low-code platform; the platform can automatically generate a cross-validation and parameter adjustment flow, so that the training of the model is simplified.
Further, the abnormality detection and alarm module comprises an abnormality judgment algorithm configuration unit, an alarm strategy visualization unit and a real-time alarm unit;
the anomaly judgment algorithm configuration unit is used for configuring an anomaly judgment algorithm, and a user configures the anomaly judgment algorithm through a low-code platform to embed the trained model into the anomaly judgment algorithm; the platform provides an interface for algorithm selection and parameter configuration, so that the configuration process is simplified;
the alarm strategy visualization unit is used for visually configuring the alarm strategy, and on the platform, the user visually configures the alarm strategy, including a notification mode and a receiver;
the real-time alarm unit is used for automatically sending out alarm notification according to the configured alarm strategy; when the system detects an abnormal condition, the low-code platform automatically sends out an alarm notification according to the configured alarm strategy. The notification content includes descriptions of anomalies, scores, and possible causes.
Further, the user feedback and optimization module comprises a user feedback interaction unit and a model optimization automation unit;
the user feedback interaction unit is used for providing feedback information, confirming abnormal conditions or providing additional information, and a user replies an alarm notification through an interface on the platform; the feedback information of the user is recorded and used for optimizing the subsequent model;
the model optimization automation unit is used for updating the model, improving the judgment accuracy, and the model optimization process is automated by utilizing the low-code platform; the feedback information of the user can automatically update the model, so that the judgment accuracy is improved.
Further, the resource optimal configuration module comprises a resource monitoring and configuration unit and an automatic resource scheduling unit;
the resource monitoring and configuration unit is used for automatically adjusting the distribution of computing resources according to the monitoring data, and the low-code platform integrates a resource monitoring tool to monitor the use condition of system resources in real time;
the automatic resource scheduling unit is used for automatically adjusting the computing resource allocation of the analysis and judgment module, and the resource scheduling component on the platform automatically adjusts the computing resource allocation of the analysis and judgment module according to the monitoring data so as to ensure the performance.
In a second aspect, the present invention provides a working method of the low-code platform-based intelligent log abnormality diagnosis system, which comprises the following steps: on a low-code platform, a system administrator configures a data source, defines rules, performs data acquisition, and periodically acquires access log data from a website server;
extracting keywords in the access log by using a feature extraction component, selecting a random forest model for training, and extracting features; the configuration process comprises selecting a feature extraction algorithm and model parameters;
and configuring an anomaly detection component, embedding the trained model, and setting an alarm strategy. When an abnormal access mode exists, the system automatically sends out alarm notification, including abnormal description, scoring and possible reasons;
after receiving the alarm, the user provides feedback information through the platform interface to confirm abnormal conditions or provide additional information; the feedback information is automatically recorded and used for model optimization;
the platform monitors the load condition of the server and automatically allocates more computing resources to the anomaly detection module so as to ensure analysis performance.
Compared with the prior art, the invention has the advantages and positive effects that:
1. the invention has the advantages that the automatic log data processing, feature extraction, model training and anomaly detection are realized through the low-code platform, the technical threshold is reduced, and the system maintenance efficiency is improved. Meanwhile, the functions of user feedback, optimization, resource allocation optimization and the like enable the system to be more intelligent, can adapt to different operation and maintenance requirements, improve the stability and maintainability of the system, and reduce maintenance cost.
2. The invention provides an intelligent log abnormality diagnosis system and method based on a low-code platform, and provides an efficient and intelligent solution for the field of system operation and maintenance. The system can automatically process log data, improve the abnormality detection accuracy, and perform intelligent optimization according to user feedback and resource monitoring data, and is suitable for system maintenance tasks of various scales.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it will be obvious that the drawings in the following description are some embodiments of the present invention, and that other drawings can be obtained according to these drawings without inventive effort to a person skilled in the art.
FIG. 1 is a schematic diagram of the low-code platform-based intelligent log anomaly diagnosis system.
Detailed Description
In order that the above objects, features and advantages of the invention will be more clearly understood, a further description of the invention will be rendered by reference to the appended drawings and examples. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced otherwise than as described herein, and therefore the present invention is not limited to the specific embodiments of the disclosure that follow.
As noted in the background, managing and maintaining log data for large-scale systems is a cumbersome and critical task in modern system operation and maintenance. Manual analysis of log data is not only time consuming and laborious, but also prone to error. Therefore, it is necessary to introduce automated and intelligent solutions to increase efficiency and reduce operation and maintenance costs. Current system log management field current problems:
manual data acquisition: traditionally, system administrators have been required to manually configure and perform data collection tasks to obtain log data from the various system components and devices, which is time-consuming and labor-intensive. Data processing is difficult: unstructured log data requires complex processing and cleaning for further analysis. Manual anomaly detection: the manual detection of the exception log mode requires a significant amount of time and resources and is prone to missing potential problems. Model optimization is complex: in terms of machine learning models, extensive expertise and a great deal of manual work are required to perform model training and optimization.
According to the invention, automatic log data processing, feature extraction, model training and anomaly detection are realized through the low-code platform, so that the technical threshold is reduced, and the system maintenance efficiency is improved. Meanwhile, the functions of user feedback, optimization, resource allocation optimization and the like enable the system to be more intelligent, can adapt to different operation and maintenance requirements, improve the stability and maintainability of the system, and reduce maintenance cost.
Embodiment 1, as shown in fig. 1, the main objective of the present embodiment is to provide an intelligent log anomaly diagnosis system based on a low code platform, which includes: the system comprises a data acquisition and processing module, a feature extraction and learning module, an abnormality detection and alarm module, a user feedback and optimization module and a resource optimization configuration module;
the data acquisition and processing module comprises an automatic data acquisition unit, a log data preprocessing unit, a data cleaning and filtering unit and a log data storage unit;
the automatic data acquisition unit is used for automatically acquiring log data of various types from different system components and devices, and a user can define a data source and acquisition rules through a configuration interface of the low-code platform, and the configuration rules are used for periodically acquiring access logs from a website server.
Rule definition: a configuration rule is a set of instructions or criteria that directs the system how to analyze log data. Rules may be based on specific log events such as error codes, frequent login attempts, or unusual user behavior.
Conditions and triggers: rules contain conditions and triggers. The conditions are the specific log patterns or actions that need to be met, and the triggers define the actions that the system should take when these conditions are met.
Action and response: the configuration rules specify the actions that the system should take when a particular condition is detected. Such actions may include generating an alarm, initiating a particular flow, or notifying a system administrator.
Flexibility and customization: on low code platforms, configuration rules are typically designed to be very flexible, allowing users to customize the rules according to their own needs. This means that the user can adjust the rules according to specific business scenarios and requirements.
Automation and efficiency: by means of automatic log analysis, the rules greatly improve the efficiency and accuracy of anomaly detection and reduce the dependence on manual monitoring.
Case:
configuration rule example: a continuous login failure detection rule; purpose and action: this rule is intended to identify potential security threats, such as brute force attacks. When the system detects that login fails from the same IP address three or more times in succession, a security alarm is triggered. Rule setting step: a new rule is defined in the log diagnostic system. The rule condition is set to "three consecutive login failures". The trigger condition is specified as "these login attempts come from the same IP address". A response action specifying the rule, such as sending an alert mail to a system administrator. Application cases: on a certain day, five consecutive log-in failure records from IP address "192.168.1.10" appear in the log. According to the set rules, the system automatically recognizes this pattern and sends an alert mail to the administrator, which contains the relevant log details and possible security suggestions. After receiving the mail, the administrator checks the activity of the IP address and takes corresponding security measures.
The log data preprocessing unit is used for converting unstructured data into structured data, and the low-code platform provides a data preprocessing component for unstructured log data, so that a user can easily define rules, for example, converting text-type log data into log records in a standard format.
Standard formats of log data, including time stamps, log levels, sources, message content, user information, and other metadata; wherein, timestamp: the exact time of event occurrence is recorded. Log level: e.g., INFO, ERROR, WARNING, etc., indicates the importance of the log message. The source is as follows: a component or service that indicates the source of the log message. Message content: the log information actually recorded may include error descriptions, status updates, etc. User information (if applicable): and recording the user identity information of the operation. Other metadata: such as process ID, thread ID, hostname, etc.
The data cleansing and filtering unit is used for removing invalid log data, ensuring the quality of the analyzed data, and the user can define data cleansing rules, such as removing duplicate logs or unnecessary fields, by utilizing the data cleansing component of the low code platform.
The processing process of the data cleaning and filtering unit comprises input processing, data verification, noise filtering, normalization, anomaly detection and output; wherein, input processing: receiving original log data, and identifying and analyzing each component in a standard format; and (3) data verification: checking data integrity and format accuracy, such as timestamp format and log level validity; noise filtering: removing insignificant log entries, such as conventional status messages; normalizing: converting the log data into a consistent format for subsequent processing; abnormality detection: identifying abnormal or unusual journal entries for further analysis; and (3) outputting: the cleaned log data is passed to the next component of the system, such as a storage, analysis, or anomaly diagnostic module.
The log data storage unit is used for storing and managing data, the processed log data can be automatically stored in the database, the low-code platform provides a database operation component, and a user can configure the data storage mode and period.
The component structure of the log data storage unit comprises an interface layer, a logic processing layer, a data access layer and a security and integrity layer; wherein, the interface layer: providing interfaces for interaction with other parts of the system, such as receiving a query request and returning a query result; logic processing layer: the method comprises the steps of analyzing query instructions, and generating and optimizing a database query plan; data access layer: interacting with a database, and executing actual data retrieval, updating and storage operations; security and integrity layer: the security of data operation and the integrity of data are ensured.
Working principle:
when the system generates log analysis requests, the interface layer receives the requests and forwards the requests to the logic processing layer;
the logic processing layer analyzes the request, converts the request into a query language suitable for understanding the database, and optimizes the query efficiency;
the data access layer interacts with the database to perform inquiry, retrieve data or update database records;
the security and integrity layer monitors all operations, ensures compliance with the data access protocol, and protects the data from unauthorized access or tampering.
The feature extraction and learning module comprises a feature extraction algorithm unit, a machine learning model configuration unit and a model training and optimizing unit; the feature extraction algorithm unit is used for extracting features, a user selects a feature extraction component, and different feature extraction algorithms are configured according to different log data types; the components are packaged, so that a user does not need to know the details of the algorithm, the used algorithm is mature in the market, and a mature interface can be directly called; as one implementation, the user may choose to use TF-IDF algorithm to extract keyword features of the text log. The machine learning model configuration unit is used for generating a model training and evaluating flow by taking the characteristic extraction result as input; on the low-code platform, a user can select a machine learning model, configure parameters and take the result of feature extraction as input by using a visual interface; the platform automatically generates a model training and evaluating flow; as one implementation, the user may select a random forest model and set the number and depth of trees. The model training and optimizing unit is used for performing model training and optimizing, and through the low-code platform, a user can automatically perform model training and optimizing. The platform can automatically generate the processes of cross verification, parameter adjustment and the like, so that the training of the model is simplified. The user may monitor the training process and adjust based on the results.
Different feature extraction algorithms configured for different log data types are as follows:
feature extraction algorithm of system log: analyzing the regular expression and extracting keywords; application: for identifying system faults, configuration errors, etc.
Feature extraction algorithm for application log: log template matching and sequence pattern mining. Application: for understanding the running state and abnormal behavior of the application.
Feature extraction algorithm of security log: abnormal behavior analysis and rule-based detection. Application: for detecting security threats such as intrusion attempts.
Feature extraction algorithm of weblog: traffic analysis, IP address and port behavior analysis. Application: the method is used for monitoring network traffic and detecting abnormal traffic.
Feature extraction algorithm of database log: transaction pattern analysis, SQL query analysis. Application: for monitoring database operations and detecting abnormal transactions.
The abnormality detection and alarm module comprises an abnormality judgment algorithm configuration unit, an alarm strategy visualization unit and a real-time alarm unit; the anomaly judgment algorithm configuration unit is used for configuring an anomaly judgment algorithm, and a user can configure the anomaly judgment algorithm through a low-code platform to embed the trained model into the anomaly judgment algorithm. The platform provides an interface for algorithm selection and parameter configuration, and simplifies the configuration process. As one implementation, the user may choose to use the anomaly score of the model to determine anomalies.
The specific algorithm is as follows: the anomaly determination may use a variety of algorithms, such as: isolation Forest (Isolation Forest): outliers are isolated based on tree-structured algorithms. Local anomaly factor (Local Outlier Factor, LOF): outliers with low densities are identified based on an algorithm of the density of neighboring points. Single Class support vector machine (One-Class SVM): and searching a decision boundary which can most represent normal data in the feature space.
Training model embedding: the model is trained using historical data (including normal and abnormal instances). After the model is trained, the model is embedded into a system for anomaly judgment of real-time data.
User configuration: the user can adjust algorithm parameters such as the neighbor number of the LOF or the number of trees of the isolated forest according to the service requirement. The user may also update the training data set periodically to accommodate new data trends and anomaly patterns.
As one embodiment, if a system uses an isolated forest algorithm, the user can adjust the number of trees to change the sensitivity of the detection. After the dataset is updated, the system may retrain the model to improve its accuracy.
The alarm strategy visualization unit is used for visually configuring the alarm strategy, and on the platform, a user can visually configure the alarm strategy, including a notification mode and a receiver. This facilitates the adjustment of the alarm strategy. As one embodiment, the user may set the alert notification manner to mail and designate the recipient.
The real-time alarm unit is used for automatically sending out alarm notification according to the configured alarm strategy; when the system detects an abnormal condition, the low-code platform automatically sends out an alarm notification according to the configured alarm strategy. The notification content includes descriptions of anomalies, scores, and possible causes. As one embodiment, the system automatically sends mail alerts, the content including exception log information and suggested processing steps.
Obtaining abnormal log information: the system analyzes the log data in real time through a preconfigured abnormality judgment algorithm. When abnormal behavior is detected, the system may record relevant log information such as time stamps, event types, error codes, affected system parts, etc.
Suggested processing steps:
primary diagnosis: based on the anomaly log, the system or administrator makes a preliminary diagnosis, determining the nature and possible cause of the anomaly.
Notifying relevant personnel: automatically sending a notification to the relevant technical support team or administrator.
Backing up data: the affected data and system state are backed up before further operations are performed.
Detailed analysis: a more extensive log analysis and system checks are performed to determine the specific cause of the anomaly.
The implementation solution is as follows: based on the analysis results, repair measures such as updating configuration, restarting services, applying patches, etc. are implemented.
And (3) verification and monitoring: after implementing the solution, it is verified whether the problem has been solved and the system is continuously monitored.
Document record: the process and results are recorded for future reference.
The user feedback and optimization module comprises a user feedback interaction unit and a model optimization automation unit; the user feedback interaction unit is used for providing feedback information, confirming abnormal conditions or providing additional information, and on the platform, a user can reply an alarm notice through an interface, provide the feedback information, and confirm the abnormal conditions or provide the additional information. The feedback information of the user is recorded and used for the optimization of the subsequent model. As one implementation, the user may click on a link in an alert mail, enter a platform interface, view more detailed information, and provide feedback.
The model optimization automation unit is used for updating the model, improving judgment accuracy, and the model optimization process is automated by using the low-code platform. The feedback information of the user can automatically update the model, so that the judgment accuracy is improved. As one embodiment, the information fed back by the user may be used for model retraining to improve the performance of anomaly detection.
The user feedback content:
performance feedback: user assessment of system performance such as response time and processing speed.
Error reporting: a detailed description of the errors or problems encountered by the user.
Function advice: user suggestions for system improvements or new functions.
The use experience is as follows: experience and satisfaction of the user during use.
Retraining the model using the feedback information:
data integration: the user feedback is integrated into training data, in particular error reporting and function advice parts.
Feature extraction: key features are extracted from the feedback, such as error patterns, user behavior, etc.
Model updating: the integrated data and extracted features are used to update the existing model to improve its accuracy and performance.
And (3) verification and adjustment: the updated model is tested to verify its performance, with further adjustments made as necessary.
The resource optimization configuration module comprises a resource monitoring and configuration unit and an automatic resource scheduling unit; the resource monitoring and configuration unit is used for automatically adjusting the distribution of computing resources according to the monitoring data, the low-code platform can integrate a resource monitoring tool and monitor the use condition of system resources in real time, and as an implementation mode, the platform can monitor the CPU use rate and the memory occupation condition of the server.
The automatic resource scheduling unit is used for automatically adjusting the computing resource allocation of the analysis and judgment module, and the resource scheduling component on the platform can automatically adjust the computing resource allocation of the analysis and judgment module according to the monitoring data so as to ensure the performance. As one implementation, the platform may automatically allocate more computing resources to the anomaly detection module to speed up processing. The dynamic expansion and contraction capacity of the service is carried out by monitoring the IO, memory, CPU service condition and QPS (quality of service) values of the server.
Example 2
A working method of an intelligent log abnormality diagnosis system based on a low-code platform comprises the following steps:
on a low-code platform, a system administrator configures a data source, defines rules, performs data acquisition, and periodically acquires access log data from a website server;
and extracting keywords in the access log by using a feature extraction component, and selecting a random forest model for training to extract features. The configuration process comprises selecting a feature extraction algorithm and model parameters;
and configuring an anomaly detection component, embedding the trained model, and setting an alarm strategy. When an abnormal access mode exists, the system automatically sends out alarm notification, including abnormal description, scoring and possible reasons.
After receiving the alarm, the user can provide feedback information through the platform interface, confirm abnormal conditions or provide additional information. Feedback information is automatically recorded and used for model optimization.
The platform monitors the load condition of the server and automatically allocates more computing resources to the anomaly detection module so as to ensure analysis performance.
The present invention is not limited to the above-mentioned embodiments, and any equivalent embodiments which can be changed or modified by the technical content disclosed above can be applied to other fields, but any simple modification, equivalent changes and modification made to the above-mentioned embodiments according to the technical substance of the present invention will still fall within the protection scope of the technical solution of the present invention.
Claims (7)
1. An intelligent log anomaly diagnosis system based on a low-code platform, which is characterized by comprising: the system comprises a data acquisition and processing module, a feature extraction and learning module, an abnormality detection and alarm module, a user feedback and optimization module and a resource optimization configuration module;
the data acquisition and processing module is used for acquiring log data and removing repeated logs;
the feature extraction and learning module is used for extracting features of the processed log data and inputting the features into the model for training to obtain a trained model;
the abnormality detection and alarm module is used for configuring an abnormality judgment algorithm, embedding the trained model into the abnormality detection and alarm module to acquire abnormality detection information and carry out alarm notification when an abnormality access mode exists;
the user feedback and optimization module is used for receiving and recording user feedback information after the user receives the alarm notification;
the optimizing module and the resource optimizing configuration module are used for monitoring the load condition of the server and reasonably distributing the computing resources so as to ensure the analysis performance.
2. The intelligent log anomaly diagnostic system based on the low-code platform according to claim 1, wherein the data acquisition and processing module comprises an automatic data acquisition unit, a log data preprocessing unit, a data cleaning and filtering unit and a log data storage unit;
the automatic data acquisition unit is used for automatically acquiring log data of various types from different system components and devices, and a user defines a data source and acquisition rules through a configuration interface of the low-code platform;
the log data preprocessing unit is used for converting unstructured data into structured data, and the low-code platform provides a data preprocessing component for the unstructured log data so that a user can easily define rules;
the data cleaning and filtering unit is used for removing invalid log data, ensuring the quality of the analyzed data, utilizing a data cleaning component of the low-code platform, and defining data cleaning rules by a user;
the log data storage unit is used for storing and managing data, the processed log data can be automatically stored in the database, the low-code platform provides a database operation component, and a user configures the data storage mode and period.
3. The intelligent log anomaly diagnostic system based on the low-code platform according to claim 1, wherein the feature extraction and learning module comprises a feature extraction algorithm unit, a machine learning model configuration unit, a model training and optimization unit;
the feature extraction algorithm unit is used for extracting features, a user selects a feature extraction component, and different feature extraction algorithms are configured according to different log data types; these components are already packaged so that the user does not have to know the details of the algorithm;
the machine learning model configuration unit is used for generating a model training and evaluating flow by taking the characteristic extraction result as input; on a low-code platform, a user selects a machine learning model, configures parameters and takes a result of feature extraction as input by using a visual interface; the platform automatically generates a model training and evaluating flow;
the model training and optimizing unit is used for carrying out model training and optimizing, and the user automatically carries out model training and optimizing through the low-code platform; the platform can automatically generate a cross-validation and parameter adjustment flow, so that the training of the model is simplified.
4. The low-code platform-based intelligent log abnormality diagnosis system according to claim 1, wherein the abnormality detection and alarm module comprises an abnormality judgment algorithm configuration unit, an alarm policy visualization unit and a real-time alarm unit;
the anomaly judgment algorithm configuration unit is used for configuring an anomaly judgment algorithm, and a user configures the anomaly judgment algorithm through a low-code platform to embed the trained model into the anomaly judgment algorithm; the platform provides an interface for algorithm selection and parameter configuration, so that the configuration process is simplified;
the alarm strategy visualization unit is used for visually configuring the alarm strategy, and on the platform, the user visually configures the alarm strategy, including a notification mode and a receiver;
the real-time alarm unit is used for automatically sending out alarm notification according to the configured alarm strategy; when the system detects an abnormal condition, the low-code platform automatically sends out an alarm notification according to a configured alarm strategy, and the notification content comprises abnormal description, scoring and possible reasons.
5. The intelligent log anomaly diagnosis system based on the low-code platform according to claim 1, wherein the user feedback and optimization module comprises a user feedback interaction unit and a model optimization automation unit;
the user feedback interaction unit is used for providing feedback information, confirming abnormal conditions or providing additional information, and a user replies an alarm notification through an interface on the platform; the feedback information of the user is recorded and used for optimizing the subsequent model;
the model optimization automation unit is used for updating the model, improving the judgment accuracy, and the model optimization process is automated by utilizing the low-code platform; the feedback information of the user can automatically update the model, so that the judgment accuracy is improved.
6. The intelligent log anomaly diagnostic system based on the low-code platform according to claim 1, wherein the resource optimal configuration module comprises a resource monitoring and configuration unit and an automatic resource scheduling unit;
the resource monitoring and configuration unit is used for automatically adjusting the distribution of computing resources according to the monitoring data, and the low-code platform integrates a resource monitoring tool to monitor the use condition of system resources in real time;
the automatic resource scheduling unit is used for automatically adjusting the computing resource allocation of the analysis and judgment module, and the resource scheduling component on the platform automatically adjusts the computing resource allocation of the analysis and judgment module according to the monitoring data so as to ensure the performance.
7. A method of operating a low code platform based intelligent log anomaly diagnostic system as claimed in any one of claims 1 to 6, comprising: on a low-code platform, a system administrator configures a data source, defines rules, performs data acquisition, and periodically acquires access log data from a website server;
extracting keywords in the access log by using a feature extraction component, selecting a random forest model for training, and extracting features; the configuration process comprises selecting a feature extraction algorithm and model parameters;
configuring an anomaly detection component, embedding a trained model, and setting an alarm strategy; when an abnormal access mode exists, the system automatically sends out alarm notification, including abnormal description, scoring and possible reasons;
after receiving the alarm, the user provides feedback information through the platform interface to confirm abnormal conditions or provide additional information; the feedback information is automatically recorded and used for model optimization;
the platform monitors the load condition of the server and automatically allocates more computing resources to the anomaly detection module so as to ensure analysis performance.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410244864.1A CN117828515A (en) | 2024-03-05 | 2024-03-05 | Intelligent log abnormality diagnosis system and method based on low-code platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410244864.1A CN117828515A (en) | 2024-03-05 | 2024-03-05 | Intelligent log abnormality diagnosis system and method based on low-code platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117828515A true CN117828515A (en) | 2024-04-05 |
Family
ID=90521160
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410244864.1A Pending CN117828515A (en) | 2024-03-05 | 2024-03-05 | Intelligent log abnormality diagnosis system and method based on low-code platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117828515A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220405592A1 (en) * | 2022-03-10 | 2022-12-22 | University Of Electronic Science And Technology Of China | Multi-feature log anomaly detection method and system based on log full semantics |
| CN116627684A (en) * | 2023-03-24 | 2023-08-22 | 华东师范大学 | Log anomaly detection method and device for generating countermeasure network based on deep learning |
| CN117391214A (en) * | 2022-07-05 | 2024-01-12 | 华为云计算技术有限公司 | Model training method and device and related equipment |
| CN117454190A (en) * | 2023-09-11 | 2024-01-26 | 西安四叶草信息技术有限公司 | Log data analysis method and device |
| CN117596119A (en) * | 2023-12-22 | 2024-02-23 | 西藏腾狮软件有限公司 | Equipment data acquisition and monitoring method and system based on SNMP (simple network management protocol) |
-
2024
- 2024-03-05 CN CN202410244864.1A patent/CN117828515A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220405592A1 (en) * | 2022-03-10 | 2022-12-22 | University Of Electronic Science And Technology Of China | Multi-feature log anomaly detection method and system based on log full semantics |
| CN117391214A (en) * | 2022-07-05 | 2024-01-12 | 华为云计算技术有限公司 | Model training method and device and related equipment |
| CN116627684A (en) * | 2023-03-24 | 2023-08-22 | 华东师范大学 | Log anomaly detection method and device for generating countermeasure network based on deep learning |
| CN117454190A (en) * | 2023-09-11 | 2024-01-26 | 西安四叶草信息技术有限公司 | Log data analysis method and device |
| CN117596119A (en) * | 2023-12-22 | 2024-02-23 | 西藏腾狮软件有限公司 | Equipment data acquisition and monitoring method and system based on SNMP (simple network management protocol) |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111209131B (en) | Method and system for determining faults of heterogeneous system based on machine learning | |
| Debnath et al. | LogLens: A real-time log analysis system | |
| EP2487860B1 (en) | Method and system for improving security threats detection in communication networks | |
| US8621637B2 (en) | Systems, program product and methods for performing a risk assessment workflow process for plant networks and systems | |
| US8214364B2 (en) | Modeling user access to computer resources | |
| KR102118670B1 (en) | System and method for management of ict infra | |
| US6792456B1 (en) | Systems and methods for authoring and executing operational policies that use event rates | |
| US10692012B2 (en) | Classifying transactions at network accessible storage | |
| CN101321084A (en) | Method and apparatus for generating configuration rules for computing entities within a computing environment using association rule mining | |
| EP1058886A1 (en) | System and method for optimizing performance monitoring of complex information technology systems | |
| CN115809183A (en) | Method for discovering and disposing information-creating terminal fault based on knowledge graph | |
| CN110971464A (en) | Operation and maintenance automatic system suitable for disaster recovery center | |
| CN111181775B (en) | Integrated operation and maintenance management alarm method based on automatic host asset discovery | |
| CN110363381B (en) | Information processing method and device | |
| CN116112194A (en) | User behavior analysis method and device, electronic equipment and computer storage medium | |
| CN117742241B (en) | Data acquisition method and system based on cloud computing | |
| US11822578B2 (en) | Matching machine generated data entries to pattern clusters | |
| CN116895046B (en) | Abnormal operation and maintenance data processing method based on virtualization | |
| CN112039907A (en) | Automatic testing method and system based on Internet of things terminal evaluation platform | |
| Li et al. | An integrated data-driven framework for computing system management | |
| CN115794479A (en) | Log data processing method and device, electronic equipment and storage medium | |
| CN117828515A (en) | Intelligent log abnormality diagnosis system and method based on low-code platform | |
| CN115062144A (en) | Log anomaly detection method and system based on knowledge base and integrated learning | |
| CN112214278A (en) | Method for identifying and processing private cloud zombie virtual machine | |
| CN114945919A (en) | Abnormality detection device, abnormality detection method, and abnormality detection program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |