CN111475380A - Log analysis method and device - Google Patents
Log analysis method and device Download PDFInfo
- Publication number
- CN111475380A CN111475380A CN202010254054.6A CN202010254054A CN111475380A CN 111475380 A CN111475380 A CN 111475380A CN 202010254054 A CN202010254054 A CN 202010254054A CN 111475380 A CN111475380 A CN 111475380A
- Authority
- CN
- China
- Prior art keywords
- log
- analyzed
- analysis
- rule
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 94
- 238000000034 method Methods 0.000 claims abstract description 39
- 238000012545 processing Methods 0.000 claims abstract description 39
- 238000013507 mapping Methods 0.000 claims abstract description 16
- 238000003860 storage Methods 0.000 claims description 10
- 238000004422 calculation algorithm Methods 0.000 claims description 9
- 238000005520 cutting process Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 6
- 238000007635 classification algorithm Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 5
- 238000005094 computer simulation Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 12
- 238000005516 engineering process Methods 0.000 abstract description 8
- 239000000284 extract Substances 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 9
- 238000004519 manufacturing process Methods 0.000 description 3
- 239000000203 mixture Substances 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000010606 normalization Methods 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000012098 association analyses Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000003064 k means clustering Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The utility model provides a log analysis method, which comprises the steps of extracting original log information from a system log as a log to be analyzed, and processing the log to be analyzed according to a preset rule; mapping the processed log to be analyzed into a uniform paradigm expression; and generating a judgment analysis result for the log to be analyzed which is expressed by the normal form through a preset judgment rule. The method increases the adaptability of log normal-form recognition by adopting an innovative technology, can automatically recognize and process variant log formats with small differences, can automatically finish normal-form processing of specific log formats without manual intervention, extracts effective fields and assigns the effective fields to specified fields, is convenient for subsequent processing and analysis of a system, and has feasibility and usability of analysis. The present disclosure also provides a log analysis device.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a log analysis method and apparatus.
Background
The log is data generated by the IT system, records the running states of equipment, an operating system and application software, and is an important basis for daily operation and maintenance troubleshooting and attack tracing analysis of IT operation and maintenance managers and safety managers. Log data are ubiquitous and they reflect the real conditions inherent in the information system at all times. Therefore, analysis and management of logs is an efficient and important technical means for security administrators to solve problems, and is becoming increasingly important. It has several significant features: the source is very wide, and the system not only comprises logs and alarms generated by each device and system in the traditional IT environment, but also comprises a large amount of logs generated by a mobile client and a sensor; the quantity is huge, and various machines and systems generate data all the time, and the data are collected to be astronomical quantity; the formats generated by various systems are different, the meanings are different, and the requirements on professional skills of personnel are different; the storage is distributed, different logs are stored in different systems and devices, and different methods are needed for collection and reading.
In addition, the existing scheme mainly carries out classification and identification on specific logs. And (4) strictly establishing classification according to formats, matching corresponding data field processing paradigm rules, extracting and assigning fields to the specified fields, and finishing the extraction of data. The main problem of the technical scheme is that the scheme has poor suitability, the log format is required to be well defined in advance, and each log format is adapted. If the system of the user modifies the format of the log, the format error of the new log can be identified, and the identification of the whole log is failed. Background service personnel are required to establish a core data format matching rule for solving the problem. The whole process is usually supported by plant technicians, time and labor are wasted, the period is long, and the user experience is poor.
Disclosure of Invention
In order to solve the technical problems in the prior art, the embodiment of the disclosure provides a log analysis method and a log analysis device, and by adopting an innovative technology, the adaptability of log normal form identification is improved, variant log formats with smaller differences can be automatically identified and processed, normal form of a specific log format can be automatically completed without manual intervention, effective fields are extracted and assigned to designated fields, the subsequent processing and analysis of a system are facilitated, and the log analysis method and the log analysis device have feasibility and usability of analysis.
In a first aspect, an embodiment of the present disclosure provides a log analysis method, where the method includes: extracting original log information from the system log as a log to be analyzed, and processing the log to be analyzed according to a preset rule; mapping the processed log to be analyzed into a uniform paradigm expression; and generating a judgment analysis result for the log to be analyzed which is expressed by the normal form through a preset judgment rule.
In one embodiment, the method further comprises the following steps: and storing the generated judgment analysis result into a log analysis result library.
In one embodiment, the system log includes a system log, an application log, and a security log.
In one embodiment, the processing the log to be analyzed according to the preset rule includes: and sequentially cutting, dividing and classifying the logs to be analyzed, and extracting a data format suitable for log analysis.
In one embodiment, the method further comprises the following steps: and analyzing and cutting the log to be analyzed expressed in the normal form through a calculation model to extract log information.
In one embodiment, the preset determination rule is a rule defined on the basis of a predicate set composed of predicates corresponding to the log paradigm elements, and the condition of the log paradigm elements is determined to be described and processed through uniformly defined predicates.
In one embodiment, the computational model is a new algorithm generated by a combination of a clustering algorithm and a classification algorithm.
In a second aspect, the disclosed embodiments provide a computer-readable storage medium, on which a computer program is stored, which when executed by a processor, implements the steps of the method described above.
In a third aspect, the disclosed embodiments provide a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method described above when executing the program.
In a fourth aspect, an embodiment of the present disclosure provides a log analysis apparatus, where the apparatus includes: the extraction and processing module is used for extracting original log information from the system logs as logs to be analyzed and processing the logs to be analyzed according to preset rules; the mapping module is used for mapping the processed log to be analyzed into a uniform paradigm expression; and the analysis module is used for generating a judgment analysis result for the log to be analyzed which is expressed by the normal form through a preset judgment rule.
According to the log analysis method and device provided by the invention, original log information is extracted from a system log to be used as a log to be analyzed, and the log to be analyzed is processed according to a preset rule; mapping the processed log to be analyzed into a uniform paradigm expression; and generating a judgment analysis result for the log to be analyzed which is expressed by the normal form through a preset judgment rule. The method increases the adaptability of log normal-form recognition by adopting an innovative technology, can automatically recognize and process variant log formats with small differences, can automatically finish normal-form processing of specific log formats without manual intervention, extracts effective fields and assigns the effective fields to specified fields, is convenient for subsequent processing and analysis of a system, and has feasibility and usability of analysis.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed to be used in the description of the embodiments are briefly introduced as follows:
FIG. 1 is a flow chart illustrating steps of a log analysis method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a log analysis method according to another embodiment of the present invention;
FIG. 3 is an exemplary diagram of a log analysis method in a further embodiment of the invention;
FIG. 4 is a schematic structural diagram of a log analysis apparatus according to an embodiment of the present invention;
FIG. 5 is a hardware block diagram of a log analysis device according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a computer-readable storage medium in one embodiment of the invention.
Detailed Description
The present application will now be described in further detail with reference to the accompanying drawings and examples.
In the following description, the terms "first" and "second" are used for descriptive purposes only and are not intended to indicate or imply relative importance. The following description provides embodiments of the disclosure, which may be combined or substituted for various embodiments, and this application is therefore intended to cover all possible combinations of the same and/or different embodiments described. Thus, if one embodiment includes feature A, B, C and another embodiment includes feature B, D, then this application should also be considered to include an embodiment that includes one or more of all other possible combinations of A, B, C, D, even though this embodiment may not be explicitly recited in text below.
In order to make the objects, technical solutions and advantages of the present invention more clearly apparent, the following describes in detail a specific implementation of the log analysis method and apparatus according to the present invention by way of example with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Example 1
As shown in fig. 1, a schematic flow chart of a log analysis method in an embodiment specifically includes the following steps:
and 11, extracting original log information from the system log as a log to be analyzed, and processing the log to be analyzed according to a preset rule. Wherein the system log comprises a system log, an application log and a security log. Therefore, the diversity of the system logs is improved.
Specifically, the processing the log to be analyzed according to the preset rule includes: and sequentially cutting, dividing and classifying the logs to be analyzed, and extracting a data format suitable for log analysis. Therefore, the usability and the rapidness of subsequent log analysis are improved.
And step 12, mapping the processed log to be analyzed into a unified paradigm expression.
And step 13, generating a judgment analysis result for the log to be analyzed which is expressed by the paradigm through a preset judgment rule. The preset judgment rule is defined on the basis of a predicate set consisting of predicates corresponding to the log paradigm elements, and the condition judgment of the log paradigm elements is that description and processing are performed through the uniformly defined predicates. Therefore, the accuracy, flexibility and usability of log analysis are improved.
In this embodiment, original log information is extracted from a system log as a log to be analyzed, and the log to be analyzed is processed according to a preset rule; mapping the processed log to be analyzed into a uniform paradigm expression; and generating a judgment analysis result for the log to be analyzed which is expressed by the normal form through a preset judgment rule. The method increases the adaptability of log normal-form recognition by adopting an innovative technology, can automatically recognize and process variant log formats with small differences, can automatically finish normal-form processing of specific log formats without manual intervention, extracts effective fields and assigns the effective fields to specified fields, is convenient for subsequent processing and analysis of a system, and has feasibility and usability of analysis.
Example 2
As shown in fig. 2, a schematic flow chart of a log analysis method in another embodiment specifically includes the following steps:
and step 21, extracting original log information from the system log as a log to be analyzed, and processing the log to be analyzed according to a preset rule. Wherein the system log comprises a system log, an application log and a security log. Therefore, the diversity of the system logs is improved.
Specifically, the processing the log to be analyzed according to the preset rule includes: and sequentially cutting, dividing and classifying the logs to be analyzed, and extracting a data format suitable for log analysis. Therefore, the usability and the rapidness of subsequent log analysis are improved.
And step 22, mapping the processed log to be analyzed into a unified paradigm expression.
And step 23, analyzing and cutting the log to be analyzed expressed in the normal form through a calculation model to extract log information. The calculation model is a new algorithm generated by combining a clustering algorithm and a classification algorithm. Therefore, the accuracy of log analysis is improved.
And 24, generating a judgment analysis result for the log to be analyzed which is expressed by the paradigm through a preset judgment rule. The preset judgment rule is defined on the basis of a predicate set consisting of predicates corresponding to the log paradigm elements, and the condition judgment of the log paradigm elements is that description and processing are performed through the uniformly defined predicates. Therefore, the accuracy, flexibility and usability of log analysis are improved.
And step 25, storing the generated judgment analysis result into a log analysis result library.
In this embodiment, a computing model is added to analyze and crop the log to be analyzed expressed in a normal form to extract log information; and an implementation step of storing the generated judgment analysis result in a log analysis result library. Therefore, the accuracy, the real-time performance and the usability of log analysis are improved.
Example 3
Fig. 3 is a schematic flow chart of a log analysis method in another embodiment.
It should be noted that, the log analysis method disclosed by the present disclosure adopts a new technology to collect, analyze and store mass log data, so as to realize equipment log collection, performance operation monitoring, policy configuration, accurately describe security event rules and various monitoring queries, perform association analysis on logs through machine learning, automatically analyze various complex strange logs through an optimized algorithm, extract key fields in an automatic paradigm, reduce the workload of developing log processing programs one by one, and integrate log information from different systems, the log information of the system is information of recording hardware, software and system problems in the system, and simultaneously, events occurring in the system can be monitored. Through which the user can check the cause of the error or look for traces left by the attacker when under attack. The system log includes a system log, an application log, and a security log.
Different system logs are that the information system comprises hardware such as a server, a network, a pc, a storage and the like, software such as an operating system, an application program, middleware, a database and the like, and the system information generated by the software is summarized in a log form through a time stamp. So as to facilitate the auditing, monitoring, analyzing and predicting of the information system. Has wide adaptability and flexibility. The state analysis is carried out on the service system by combining the asset data, the problem of analyzing the log is solved, the log can be checked and analyzed when the log is obtained, and the current situation that the log data is difficult to analyze and is subjected to paradigm management in a manual error correction mode and only can be analyzed afterwards is changed.
Specifically, the original information is processed under a certain rule, the required elements are extracted, the irrelevant components are filtered out, and the mapping dimension is expressed in a unified normal form. The log processing is to cut, divide, classify and extract a data format which is beneficial to log analysis from a complex, long, unintelligible and irregular log.
Data formats are exemplified:
{"Url":"item.jd.com/11381983.html","EndDate":"2018-04-25T13:46:50.345631+08:00","FieldValueDic":{"IsDeleted":"False","AF1":"9787543699762"}}
{"Url":"item.jd.com/11381983.html","EndDate":"2018-04-25T20:46:50.565631+08:00","FieldValueDic":{"IsDeleted":"False","AF1":"66666"}}
the difference between the two data is: the value of the AF1 field obtains different results at different times. Target of data processing: in the same url, the content of each field is updated in real time.
Target result after data processing:
{"Url":"item.jd.com/11381983.html","IsDeleted":"False","AF1":"66666"}
in addition, the definition of an element is a log system configuration file (defining some rules of the log):
logging.basicConfig(filename＝'test.log',level＝logging.INFO,filemode＝'a', format＝'％(levelname)s:％(asctime)s:％(message)s')
wherein, filename: into which file the journal is to be saved (defining that this post-journal will not be printed on the screen); level: what level of log above needs to be saved; filemode: the method has two modes of 'w' and 'a', and is similar to open, added with 'a' and covered with 'w'; and (3) format: defining a log format (a log element table is provided later, and various log elements are connected into a reasonable log format through:); the format defines a log element table used in the format.
It should be noted that, there are many information systems, the log format of each system is different, and the log elements may be attributes that use the logs of various systems as categories.
Furthermore, it should be noted that the paradigm processing is adopted, which reduces the workload of developing log processing programs one by one, and can integrate information from different logs, thereby having wide adaptability and strong flexibility. The decision rules are defined in terms of dimensional decision conditions and corresponding logical conclusions, and decision conditions are typically defined for event elements (events, locations, etc.) and their synthetic relationships. Since all logs are mapped to a constant paradigm expression, conditional decisions on elements can be described and processed with uniformly defined predicates. And because the elements of the log normal form are relatively fixed, the predicates form a relatively determined predicate set, a rule normal form can be defined on the basis of the sub-predicate set, and a normal form with a wider application range is obtained. Through the optimized calculation model, various heterogeneous logs can be accurately analyzed and cut, effective log information is extracted, the trained structure is corrected again through a manual intervention mode, and finally, the algorithm can quickly process logs of products of mainstream types such as application, safety, network, middleware, databases and the like. The log format of the strange application system can be solved without secondary development.
Furthermore, it should be noted that the calculation model is a combination of a clustering algorithm and a classification algorithm. The method specifically comprises the following steps:
where EO is the number of operations from S1 to S2, xi is the index of the word at the i-th operation, and v is the hyperparameter controlling the weight.
For two different logs, their weighted edit distance is calculated first, and if less than a threshold, there is a link between them, and we divide all logs with links between them into a group logroup. A method for determining a threshold value is provided, by calculating weighted edit distances between each pair of messages, a set of weighted edit distances of all pairs in a training set can be obtained, k-means clustering is carried out on the weighted edit distances, k is selected to be 2, and the maximum weighted distance in a class with a smaller class center value is selected to be used as the threshold value. The logs of the same system can be classified into logs of the same system type from the clustering algorithm.
Different log contents are generated due to different operations at different times and different places of the same system. For a certain category, for each row of specific logs we compare with the similarity of each row of specific logs of the final category array of the category:
if the similarity of the character strings with a certain row of specific logs in the final category exceeds a threshold value, the two character strings are classified into one category, only the time point of the specific log to be analyzed is stored in the category, and the analysis of the row of logs is stopped.
If the similarity with the string of any row of the specific log in the final category is below the threshold. We find a new category. A row of records is added to the final category.
The first step is to find the log type of the same type through clustering; the second step finds different contents of the same system similar to the log by classification. The classification algorithm introduces an artificial neuron network to normalize the log. The method comprises the steps of continuously detecting the head of a log cache queue in an independent thread operation mode, taking out data from the cache queue if new data exists in the log cache queue, normalizing log contents according to field description information of a log normalization configuration file, and calling a log storage module to store logs into a uniform format. The learning model can automatically complete, map and assign values to the original log, and the automatic normalization of the whole log is completed.
In summary, it can be seen that the log analysis method disclosed by the present disclosure adds a learning model to analyze and process the log format, can automatically adjust and adapt to the log subjected to small change, realizes automatic log identification and automatic normalization, reduces manual intervention, and improves the efficiency of system processing.
Based on the same inventive concept, a log analysis device is also provided. Because the principle of the device for solving the problems is similar to that of the log analysis method, the implementation of the device can be realized according to the specific steps of the method, and repeated parts are not repeated.
Fig. 4 is a schematic structural diagram of a log analysis apparatus in an embodiment. The log analysis device 10 includes: an extraction and processing module 100, a mapping module and an analysis module 300.
The extraction and processing module 100 is configured to extract original log information from a system log as a log to be analyzed, and process the log to be analyzed according to a preset rule; the mapping module 200 is configured to map the processed log to be analyzed into a uniform paradigm expression; the analysis module 300 is configured to generate a judgment analysis result for the to-be-analyzed log expressed by the paradigm through a preset judgment rule.
In this embodiment, the extraction and processing module extracts original log information from the system log as a log to be analyzed, and processes the log to be analyzed according to a preset rule; mapping the processed log to be analyzed into a uniform paradigm expression through a mapping module; and finally, generating a judgment analysis result for the log to be analyzed expressed by the normal form through a preset judgment rule by an analysis module. The device increases the adaptability of log normal-form recognition by adopting an innovative technology, can automatically recognize and process variant log formats with smaller differences, can automatically complete normal-form processing of specific log formats without manual intervention, extracts effective fields and assigns the effective fields to designated fields, facilitates subsequent processing and analysis of a system, and has feasibility and usability of analysis.
Fig. 5 is a hardware block diagram illustrating a log analysis apparatus according to an embodiment of the present disclosure. As shown in fig. 5, a log analysis device 50 according to an embodiment of the present disclosure includes a memory 501 and a processor 502. The components of a log analysis device 50 are interconnected by a bus system and/or other form of connection mechanism (not shown).
The memory 501 is used to store non-transitory computer readable instructions. In particular, memory 501 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory can include, for example, Random Access Memory (RAM), cache memory (or the like). The non-volatile memory may include, for example, Read Only Memory (ROM), a hard disk, flash memory, and the like.
The processor 502 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in one type of log analysis device 50 to perform desired functions. In one embodiment of the present disclosure, the processor 502 is configured to execute computer readable instructions stored in the memory 501, so that a log analysis apparatus 50 executes a log analysis method as described above. A log analyzing apparatus is the same as the embodiment described in the above-described one log analyzing method, and a repetitive description thereof will be omitted herein.
Fig. 6 is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the present disclosure. As shown in fig. 6, a computer-readable storage medium 600 according to embodiments of the present disclosure has non-transitory computer-readable instructions 601 stored thereon. The non-transitory computer readable instructions 601, when executed by a processor, perform a method of log analysis according to embodiments of the present disclosure described above with reference to the foregoing description.
In the above, according to the log analysis method and apparatus and the computer-readable storage medium of the embodiments of the present disclosure, by adopting the innovative technology, the adaptability of log paradigm recognition is increased, variant log formats with smaller differences can be automatically recognized and processed, the paradigm of a specific log format can be automatically completed without manual intervention, valid fields are extracted and assigned to designated fields, the subsequent processing and analysis of a system are facilitated, and the method and apparatus have the beneficial effects of feasibility and usability of analysis.
The foregoing describes the general principles of the present disclosure in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present disclosure are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present disclosure. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the disclosure is not intended to be limited to the specific details so described.
The block diagrams of devices, apparatuses, systems referred to in this disclosure are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
Also, as used herein, "or" as used in a list of items beginning with "at least one" indicates a separate list, such that, for example, a list of "A, B or at least one of C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the described example is preferred or better than other examples.
It is also noted that in the systems and methods of the present disclosure, components or steps may be decomposed and/or re-combined. These decompositions and/or recombinations are to be considered equivalents of the present disclosure.
Various changes, substitutions and alterations to the techniques described herein may be made without departing from the techniques of the teachings as defined by the appended claims. Moreover, the scope of the claims of the present disclosure is not limited to the particular aspects of the process, machine, manufacture, composition of matter, means, methods and acts described above. Processes, machines, manufacture, compositions of matter, means, methods, or acts, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein may be utilized. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or acts.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the disclosure to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.
Claims (10)
1. A method of log analysis, the method comprising:
extracting original log information from the system log as a log to be analyzed, and processing the log to be analyzed according to a preset rule;
mapping the processed log to be analyzed into a uniform paradigm expression;
and generating a judgment analysis result for the log to be analyzed which is expressed by the normal form through a preset judgment rule.
2. The log analysis method of claim 1, further comprising: and storing the generated judgment analysis result into a log analysis result library.
3. A log analysis method as claimed in claim 1, wherein the system log comprises a system log, an application log and a security log.
4. The log analysis method according to claim 1, wherein the processing the log to be analyzed according to the preset rule comprises: and sequentially cutting, dividing and classifying the logs to be analyzed, and extracting a data format suitable for log analysis.
5. The log analysis method of claim 1, further comprising: and analyzing and cutting the log to be analyzed expressed in the normal form through a calculation model to extract log information.
6. The log analysis method according to claim 1, wherein the preset determination rule is a rule defined on the basis of a predicate set composed of predicates corresponding to the log paradigm elements, and the condition determination of the log paradigm elements is described and processed by uniformly defined predicates.
7. A log analysis method as claimed in claim 5, wherein the computational model is a new algorithm generated by a combination of clustering and classification algorithms.
8. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1-7 are implemented when the program is executed by the processor.
10. An apparatus for log analysis, the apparatus comprising:
the extraction and processing module is used for extracting original log information from the system logs as logs to be analyzed and processing the logs to be analyzed according to preset rules;
the mapping module is used for mapping the processed log to be analyzed into a uniform paradigm expression;
and the analysis module is used for generating a judgment analysis result for the log to be analyzed which is expressed by the normal form through a preset judgment rule.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010254054.6A CN111475380B (en) | 2020-04-02 | 2020-04-02 | Log analysis method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010254054.6A CN111475380B (en) | 2020-04-02 | 2020-04-02 | Log analysis method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111475380A true CN111475380A (en) | 2020-07-31 |
| CN111475380B CN111475380B (en) | 2024-03-12 |
Family
ID=71750377
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010254054.6A Active CN111475380B (en) | 2020-04-02 | 2020-04-02 | Log analysis method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111475380B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114186227A (en) * | 2021-12-08 | 2022-03-15 | 上海观安信息技术股份有限公司 | Method, device and storage medium for converting safety alarm into safety event |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017094262A1 (en) * | 2015-11-30 | 2017-06-08 | 日本電気株式会社 | Log analysis system, method, and program |
| CN109324996A (en) * | 2018-10-12 | 2019-02-12 | 平安科技（深圳）有限公司 | Journal file processing method, device, computer equipment and storage medium |
| CN109902072A (en) * | 2019-02-21 | 2019-06-18 | 云南电网有限责任公司红河供电局 | A kind of log processing system |
-
2020
- 2020-04-02 CN CN202010254054.6A patent/CN111475380B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017094262A1 (en) * | 2015-11-30 | 2017-06-08 | 日本電気株式会社 | Log analysis system, method, and program |
| CN109324996A (en) * | 2018-10-12 | 2019-02-12 | 平安科技（深圳）有限公司 | Journal file processing method, device, computer equipment and storage medium |
| CN109902072A (en) * | 2019-02-21 | 2019-06-18 | 云南电网有限责任公司红河供电局 | A kind of log processing system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114186227A (en) * | 2021-12-08 | 2022-03-15 | 上海观安信息技术股份有限公司 | Method, device and storage medium for converting safety alarm into safety event |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111475380B (en) | 2024-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Deshpande et al. | HIDS: A host based intrusion detection system for cloud computing environment | |
| CA2933423A1 (en) | Data acceleration | |
| CN111177714A (en) | Abnormal behavior detection method and device, computer equipment and storage medium | |
| CN111538741B (en) | Deep learning analysis method and system for big data of alarm condition | |
| CN110855648B (en) | Early warning control method and device for network attack | |
| Jiang et al. | A family of joint sparse PCA algorithms for anomaly localization in network data streams | |
| CN116662989B (en) | Security data analysis method and system | |
| CN113486983A (en) | Big data office information analysis method and system for anti-fraud processing | |
| CN115879017A (en) | Automatic classification and grading method and device for power sensitive data and storage medium | |
| CN112988509A (en) | Alarm message filtering method and device, electronic equipment and storage medium | |
| RU148692U1 (en) | COMPUTER SECURITY EVENTS MONITORING SYSTEM | |
| KR102189127B1 (en) | A unit and method for processing rule based action | |
| CN112583847B (en) | Method for network security event complex analysis for medium and small enterprises | |
| RU180789U1 (en) | DEVICE OF INFORMATION SECURITY AUDIT IN AUTOMATED SYSTEMS | |
| CN111475380A (en) | Log analysis method and device | |
| CN117172093A (en) | Method and device for optimizing strategy of Linux system kernel configuration based on machine learning | |
| CN111352820A (en) | Method, equipment and device for predicting and monitoring running state of high-performance application | |
| CN116599743A (en) | 4A abnormal detour detection method and device, electronic equipment and storage medium | |
| CN115168848B (en) | Interception feedback processing method based on big data analysis interception | |
| CN116707859A (en) | Feature rule extraction method and device, and network intrusion detection method and device | |
| US11822578B2 (en) | Matching machine generated data entries to pattern clusters | |
| CN114218569A (en) | Data analysis method, device, equipment, medium and product | |
| KR20210142443A (en) | Method and system for providing continuous adaptive learning over time for real time attack detection in cyberspace | |
| CN117540372B (en) | Database intrusion detection and response system for intelligent learning | |
| CN111930545B (en) | SQL script processing method, SQL script processing device and SQL script processing server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |