CN113535667A - Method, device and system for automatically analyzing system logs - Google Patents
Method, device and system for automatically analyzing system logs Download PDFInfo
- Publication number
- CN113535667A CN113535667A CN202010313503.XA CN202010313503A CN113535667A CN 113535667 A CN113535667 A CN 113535667A CN 202010313503 A CN202010313503 A CN 202010313503A CN 113535667 A CN113535667 A CN 113535667A
- Authority
- CN
- China
- Prior art keywords
- log
- link
- logs
- evaluated
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000004458 analytical method Methods 0.000 claims abstract description 98
- 238000011156 evaluation Methods 0.000 claims abstract description 46
- 238000012545 processing Methods 0.000 claims abstract description 25
- 230000008569 process Effects 0.000 claims abstract description 23
- 230000009467 reduction Effects 0.000 claims abstract description 12
- 238000004519 manufacturing process Methods 0.000 claims description 36
- 239000013598 vector Substances 0.000 claims description 34
- 238000013210 evaluation model Methods 0.000 claims description 27
- 238000012827 research and development Methods 0.000 claims description 16
- 230000015654 memory Effects 0.000 claims description 15
- 238000004422 calculation algorithm Methods 0.000 claims description 7
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 6
- 230000006835 compression Effects 0.000 claims description 5
- 238000007906 compression Methods 0.000 claims description 5
- 239000000284 extract Substances 0.000 claims description 4
- 238000013473 artificial intelligence Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000009826 distribution Methods 0.000 claims description 3
- 239000003550 marker Substances 0.000 claims description 3
- 238000011835 investigation Methods 0.000 claims description 2
- 230000008676 import Effects 0.000 claims 1
- 238000011946 reduction process Methods 0.000 claims 1
- 230000008859 change Effects 0.000 abstract description 4
- 238000007726 management method Methods 0.000 description 9
- 238000012423 maintenance Methods 0.000 description 8
- 238000003860 storage Methods 0.000 description 6
- 230000009471 action Effects 0.000 description 5
- 238000012356 Product development Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000015572 biosynthetic process Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 239000012141 concentrate Substances 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000012098 association analyses Methods 0.000 description 1
- 238000007621 cluster analysis Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000000513 principal component analysis Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to the technical field of log analysis, and provides a method, a device and a system for automatically analyzing system logs. The method comprises automatically collecting logs of one or more target systems; extracting key elements of the log, and performing dimension reduction processing on the key feature dimensions of the log when the key feature dimensions of the log, which are used for representing the number of the key feature elements of the log, exceed a first preset threshold; outputting one or more models to be evaluated according to the content of the key characteristic elements of the log; and finishing the evaluation classification of the model to be evaluated by combining a big data resource pool and/or an expert team formed by historical evaluated models. The invention provides a log characteristic dimension reduction processing means, so that the whole analysis process is not redundant and uncontrollable due to the change of log characteristic dimensions of different links.
Description
[ technical field ] A method for producing a semiconductor device
The invention relates to the technical field of log analysis, in particular to a method, a device and a system for automatically analyzing system logs.
[ background of the invention ]
The traditional software delivery problem processing link is as follows: research and development links, sales, customer service and after sales, wherein each link may have certain problems.
In the research and development link, because of different company management levels and different research and development levels, good logs made in the research and development link have strict standard requirements, poor basic individuals can freely play, the same logs can be restrained even if basic formats, but the degree of freedom of specific contents is high, once some key information is not output, the problem can be very difficult to be checked, the log contents can only be adjusted in a software upgrading mode, and the cost is huge for enterprise-level software.
Generally, the management level of an enterprise is limited, log analysis in a customer service link and an after-sales link depends on more manpower, seamless training is needed along with software updating, the labor cost for 24-hour online assistance is high, complex problems can only be transferred to research and development, high communication cost exists, problem analysis efficiency is difficult to improve, and meanwhile, a company is forced to increase research and development resources in a maintenance stage.
If the application range of enterprise products is increased, even the products spread out in and out of the sea, the scale of customer service and after sales is expanded, and the related cost far exceeds the product development. In response to the above-mentioned complications, an effective solution is lacking to improve log management in such situations.
[ summary of the invention ]
The invention aims to solve the technical problem of how to improve log analysis under complex conditions, and can lead software developers to focus on software core service logic so as not to be disturbed by fussy system log output logic.
In a first aspect, the present invention provides a method for automatically analyzing a system log, including:
automatically collecting logs of one or more target systems;
extracting key elements of the log, and performing dimension reduction processing on the key feature dimensions of the log when the key feature dimensions of the log, which are used for representing the number of the key feature elements of the log, exceed a first preset threshold;
outputting one or more models to be evaluated according to the content of the key characteristic elements of the log;
and finishing the evaluation classification of the model to be evaluated by combining a big data resource pool and/or an expert team formed by historical evaluated models.
Preferably, the completing the evaluation classification of the model to be evaluated in combination with a big data resource pool and/or an expert team formed by historical evaluated models specifically includes:
and performing similarity matching between the historical evaluated model and the model to be evaluated in a big data resource pool formed by the historical evaluated model, and taking the evaluation classification of the corresponding evaluated model as the evaluation classification of the model to be evaluated if the matching similarity exceeds a second preset threshold.
Preferably, the completing the evaluation classification of the model to be evaluated in combination with a big data resource pool and/or an expert team formed by historical evaluated models specifically includes:
similarity matching between the historical evaluated model and the model to be evaluated is carried out in a big data resource pool formed by the historical evaluated model, and if the historical evaluated model with the similarity exceeding a second preset threshold value is not found, evaluation classification of the expert team on the model to be evaluated is obtained;
and if the evaluation classification given by the expert team is different from the evaluation classification given by the historical evaluated model, adding the evaluation classification given by the model to be evaluated and the expert team as a new historical evaluated model into the big data resource pool.
Preferably, the automatically collecting logs of one or more target systems specifically includes:
a log system is arranged aiming at one or more links of a research and development link, a production link, a sales link, a customer service link and an after-sales link;
and taking one or more log systems corresponding to the research and development link, the production link, the sales link, the customer service link and the after-sales link as target systems, and collecting logs in the corresponding target systems.
Preferably, the extracting of the log key features and the performing of the dimensionality reduction processing of the log feature dimensions when the log feature dimensions used for representing the number of the log key feature features exceed a first preset threshold specifically include:
the log key elements are composed of various contents contained in the log, and if the log feature dimension composed of the number of the log key feature elements in the first link exceeds a first preset threshold value;
confirming that the first link can be used for associating a plurality of log key feature elements aiming at the same target object in other links; compressing the plurality of key characteristic elements of the log according to the incidence relation between the first link and other links; and/or the presence of a gas in the gas,
confirming a plurality of log key feature elements in the first link, and compressing the plurality of log key feature elements when the plurality of log key feature elements are used for representing the same meaning in a historical use record; and/or the presence of a gas in the gas,
confirming one or more log key feature elements in the first link, and compressing one or more log key feature elements when the utilization rate in the historical utilization record is lower than a third preset threshold and no substantial correlation exists between the utilization content and the final result; and/or the presence of a gas in the gas,
confirming one or more log key feature elements in the first link, wherein the respective contents of the one or more log key feature elements are parameter objects with specified quantity; and performing merging parameter object type compression processing on the one or more log key feature elements.
Preferably, the compressing the plurality of key feature elements of the log according to the association relationship between the first link and each of the other links specifically includes:
the first link is respectively associated with a second link through a first log key feature element, and is associated with a third link through a second log key feature element;
if the second link and the third link are also related through a third log key feature element, selecting one of the first log key feature element and the second log key feature element;
and generating an association relation among the first link log, the second link log and the third link log according to the selected first log key feature element, the selected second log key feature element and the selected third log key feature element.
Preferably, when the model to be evaluated generated by the log in each link is performed on a specific product object and the corresponding evaluation result is the first fault type, the analysis method further includes:
according to the logs of the production links in the model to be evaluated of the product objects, the logs of other product objects in the same batch or the same production line are searched and are used as corresponding logs of the potential first fault type for sorting;
obtaining a related consumer contact way according to the key characteristic elements of the log recorded in the sales link in the potential first fault type log arrangement result;
and the customer service triggering link carries out pre-attention and/or investigation on the related first fault type according to the contact way of the consumer.
Preferably, according to the distribution of the associated logs in the evaluation model of fault evaluation classification, determining the acquisition periods of the logs in different links;
and each link has respective log generation speed, and the log of each link realizes the establishment of the incidence relation through the evaluation model.
The second invention also provides an automatic system log analysis system, which comprises:
an independent log collection unit is constructed in the online production system, and logs of a target system are automatically collected and uploaded to an offline analysis server;
the offline analysis server performs data preprocessing, extracts key elements of the logs and reduces characteristic dimensions of the logs, so that subsequent intelligent analysis engines can conveniently cluster the logs according to faults;
the offline analysis server starts an intelligent analysis engine, and manually marks faults according to actual conditions by matching with an expert team through analyzing key characteristic elements of the logs, so as to output an evaluation model;
an intelligent analysis engine is deployed in an online production system, an evaluation model output offline is imported, and output faults are calculated through real-time monitoring logs.
Preferably, the online production system constructs an independent log collection unit, automatically collects the log of the target system, and uploads the log to the offline analysis server, and specifically includes:
an independent log acquisition unit is constructed, and automatic timing log copying and transmission are realized by configuring a target log path and a log uploading path;
one or more log acquisition units are deployed in a centralized or distributed deployment mode matched with the characteristics of the target software system;
and starting a log acquisition unit, and extracting the log to a specified server path in real time.
Preferably, the offline analysis server performs data preprocessing, extracts key elements of the log and reduces characteristic dimensionality of the log, so that a subsequent intelligent analysis engine can cluster the log according to faults, and the method specifically comprises the following steps:
constructing a data preprocessor, loading log data collected by a log collection unit, and classifying according to different log entry filling modules;
the data preprocessor groups the slices according to time elements generated by the logs, and gathers the logs of a plurality of log item filling modules in the same time slice together to form a group of logs, which are named as log vectors;
the data preprocessor analyzes and processes data by using a log vector as a basic unit, digitalizes time elements and log classification types, and discards logs lacking core time elements and log type information in each group of data.
Preferably, an intelligent analysis engine is constructed, and a log analysis evaluation model is output by analyzing log vector data and fault marks output by a data preprocessor, and the method specifically includes the following steps:
the intelligent analysis engine needs to prepare reasonable fault classification as a fault marker in advance when analyzing log vector data; the marked data, the fault mark of which is used as a new characteristic value is added into the log vector input before and is used as a new log vector to participate in analysis;
and the intelligent analysis engine analyzes and processes the log vector carrying the fault marking characteristic value through an artificial intelligence algorithm.
Preferably, the deploying of the intelligent analysis engine in the online production system, importing the evaluation model output offline, and calculating the output fault through the real-time monitoring log specifically include:
an intelligent analysis engine is deployed in an online production system environment, and an evaluation model is imported in a manual or automatic mode;
and a data preprocessor is deployed in the online production system environment, and log data are collected in real time and log vectors are output for an intelligent analysis engine to analyze and process.
In a third aspect, the present invention further provides an apparatus for automatically analyzing a system log, where the apparatus includes:
at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executable by the processor for performing the method for automatically analyzing a system log of the first aspect.
In a fourth aspect, the present invention further provides a non-transitory computer storage medium storing computer-executable instructions for execution by one or more processors to perform the method for automatically analyzing a system log according to the first aspect.
The invention provides a model evaluation method capable of collecting logs aiming at multiple links and performing comprehensive analysis, and provides a set of log feature dimension reduction processing means based on possible version updating of logs in each link and adjustment of key feature elements of the logs, so that the whole analysis process is not redundant and uncontrollable due to change of log feature dimensions in different links.
The analysis method provided by the invention can realize rapid analysis of the system log, can greatly reduce the working difficulty of operation and maintenance personnel, and can lead software developers to focus on the core service logic of the software, thereby being not disturbed by the complicated system log output logic.
The requirement for manpower learning cost in each link is reduced through automatic analysis, automatic problem positioning processing within 7X24 hours is achieved, after a relevant system is deployed, problems can be directly closed at a user side, product maintenance efficiency is improved, manpower cost of a chain in the whole link is reduced, and research and development personnel can concentrate on product development.
[ description of the drawings ]
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below. It is obvious that the drawings described below are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
Fig. 1 is a schematic flow chart of a method for automatically analyzing a system log according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a method for automatically analyzing a system log according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an architecture of an automatic system log analysis system according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an automatic analysis process of a system log according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an automatic system log analysis device according to an embodiment of the present invention.
[ detailed description ] embodiments
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In the description of the present invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", and the like indicate orientations or positional relationships based on those shown in the drawings, and are for convenience only to describe the present invention without requiring the present invention to be necessarily constructed and operated in a specific orientation, and thus should not be construed as limiting the present invention.
In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
Example 1:
embodiment 1 of the present invention provides an automatic system log analysis method, which is used for coping with the situation that the application range of enterprise products is increased, even the scale of corresponding customer service and after-sale expands throughout the sea and abroad, and the maintenance cost of related logs far exceeds the product development, as shown in fig. 1, the method includes:
in step 201, logs of one or more target systems are automatically collected.
As a typical reference scenario applicable to the embodiment of the present invention, the log of the target system may be generated in a log system set for one or more of a research and development link, a production link, a sales link, a customer service link, and an after-sales link. And taking one or more log systems corresponding to the research and development link, the production link, the sales link, the customer service link and the after-sales link as target systems, and collecting logs in the corresponding target systems. Therefore, in the embodiment of the present invention, a distributed log system is more adopted for management, and the corresponding collection process is more likely to be completed between different servers, even different databases.
It should be noted that the above-mentioned development link, production link, sales link, customer service link and after-sales link are only an exemplary example, and the link chains of the logs of different companies have different complexity, for example: some enterprise scenes may add a testing link, and some enterprises may remove similar research and development links therein, and the like, and the similar appropriately adjusted configuration forms of the target system all belong to the protection scope of the present invention.
In step 202, log key features are extracted, and when a log feature dimension used for representing the number of the log key feature features exceeds a first preset threshold, dimension reduction processing of the log feature dimension is performed.
As can be seen from the explanation of the previous step, in a typical scenario addressed by the embodiment of the present invention, a chain composed of different log links is involved, and therefore, the management of the entire log involved is more complex and varied; in order to improve the change (including addition and deletion) of the key feature elements of the log in the respective log links, in the embodiment of the present invention, particular attention is paid to the situation of the key feature elements of the newly added log (a typical performance is to add one or more rows of feature dimensions in the log), at this time, in order to ensure the high efficiency of log management, dimension reduction processing needs to be performed on the feature dimensions of the log, which is a special technical property that can be embodied only in the application scenario provided by the embodiment of the present invention. The specific dimension reduction processing will be specifically shown in the subsequent extended content, which is not described herein again.
The first preset threshold value can be correspondingly adjusted adaptively according to hardware processing capacities of different platforms; and the first preset threshold expression form is mainly the number of the key characteristic elements of the log contained in the corresponding log in each link. Preferably, when the first preset threshold is configured, it can be ensured that 30% of computing resources are still available when the corresponding server runs the method at full load.
In step 203, one or more models to be evaluated are output according to the content of the key feature elements of the log.
One manifestation of the tape evaluation model is a log vector. The log vector may be a log chain (also expressed as a link chain in the embodiment of the present invention) pointing to each product object, which is established according to the collected common identification features in the logs of one or more target systems, and then the key feature elements of the logs in each log in the log chain are integrated to form the log vector. The model to be evaluated may be understood as an evaluation classification to which the log vector is mapped, for example: for electronic products, the evaluation classification may be failure, damage, etc., and for failures therein, a main board failure, a screen failure, a sensor failure, etc. may be further subdivided.
In step 204, combining a big data resource pool formed by historical evaluated models and/or expert teams, and completing evaluation classification of the models to be evaluated.
The implementation of step 204 here includes at least two technical implications:
the first layer of technology means: and performing similarity matching between the historical evaluated model and the model to be evaluated in a big data resource pool formed by the historical evaluated model, and taking the evaluation classification of the corresponding evaluated model as the evaluation classification of the model to be evaluated if the matching similarity exceeds a second preset threshold. The second preset threshold is a dynamically produced parameter value, and the generated basis is adjusted according to a similarity difference between two evaluation classifications with the top similarity ranking, wherein the similarity difference is required to be generated to achieve the classification accuracy of over 95%. For example: for a model to be evaluated, if the difference between two evaluation classifications with the top similarity ranking is smaller, the corresponding second preset threshold value is set higher, so as to improve the accuracy of the final evaluation classification, or the situation that the accurate evaluation classification is difficult to be made is handed over to the second layer of technical meaning described next.
The meaning of the second layer technology is: similarity matching between the historical evaluated model and the model to be evaluated is carried out in a big data resource pool formed by the historical evaluated model, and if the historical evaluated model with the similarity exceeding a second preset threshold value is not found, evaluation classification of the expert team on the model to be evaluated is obtained; and if the evaluation classification given by the expert team is different from the evaluation classification given by the historical evaluated model, adding the evaluation classification given by the model to be evaluated and the expert team as a new historical evaluated model into the big data resource pool.
The embodiment of the invention provides a model evaluation method capable of collecting logs aiming at multiple links and performing comprehensive analysis, and provides a set of log feature dimension reduction processing means based on possible version updating of logs in each link and adjustment of key feature elements of the logs, so that the whole analysis process is not redundant and uncontrollable due to change of log feature dimensions in different links.
With the embodiment of the present invention, for the extraction of the log key elements involved in step 202, and when the log feature dimension used for representing the number of the log key feature elements exceeds a first preset threshold, the dimension reduction processing of the log feature dimension is performed, and the following specific descriptions are given to several scenarios:
scene one,
The log key elements are composed of various contents contained in the log, and if the log feature dimension composed of the number of the log key feature elements in the first link exceeds a first preset threshold value; the first link is only a name given for convenience in the following description, and the specific expression of the first link may mean any one of a research and development link, a production link, a sales link, a customer service link, and an after-sales link listed in the embodiments of the present invention.
Confirming that the first link can be used for associating a plurality of log key feature elements aiming at the same target object in other links; and compressing the plurality of key characteristic elements of the log according to the incidence relation between the first link and other links.
For example: the first link is respectively associated with a second link through a first log key feature element, and is associated with a third link through a second log key feature element; if the second link and the third link are also related through a third log key feature element, selecting one of the first log key feature element and the second log key feature element; and generating an association relation among the first link log, the second link log and the third link log according to the selected first log key feature element, the selected second log key feature element and the selected third log key feature element.
Also for example: and forming a pair by the characteristic value and the next characteristic value, constructing a variable based on the characteristic value in each pair, and counting the occurrence times of the simultaneous values, wherein if the values of the characteristic value 1 and the characteristic value 2 are respectively 3 and 5, the variable is recorded as I12(3, 5) ═ n, and n is the finally counted times.
Circularly traversing all records, and counting the occurrence frequency of each pair; dividing the occurrence times n by the total log record number m, namely the relevance of the characteristic value; the relevance of the characteristic values is transitive, namely A and B appear simultaneously, B and C appear simultaneously, and A and C also appear simultaneously, so that pairing only needs to consider two adjacent characteristic values; and if the final eigenvalue relevance value is more than 80%, the pairing connection is considered to be very tight, and only one of the pairing connection can be considered to be reserved (the specific judgment threshold can be freely set).
Scene two,
The log key elements are composed of various contents contained in the log, and if the log feature dimension composed of the number of the log key feature elements in the first link exceeds a first preset threshold value;
and confirming a plurality of log key feature elements in the first link, and compressing the plurality of log key feature elements when the plurality of log key feature elements are used for representing the same meaning in the historical use record.
The logs shown in the following table are given as an example:
and if the IP and the log source can represent the meaning of the same target object, correspondingly adding a new feature item in the log and exceeding the first preset threshold value, and selecting the IP and the log source from the key feature elements of the log.
Scene three,
The log key elements are composed of various contents contained in the log, and if the log feature dimension composed of the number of the log key feature elements in the first link exceeds a first preset threshold value;
and confirming one or more log key feature elements in the first link, and compressing the one or more log key feature elements when the utilization rate in the historical utilization record is lower than a third preset threshold and no substantial correlation exists between the utilization content and the final result. The compression processing here refers to deletion processing, that is, the key feature elements of the log that have no substantial relevance to the evaluation classification can be deleted in the management policy proposed in the embodiment of the present invention.
For example: and (3) calculating the difference of the characteristic values, collecting a certain number of log records, and then performing statistical analysis on each characteristic value, wherein if the difference of a certain characteristic value is not obvious and is excessively concentrated on a certain value, the influence of the characteristic value on the result is small finally, and the removal can be considered, for example, in a log on a single machine system, the IP of the output log record is the IP of the local machine, and the value of the IP is not very large at the moment and can be removed.
However, it should be noted that, in a preferred embodiment, the system log analysis system provided in the embodiment of the present invention has its own data storage and management mechanism and is independent of the original log management. The relationship between the two can be understood as that the original log management belongs to the data backup of the bottom layer, and the data compressed by the invention is specially used for classification evaluation (in a specific application scenario, the relationship can also be directly understood as fault type evaluation).
Scene four,
The log key elements are composed of various contents contained in the log, and if the log feature dimension composed of the number of the log key feature elements in the first link exceeds a first preset threshold value;
confirming one or more log key feature elements in the first link, wherein the respective contents of the one or more log key feature elements are parameter objects with specified quantity; and performing merging parameter object type compression processing on the one or more log key feature elements.
For example: the input and output parameters are merged, for example, 4 input parameters are provided, the values are sequentially 10, AA, DD and CC, the new characteristic value is 10AADDCC, and the input parameters can be combined according to the serial number of each parameter category. For another example, if the serial number of the parameter 1 is 1, the serial number of the parameter 2 is C, the serial number of the parameter 3 is d, and the serial number of the parameter 4 is 0, the new characteristic value is 1Cd 0.
It is fully considered here that the role of the original log is to provide the view and viewing use for the maintenance personnel, and for the embodiment of the present invention, too many key feature elements of the log increase the occupation of the computing resources of the server. One of the effective compression methods is to reconstruct the mapping relationship, i.e., merge multiple originally separated log key feature elements, and give a new meaning to the merged parameter values, or understand the merged parameter values as new log vector constituent elements, for participating in the specific evaluation and classification process.
In combination with the embodiment of the present invention, in order to further improve the analysis efficiency, there is an extended implementation scheme, where when the evaluation result of the model to be evaluated generated by performing the log on each link for a specific product object is the first fault type, as shown in fig. 2, the analysis method further includes:
in step 301, according to the log of the production link in the model to be evaluated of the product object, the logs of other product objects in the same batch or the same production line are searched and sorted as the corresponding log of the potential first fault type.
In step 302, the related consumer contact information is obtained according to the key characteristic elements of the log recorded in the sales link in the log arrangement result of the potential first fault type.
In step 303, the customer service triggering step performs a pre-attention and/or a troubleshooting of the relevant first fault type according to the customer contact information.
Through the steps 301 and 303, the link chain (or called as a log chain) provided by the embodiment of the invention can be quickly passed, so that one action and three actions can be quickly performed, and the potential consumers who may invent the first fault can be completed in advance. Therefore, the level of the after-sale log association is further improved to a new height, and the log association analysis which cannot be completed in the prior art and the practical benefit brought by the analysis are realized.
In the embodiment of the present invention, due to the proposed concept of the log chain, the difference configuration on the period acquired in step 201 can be brought correspondingly, and specifically, the acquisition periods of the logs in different links are determined according to the distribution of the associated logs in the evaluation model of fault evaluation classification; and each link has respective log generation speed, and the log of each link realizes the establishment of the incidence relation through the evaluation model. If the difference of the acquisition periods is configured in place, the disk reading efficiency in the subsequent log chain generation process can be further improved; and the corresponding specific configuration parameters are obtained by analyzing the storage area where the key characteristic elements of the log, which are acquired according to history and used for generating the log chain, are located and the matched acquisition time label.
Example 2:
an embodiment of the present invention provides an automatic system log analysis system, where the system can be used to run the automatic system log analysis method described in embodiment 1, and the embodiment of the present invention shows how the analysis method provided in embodiment 1 is effectively used from a more complete architecture level, as shown in fig. 3, the system includes:
and an independent log acquisition unit is constructed in the online production system, and the logs of the target system are automatically acquired and uploaded to an offline analysis server.
And the offline analysis server performs data preprocessing, extracts key elements of the logs and reduces characteristic dimensionality of the logs, so that the logs can be clustered according to faults by a subsequent intelligent analysis engine.
And the offline analysis server starts an intelligent analysis engine, and matches an expert team to manually mark faults according to actual conditions by analyzing key characteristic elements of the log, so as to output an evaluation model.
An intelligent analysis engine is deployed in an online production system, an evaluation model output offline is imported, and output faults are calculated through real-time monitoring logs.
The analysis method provided by the embodiment of the invention can realize rapid analysis of the system log, can greatly reduce the working difficulty of operation and maintenance personnel, and can lead software developers to focus on the core service logic of the software, thereby being not disturbed by the complicated output logic of the system log.
As can be seen from the embodiment of the present invention, the evaluation classification obtained in the step 201-204 in the embodiment 1 is used as a way for importing a corresponding evaluation model (including the evaluation classification) into an online production system deployment intelligent analysis engine in the embodiment of the present invention.
According to the embodiment of the invention, the requirement on the manpower learning cost in each link is reduced through automatic analysis, the automatic problem positioning processing in 7X24 hours is realized, the problem can be closed at the user side directly after a relevant system is deployed, the product maintenance efficiency is improved, the manpower cost of a chain in the whole link is reduced, and research and development personnel can concentrate on product development.
In combination with the embodiment of the present invention, an independent log collection unit is constructed for the online production system, and a log of a target system is automatically collected and uploaded to an offline analysis server, and an optional implementation manner is further provided, which specifically includes:
and constructing an independent log acquisition unit, and realizing automatic timing log copying and transmission by configuring a target log path and a log uploading path.
And deploying one or more log acquisition units by adopting a centralized or distributed deployment mode matched with the characteristics of the target software system.
And starting a log acquisition unit, and extracting the log to a specified server path in real time.
By combining the embodiment of the invention, data preprocessing is performed on the offline analysis server, the key elements of the log are extracted, the characteristic dimensionality of the log is reduced, the subsequent intelligent analysis engine can conveniently cluster the log according to faults, and an optional implementation mode is also provided, as shown in fig. 4, the method specifically comprises the following steps:
and constructing a data preprocessor, loading the log data collected by the log collection unit, and classifying according to different log entry filling modules.
The data preprocessor groups the slices according to time elements generated by the logs, and gathers the logs in the same time slice by a plurality of log entry filling modules to form a group of logs named as log vectors.
The data preprocessor analyzes and processes data by using a log vector as a basic unit, digitalizes time elements and log classification types, and discards logs lacking core time elements and log type information in each group of data. The numeralization method can adopt common binary coding, equal-scale scaling and the like or other effective and reasonable self-defining methods. The formation of a group of logs described herein is a technical object in two levels compared with the chain of logs in embodiment 1, and the formation of a group of logs refers to the formation of the log content corresponding to one object in one link, that is, each individual log constituting the chain of logs in embodiment 1. The time slicing set forth in the embodiments of the present invention is at the program code level, for example: the log corresponding to a product object in a link can be completed by filling specific contents in log key feature elements which are respectively responsible by a plurality of log entry filling modules in series or in parallel. Here, the logs in the same time slice of the plurality of log entry filling modules are aggregated to form a group of logs, so that the integrity of the corresponding logs can be ensured.
With the embodiment of the present invention, for the intelligent analysis engine construction, a log analysis evaluation model is output by analyzing log vector data and a fault flag output by the data preprocessor, and an optional implementation manner is also provided, which specifically includes:
the intelligent analysis engine needs to prepare reasonable fault classification as a fault marker in advance when analyzing log vector data; the marked data, the fault mark of which is used as a new characteristic value is added into the log vector input before and is used as a new log vector to participate in analysis; the fault marking can be manually summarized by a front-line operation and maintenance expert, and can also be automatically marked by providing keywords or fault phenomenon development tools.
And the intelligent analysis engine analyzes and processes the log vector carrying the fault marking characteristic value through an artificial intelligence algorithm. Algorithms employed include, but are not limited to, affinity analysis, cluster analysis (e.g., random forest, K-means, Gaussian mixture model), neural networks (DNN). And (4) suggesting to analyze and process through a plurality of algorithms at the same time, then transversely comparing the algorithm with the highest accuracy, and then outputting an evaluation model. Because the intelligent analysis engine inputs log vectors carrying fault marking characteristic values, when a target system is updated significantly, a new fault type is introduced or an original fault classification is changed, a new evaluation model needs to be output again.
In combination with the embodiment of the present invention, an optional implementation manner is further provided for deploying an intelligent analysis engine for the online production system, importing an offline output evaluation model, and calculating an output fault by monitoring a log in real time, and specifically includes:
an intelligent analysis engine is deployed in an online production system environment, and an evaluation model is imported in a manual or automatic mode;
and a data preprocessor is deployed in the online production system environment, and log data are collected in real time and log vectors are output for an intelligent analysis engine to analyze and process. And the offline intelligent analysis engine outputs an evaluation model which needs huge historical log vectors to be obstructed, and the online system only needs to collect current logs for fault classification, for example, the logs pushed forward for 5 minutes at the current time are collected every 5 minutes, so that the resource consumption of the server is reduced, and the influence on the online production system is reduced.
According to the embodiment of the invention, the logs of the online production system are collected through the independent log collection unit and transmitted to the offline system for analysis and processing. The log as a natural language text can not be directly analyzed, so a data preprocessor needs to be introduced, and the log is converted into vector data which can participate in artificial intelligent analysis and processing by adopting a series of methods such as classification, time slicing, natural language processing, affinity analysis or principal component analysis dimension reduction and the like. And the offline intelligent analysis engine processes the log vector data and the fault marking characteristic value set by an expert by adopting a machine learning or deep neural network method through input log vector data, and transversely compares and selects an optimal algorithm and outputs an intelligent library model. And the online intelligent engine analyzes and processes the log in real time by introducing an intelligent library model and outputs the fault reason.
Next, the optimization and improvement idea of the implementation process of the embodiment of the present invention is described by a specific log table.
Taking the above system log as an example, the core characteristics of the system log are the same and different, but the log content, the number of the instructions and the input and output parameters of the instructions are not controllable, and the number of the overall characteristic values is possible from several to hundreds.
In the embodiment of the present invention, a data optimization process in the process of converting the log file into the evaluation model is further provided, and typical parameters are picked out and exemplarily set as follows:
time parameters: the characteristic of the time characteristic value is continuous, if every moment is used as a record, the total amount of the finally output records is huge, and in order to comprehensively consider the processing efficiency and the independent accuracy of the data, the time characteristic value is merged and extracted in a slicing mode, and the method specifically comprises the following steps:
cutting the text file at a fixed time interval, wherein the general suggestion is 5s-30 s; packaging all log contents in the time slice as a record; the new log record uses the median of the slice start time as the time characteristic value of the new record (the start time or the end time can also be simply used as the time characteristic value of the new record); the Time characteristic value is subjected to data processing, the Time characteristic value can be easily converted into Universal Time Coordinated (UTC) seconds no matter the Time format of the log, and the problem of Time inconsistency caused by the fact that a distributed system spans Time zones can be avoided. Then, the UTC seconds are scaled integrally, and various methods are available, such as: fixedly subtracting a threshold value; scaling, such as dividing by a fixed dividend at the same time; and sequencing based on the time sequence, and independently giving a serial number id, wherein if the characteristic value of the first recording time is 1, the characteristic value of the nth recording time is n.
The level parameters are as follows: the log generally has keywords as levels, which are commonly used as info, debug, war and error, or may not be, and is characterized in that the number is limited, and the data processing mode is as follows: respectively giving id according to class types, such as info ═ 1, debug ═ 2, war ═ 3, error ═ 4, and if not, 0; binary coding, assuming that the total number of levels is 4, info is 00, debug is 01, war is 10, and error is 11; text keywords are used directly as feature values.
The log ID parameter: directly using the original id; the original id may also be scaled equally, such as subtracting a fixed value, dividing by a fixed value, or sorting based on id, and reassigning sequence numbers, etc.
Log source parameters: under the condition that the types and the lengths of the log sources are limited, the log sources are directly used; and based on the limited log source types, the serial numbers of the types are separately assigned.
IP parameters: directly using IP; and ordering the IPs according to the dictionary sequence, and giving sequence numbers as new characteristic values.
The action parameters are as follows: directly giving the action name as a characteristic value; and sorting the action names according to the dictionary sequence, and giving the sequence number as a new characteristic value.
Example 3:
fig. 5 is a schematic structural diagram of an automatic system log analysis device according to an embodiment of the present invention. The system log automatic analysis device of the present embodiment includes one or more processors 21 and a memory 22. In fig. 5, one processor 21 is taken as an example.
The processor 21 and the memory 22 may be connected by a bus or other means, and fig. 5 illustrates the connection by a bus as an example.
The memory 22, which is a non-volatile computer-readable storage medium, may be used to store a non-volatile software program and a non-volatile computer-executable program, such as the automatic analysis method of the system log in embodiment 1. The processor 21 executes the system log automatic analysis method by executing a nonvolatile software program and instructions stored in the memory 22.
The memory 22 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, the memory 22 may optionally include memory located remotely from the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The program instructions/modules are stored in the memory 22 and, when executed by the one or more processors 21, perform the method for automatically analyzing the system log in embodiment 1 described above, for example, perform the steps shown in fig. 1 and 2 described above.
It should be noted that, for the information interaction, execution process and other contents between the modules and units in the apparatus and system, the specific contents may refer to the description in the embodiment of the method of the present invention because the same concept is used as the embodiment of the processing method of the present invention, and are not described herein again.
Those of ordinary skill in the art will appreciate that all or part of the steps of the various methods of the embodiments may be implemented by associated hardware as instructed by a program, which may be stored on a computer-readable storage medium, which may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (14)
1. A method for automatically analyzing system logs is characterized by comprising the following steps:
automatically collecting logs of one or more target systems;
extracting key elements of the log, and performing dimension reduction processing on the key feature dimensions of the log when the key feature dimensions of the log, which are used for representing the number of the key feature elements of the log, exceed a first preset threshold;
outputting one or more models to be evaluated according to the content of the key characteristic elements of the log;
and finishing the evaluation classification of the model to be evaluated by combining a big data resource pool and/or an expert team formed by historical evaluated models.
2. The method according to claim 1, wherein the classification of the evaluation of the model to be evaluated is performed in combination with a big data resource pool and/or an expert team formed by historical evaluated models, and specifically comprises:
and performing similarity matching between the historical evaluated model and the model to be evaluated in a big data resource pool formed by the historical evaluated model, and taking the evaluation classification of the corresponding evaluated model as the evaluation classification of the model to be evaluated if the matching similarity exceeds a second preset threshold.
3. The method according to claim 1, wherein the classification of the evaluation of the model to be evaluated is performed in combination with a big data resource pool and/or an expert team formed by historical evaluated models, and specifically comprises:
similarity matching between the historical evaluated model and the model to be evaluated is carried out in a big data resource pool formed by the historical evaluated model, and if the historical evaluated model with the similarity exceeding a second preset threshold value is not found, evaluation classification of the expert team on the model to be evaluated is obtained;
and if the evaluation classification given by the expert team is different from the evaluation classification given by the historical evaluated model, adding the evaluation classification given by the model to be evaluated and the expert team as a new historical evaluated model into the big data resource pool.
4. The method according to claim 1, wherein the automatically collecting logs of one or more target systems specifically comprises:
a log system is arranged aiming at one or more links of a research and development link, a production link, a sales link, a customer service link and an after-sales link;
and taking one or more log systems corresponding to the research and development link, the production link, the sales link, the customer service link and the after-sales link as target systems, and collecting logs in the corresponding target systems.
5. The method according to claim 4, wherein the method includes extracting a log key element, and performing a dimension reduction process on a log feature dimension when the log feature dimension used for representing the number of the log key feature elements exceeds a first preset threshold, and specifically includes:
the log key elements are composed of various contents contained in the log, and if the log feature dimension composed of the number of the log key feature elements in the first link exceeds a first preset threshold value;
confirming that the first link can be used for associating a plurality of log key feature elements aiming at the same target object in other links; compressing the plurality of key characteristic elements of the log according to the incidence relation between the first link and other links; and/or the presence of a gas in the gas,
confirming a plurality of log key feature elements in the first link, and compressing the plurality of log key feature elements when the plurality of log key feature elements are used for representing the same meaning in a historical use record; and/or the presence of a gas in the gas,
confirming one or more log key feature elements in the first link, and compressing one or more log key feature elements when the utilization rate in the historical utilization record is lower than a third preset threshold and no substantial correlation exists between the utilization content and the final result; and/or the presence of a gas in the gas,
confirming one or more log key feature elements in the first link, wherein the respective contents of the one or more log key feature elements are parameter objects with specified quantity; and performing merging parameter object type compression processing on the one or more log key feature elements.
6. The method according to claim 5, wherein the compressing the plurality of key feature elements according to the association relationship between the first link and each of the other links specifically comprises:
the first link is respectively associated with a second link through a first log key feature element, and is associated with a third link through a second log key feature element;
if the second link and the third link are also related through a third log key feature element, selecting one of the first log key feature element and the second log key feature element;
and generating an association relation among the first link log, the second link log and the third link log according to the selected first log key feature element, the selected second log key feature element and the selected third log key feature element.
7. The method according to any one of claims 4 to 6, wherein when the model to be evaluated generated by the log in each link is performed on a specific product object, and the corresponding evaluation result is the first failure type, the method further comprises:
according to the logs of the production links in the model to be evaluated of the product objects, the logs of other product objects in the same batch or the same production line are searched and are used as corresponding logs of the potential first fault type for sorting;
obtaining a related consumer contact way according to the key characteristic elements of the log recorded in the sales link in the potential first fault type log arrangement result;
and the customer service triggering link carries out pre-attention and/or investigation on the related first fault type according to the contact way of the consumer.
8. The method for automatically analyzing the system logs according to any one of claims 4 to 6, wherein the collection periods of the logs of different links are determined according to the distribution of the associated logs in the evaluation model of fault evaluation classification;
and each link has respective log generation speed, and the log of each link realizes the establishment of the incidence relation through the evaluation model.
9. An automatic system log analysis system, comprising:
an independent log collection unit is constructed in the online production system, and logs of a target system are automatically collected and uploaded to an offline analysis server;
the offline analysis server performs data preprocessing, extracts key elements of the logs and reduces characteristic dimensions of the logs, so that subsequent intelligent analysis engines can conveniently cluster the logs according to faults;
the offline analysis server starts an intelligent analysis engine, and manually marks faults according to actual conditions by matching with an expert team through analyzing key characteristic elements of the logs, so as to output an evaluation model;
an intelligent analysis engine is deployed in an online production system, an evaluation model output offline is imported, and output faults are calculated through real-time monitoring logs.
10. The system log automatic analysis system according to claim 9, wherein the online production system constructs an independent log collection unit, automatically collects logs of a target system, and uploads the logs to the offline analysis server, and specifically includes:
an independent log acquisition unit is constructed, and automatic timing log copying and transmission are realized by configuring a target log path and a log uploading path;
one or more log acquisition units are deployed in a centralized or distributed deployment mode matched with the characteristics of the target software system;
and starting a log acquisition unit, and extracting the log to a specified server path in real time.
11. The system log automatic analysis system according to claim 10, wherein the offline analysis server performs data preprocessing to extract key elements of the log and reduce characteristic dimensions of the log, so that a subsequent intelligent analysis engine can conveniently cluster the log according to faults, and specifically comprises:
constructing a data preprocessor, loading log data collected by a log collection unit, and classifying according to different log entry filling modules;
the data preprocessor groups the slices according to time elements generated by the logs, and gathers the logs of a plurality of log item filling modules in the same time slice together to form a group of logs, which are named as log vectors;
the data preprocessor analyzes and processes data by using a log vector as a basic unit, digitalizes time elements and log classification types, and discards logs lacking core time elements and log type information in each group of data.
12. The system log automatic analysis system according to claim 11, wherein an intelligent analysis engine is configured to output a log analysis evaluation model by analyzing log vector data and fault flags output by the data preprocessor, and the system log automatic analysis system specifically includes:
the intelligent analysis engine needs to prepare reasonable fault classification as a fault marker in advance when analyzing log vector data; the marked data, the fault mark of which is used as a new characteristic value is added into the log vector input before and is used as a new log vector to participate in analysis;
and the intelligent analysis engine analyzes and processes the log vector carrying the fault marking characteristic value through an artificial intelligence algorithm.
13. The system log automatic analysis system of claim 12, wherein the online production system deploys an intelligent analysis engine, imports an evaluation model output offline, and calculates an output fault by monitoring the log in real time, and specifically comprises:
an intelligent analysis engine is deployed in an online production system environment, and an evaluation model is imported in a manual or automatic mode;
and a data preprocessor is deployed in the online production system environment, and log data are collected in real time and log vectors are output for an intelligent analysis engine to analyze and process.
14. An apparatus for automatically analyzing a system log, the apparatus comprising:
at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the method of automatically analyzing a system log of any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010313503.XA CN113535667B (en) | 2020-04-20 | 2020-04-20 | Automatic analysis method, device and system for system log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010313503.XA CN113535667B (en) | 2020-04-20 | 2020-04-20 | Automatic analysis method, device and system for system log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113535667A true CN113535667A (en) | 2021-10-22 |
| CN113535667B CN113535667B (en) | 2024-08-02 |
Family
ID=78093702
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010313503.XA Active CN113535667B (en) | 2020-04-20 | 2020-04-20 | Automatic analysis method, device and system for system log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113535667B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115662607A (en) * | 2022-12-13 | 2023-01-31 | 四川大学 | Internet online inquiry recommendation method based on big data analysis and server |
| CN116432469A (en) * | 2023-04-23 | 2023-07-14 | 黑龙江木水网络科技有限公司 | Data channel management system and method based on big data |
| CN117473094A (en) * | 2023-12-27 | 2024-01-30 | 南京聚铭网络科技有限公司 | Log classification method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150089309A1 (en) * | 2013-09-23 | 2015-03-26 | Emc Corporation | Troubleshooting based on log similarity |
| CN106155884A (en) * | 2015-03-31 | 2016-11-23 | 展讯通信（上海）有限公司 | A kind of log analysis method and system |
| CN107256219A (en) * | 2017-04-24 | 2017-10-17 | 卡斯柯信号有限公司 | Big data convergence analysis method applied to automatic train control system massive logs |
| US20190095313A1 (en) * | 2017-09-22 | 2019-03-28 | Nec Laboratories America, Inc. | Log-based system maintenance and management |
| CN109714187A (en) * | 2018-08-17 | 2019-05-03 | 平安普惠企业管理有限公司 | Log analysis method, device, equipment and storage medium based on machine learning |
-
2020
- 2020-04-20 CN CN202010313503.XA patent/CN113535667B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150089309A1 (en) * | 2013-09-23 | 2015-03-26 | Emc Corporation | Troubleshooting based on log similarity |
| CN106155884A (en) * | 2015-03-31 | 2016-11-23 | 展讯通信（上海）有限公司 | A kind of log analysis method and system |
| CN107256219A (en) * | 2017-04-24 | 2017-10-17 | 卡斯柯信号有限公司 | Big data convergence analysis method applied to automatic train control system massive logs |
| US20190095313A1 (en) * | 2017-09-22 | 2019-03-28 | Nec Laboratories America, Inc. | Log-based system maintenance and management |
| CN109714187A (en) * | 2018-08-17 | 2019-05-03 | 平安普惠企业管理有限公司 | Log analysis method, device, equipment and storage medium based on machine learning |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115662607A (en) * | 2022-12-13 | 2023-01-31 | 四川大学 | Internet online inquiry recommendation method based on big data analysis and server |
| CN115662607B (en) * | 2022-12-13 | 2023-04-07 | 四川大学 | Internet online inquiry recommendation method based on big data analysis and server |
| CN116432469A (en) * | 2023-04-23 | 2023-07-14 | 黑龙江木水网络科技有限公司 | Data channel management system and method based on big data |
| CN117473094A (en) * | 2023-12-27 | 2024-01-30 | 南京聚铭网络科技有限公司 | Log classification method and system |
| CN117473094B (en) * | 2023-12-27 | 2024-03-22 | 南京聚铭网络科技有限公司 | Log classification method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113535667B (en) | 2024-08-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113535667A (en) | Method, device and system for automatically analyzing system logs | |
| CN112221156B (en) | Data abnormality recognition method, data abnormality recognition device, storage medium, and electronic device | |
| CN112383052B (en) | Power grid fault repairing method and device based on power internet of things | |
| CN110703057B (en) | Power equipment partial discharge diagnosis method based on data enhancement and neural network | |
| CN111177095A (en) | Log analysis method and device, computer equipment and storage medium | |
| CN111738462B (en) | Fault first-aid repair active service early warning method for electric power metering device | |
| CN112559376A (en) | Automatic positioning method and device for database fault and electronic equipment | |
| CN111967675A (en) | Photovoltaic power generation amount prediction method and prediction device | |
| CN114942923B (en) | Cloud platform-based unified management system for big data calculation and analysis | |
| CN112632127B (en) | Data processing method for real-time data acquisition and time sequence of equipment operation | |
| CN106534784A (en) | Acquisition analysis storage statistical system for video analysis data result set | |
| CN114328277A (en) | Software defect prediction and quality analysis method, device, equipment and medium | |
| CN113268370A (en) | Root cause alarm analysis method, system, equipment and storage medium | |
| CN117221087A (en) | Alarm root cause positioning method, device and medium | |
| CN106846170B (en) | Generator set trip monitoring method and monitoring device thereof | |
| CN116541782A (en) | Power marketing data anomaly identification method | |
| CN114021425A (en) | Power system operation data modeling and feature selection method and device, electronic equipment and storage medium | |
| CN113052636A (en) | Agricultural product sales prediction system based on multi-source heterogeneous data | |
| CN114465875B (en) | Fault processing method and device | |
| CN114048854B (en) | Deep neural network big data internal data file management method | |
| CN112800219B (en) | Method and system for feeding back customer service log to return database | |
| CN108009224A (en) | The sorting technique and device of power customer | |
| CN110196880A (en) | Synchronization of data in heterogeneous database method and apparatus, storage medium and electronic device | |
| CN113379211A (en) | Block chain-based logistics information platform default risk management and control system and method | |
| CN114329086A (en) | Method and system for screening question clues of bidding data in electronic government affairs |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |