CN117827620A - Abnormality diagnosis method, training device, training equipment, and recording medium - Google Patents
Abnormality diagnosis method, training device, training equipment, and recording medium Download PDFInfo
- Publication number
- CN117827620A CN117827620A CN202410247168.6A CN202410247168A CN117827620A CN 117827620 A CN117827620 A CN 117827620A CN 202410247168 A CN202410247168 A CN 202410247168A CN 117827620 A CN117827620 A CN 117827620A
- Authority
- CN
- China
- Prior art keywords
- log
- template
- container engine
- sequence
- abnormal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003745 diagnosis Methods 0.000 title claims abstract description 106
- 238000012549 training Methods 0.000 title claims abstract description 82
- 238000000034 method Methods 0.000 title claims abstract description 73
- 230000005856 abnormality Effects 0.000 title claims abstract description 56
- 230000002159 abnormal effect Effects 0.000 claims description 118
- 238000006243 chemical reaction Methods 0.000 claims description 24
- 230000003993 interaction Effects 0.000 claims description 20
- 238000007637 random forest analysis Methods 0.000 claims description 13
- 238000002372 labelling Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 17
- 238000012423 maintenance Methods 0.000 description 15
- 238000012795 verification Methods 0.000 description 13
- 238000000605 extraction Methods 0.000 description 9
- 238000001514 detection method Methods 0.000 description 8
- 238000013507 mapping Methods 0.000 description 8
- 238000007635 classification algorithm Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000003066 decision tree Methods 0.000 description 3
- 238000012937 correction Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Abstract
The invention provides an abnormality diagnosis method, a training device, training equipment and a storage medium of a model, wherein the diagnosis method comprises the following steps: receiving a container engine error reporting instruction, and acquiring all operation logs of the container engine corresponding to the container engine error reporting instruction; removing all variables in the operation log to obtain a first log template; converting the first log template into a first vector sequence, wherein elements in the first vector sequence represent reverse text frequencies of fields in the first log template in a one-to-one correspondence manner; and inputting the first vector sequence into a pre-trained running log abnormality diagnosis model to obtain an abnormality diagnosis result. The invention can realize high-efficiency diagnosis on the container engine; in addition, the fault diagnosis result with high accuracy can be obtained for a long period of time.
Description
Technical Field
The embodiment of the invention relates to the technical field of software operation and maintenance, in particular to an abnormality diagnosis method, a training device, training equipment and a storage medium of a model.
Background
The container engine is used for providing the environment and functions necessary for container operation and container management, and comprises a high-level container operation time (high-level container runtime) and a low-level container operation time (low-level container runtime), wherein Containerd is a daemon, and the actual container control is completed by the Containerd-shim to call the run and other low-level container operation time.
The existing abnormality diagnosis method of the container engine is mostly manually detected by operation and maintenance personnel, and has low detection efficiency. In addition, a plurality of operation and maintenance personnel are required for large-scale containerization deployment to carry out long-term maintenance, under a manual detection scheme, fault diagnosis capability is unstable due to skill difference among different operation and maintenance personnel, accurate diagnosis results can be obtained by the operation and maintenance personnel due to different physical or psychological conditions, and high-accuracy fault diagnosis results are difficult to keep for a long time by the manual detection scheme.
Disclosure of Invention
The embodiment of the invention provides an abnormality diagnosis method, a training method of a model, a training device of the model, equipment and a storage medium, which are used for solving the problems that most of the existing abnormality diagnosis methods of container engines are manually detected by operation and maintenance personnel, and the detection efficiency is low. In addition, a plurality of operation and maintenance personnel are required for large-scale containerization deployment to carry out long-term maintenance, under a manual detection scheme, fault diagnosis capability is unstable due to skill difference among different operation and maintenance personnel, accurate diagnosis results can be obtained by the operation and maintenance personnel due to different physical or psychological conditions, and the problem that high-accuracy fault diagnosis results are difficult to keep in the manual detection scheme for a long time is solved.
In order to solve the technical problems, the invention is realized as follows:
in a first aspect, an embodiment of the present invention provides a method for diagnosing an anomaly of a container engine, including:
receiving a container engine error reporting instruction, and acquiring all operation logs of the container engine corresponding to the container engine error reporting instruction;
removing all variables in the operation log to obtain a first log template;
converting the first log template into a first vector sequence, wherein elements in the first vector sequence represent reverse text frequencies of fields in the first log template in a one-to-one correspondence manner;
and inputting the first vector sequence into a pre-trained running log abnormality diagnosis model to obtain an abnormality diagnosis result.
Alternatively, the process may be carried out in a single-stage,
converting the first log template into a first vector sequence, comprising:
determining a field number corresponding to each field in the first log template, and converting the first log template into a first template sequence represented by the field number;
generating a first template sequence checking instruction according to the first template sequence, and sending the first template sequence checking instruction to an interaction end associated with a user;
And receiving a first template sequence correct instruction sent by the interaction end, and converting the first template sequence into the first vector sequence.
In a second aspect, an embodiment of the present invention provides a training method for an anomaly diagnosis model of a container engine, including:
acquiring a plurality of historical abnormal operation logs of a container engine;
removing all variables in each historical abnormal operation log to obtain an abnormal log template; converting the abnormal log template into a vector sequence, wherein elements in the vector sequence represent reverse text frequencies of fields in the abnormal log template in a one-to-one correspondence manner;
labeling the vector sequence with an abnormal type label, and combining the vector sequences which are completely labeled to obtain a target training set; and training a diagnosis model by adopting the target training set to obtain an operation log abnormal diagnosis model.
Alternatively, the process may be carried out in a single-stage,
converting the exception log template into a vector sequence, comprising:
determining field numbers corresponding to all fields in the abnormal log template, and converting the abnormal log template into a template sequence represented by the field numbers;
generating a template sequence checking instruction according to the template sequence, and sending the template sequence checking instruction to an interaction end associated with a user;
And receiving a template sequence correct instruction sent by the interaction end, and converting the template sequence into the vector sequence.
Alternatively, the process may be carried out in a single-stage,
the diagnostic model is a random forest model.
Alternatively, the process may be carried out in a single-stage,
the history abnormal operation log includes: the first abnormal operation log generated by the container engine operation and the second abnormal operation log generated by each container operation associated with the container engine.
In a third aspect, an embodiment of the present invention provides an abnormality diagnosis apparatus for a container engine, including:
the receiving module is used for receiving a container engine error reporting instruction and acquiring all running logs of the container engine corresponding to the container engine error reporting instruction;
the execution module is used for eliminating all variables in the running log to obtain a first log template;
the execution module is further used for converting the first log template into a first vector sequence, and elements in the first vector sequence represent reverse text frequencies of fields in the first log template in a one-to-one correspondence manner;
the execution module is further configured to input the first vector sequence into a running log anomaly diagnosis model that is trained in advance, so as to obtain an anomaly diagnosis result.
In a fourth aspect, an embodiment of the present invention provides a training apparatus for an anomaly diagnosis model of a container engine, including:
the acquisition module is used for acquiring a plurality of historical abnormal operation logs of the container engine;
the conversion module is used for removing all variables in each historical abnormal operation log to obtain an abnormal log template; converting the abnormal log template into a vector sequence, wherein elements in the vector sequence represent reverse text frequencies of fields in the abnormal log template in a one-to-one correspondence manner;
the training module is used for labeling the abnormal type labels on the vector sequences, and combining the vector sequences which are completely labeled to obtain a target training set; and training a diagnosis model by adopting the target training set to obtain an operation log abnormal diagnosis model.
In a fifth aspect, an embodiment of the present invention provides an electronic device, including a processor, a memory, and a program or an instruction stored on the memory and executable on the processor, the program or the instruction implementing steps in the method for diagnosing an abnormality of a container engine according to any one of the first aspects, or implementing steps in the method for training an abnormality diagnosis model of a container engine according to any one of the second aspects, when executed by the processor.
In a sixth aspect, an embodiment of the present invention provides a readable storage medium having stored thereon a program or instructions that when executed by a processor implement steps in the method for diagnosing an anomaly of a container engine according to any one of the first aspect, or that when executed implement steps in the method for training an anomaly diagnostic model of a container engine according to any one of the second aspect.
In a seventh aspect, embodiments of the present invention provide a computer program product comprising computer instructions which, when executed by a processor, implement steps in a method for diagnosing anomalies in a container engine as described in any one of the first aspects, or which, when executed, implement steps in a method for training a model for diagnosing anomalies in a container engine as described in any one of the second aspects.
In the embodiment of the invention, all running logs of the container engine corresponding to the container engine error reporting instruction are obtained by receiving the container engine error reporting instruction; removing all variables in the operation log to obtain a first log template; converting the first log template into a first vector sequence, wherein elements in the first vector sequence represent reverse text frequencies of fields in the first log template in a one-to-one correspondence manner; the first vector sequence is input into a pre-trained running log abnormality diagnosis model to obtain an abnormality diagnosis result, and the embodiment of the invention avoids manual detection by operation and maintenance personnel and can realize high-efficiency diagnosis on a container engine; in addition, the embodiment of the invention adopts the operation log abnormality diagnosis model to realize abnormality diagnosis, and can keep high-accuracy fault diagnosis results for a long time.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
FIG. 1 is a flow chart of an anomaly diagnosis method of a container engine according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the generation of a log template;
FIG. 3 is a schematic diagram of the conversion of vector sequences;
FIG. 4 is a flow chart of a training method of an anomaly diagnostic model of a container engine according to an embodiment of the present invention;
FIG. 5 is a functional block diagram of an abnormality diagnostic device for a container engine according to an embodiment of the present invention;
FIG. 6 is a functional block diagram of a training device for an anomaly diagnostic model of a container engine according to an embodiment of the present invention;
fig. 7 is a functional block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The embodiment of the invention provides an abnormality diagnosis method for a container engine, referring to fig. 1, fig. 1 is a flow chart of the abnormality diagnosis method for the container engine according to the embodiment of the invention, and the abnormality diagnosis method comprises the following steps:
step 11: receiving a container engine error reporting instruction, and acquiring all running logs of a container engine corresponding to the container engine error reporting instruction;
step 12: removing all variables in the operation log to obtain a first log template;
step 13: converting the first log template into a first vector sequence, wherein elements in the first vector sequence represent reverse text frequencies of fields in the first log template in a one-to-one correspondence manner;
step 14: and inputting the first vector sequence into a pre-trained running log abnormality diagnosis model to obtain an abnormality diagnosis result.
The container engine is used for providing the environment and functions necessary for container operation and container management, and comprises a high-level container operation time (high-level container runtime) and a low-level container operation time (low-level container runtime), wherein Containerd is a daemon, and the actual container control is completed by the Containerd-shim to call the run and other low-level container operation time.
The operation log includes: the container engine runs the generated running log, and each container associated with the container engine runs the generated running log. In step 11 of the embodiment of the present invention, all operation logs of the container engine corresponding to the error reporting instruction of the container engine are obtained, which may specifically be operation logs generated by operation of the container engine, or operation logs generated by operation of each container associated with the container engine are obtained; it will be appreciated that it is also possible to obtain both a running log generated by the container engine running and a running log generated by each container running associated with the container engine.
It should be noted that, what kind of running log is specifically obtained is determined by the error report instruction of the container engine. In some embodiments, a user may preset a mapping relationship table between the error reporting instruction of the container engine and the running log, so as to efficiently and conveniently determine the running log according to the error reporting instruction of the container engine. In some embodiments, all running logs of the container engine corresponding to the container engine error reporting instruction may be determined by a user, specifically, before all running logs of the container engine corresponding to the container engine error reporting instruction are obtained, the container engine error reporting instruction is sent to a user side associated with the user, and a running log directory sent by the user side is received, and the running log is obtained according to the running log directory.
In practical applications, the error reporting instruction of the container engine is often not a single one, a plurality of error reporting instructions may be triggered in a short time by some complicated system problems, and in this case, determining the running log according to the mapping relation table of the error reporting instruction of the container engine and the running log often cannot determine the abnormal running instruction which can represent the deep error reporting problem. For example: in a preset first time period, receiving container engine error reporting instructions exceeding a preset quantity threshold, wherein the container engine error reporting instructions all point to running logs generated by container operation associated with a container engine (determined according to a mapping relation table of the container engine error reporting instructions and the running logs), however, a plurality of container engine error reporting instructions also represent that the container engine has problems, the running logs generated by the container engine operation should be acquired, the running logs are determined according to the mapping relation table of the container engine error reporting instructions and the running logs to cause the running logs to acquire error leakage, and the accuracy of an abnormal diagnosis result is reduced. Also for example: in a preset first time period, receiving container engine error reporting instructions exceeding a preset quantity threshold, wherein the container engine error reporting instructions all point to running logs generated by container running associated with a container engine (determined according to a container engine error reporting instruction and running log mapping relation table), however, a plurality of container engine error reporting instructions may represent that the association relation between containers associated with the container engine is abnormal, each container causing error reporting should be checked, a target container associated with the error reporting root is determined, the running logs of the target container should be acquired as the running logs, the running logs are determined according to the container engine error reporting instructions and the running log mapping relation table, the running logs are caused to acquire error leakage, and the accuracy of an abnormal diagnosis result is reduced.
Based on the acquisition error problem, the embodiment of the invention can also have a two-stage acquisition scheme, wherein the first stage determines the running log by the mapping relation table of the error reporting instruction and the running log of the container engine, so that high efficiency is ensured; and the second stage confirms the operation log determined by the first stage by a user, so that the error reporting root cause can be accurately positioned, and the high accuracy of the abnormal diagnosis result is ensured. Specifically, obtaining all running logs of the container engine corresponding to the container engine error reporting instruction includes: checking whether the number of the error-reporting instructions of the container engine received in a preset first time period exceeds a preset number threshold value or not to obtain a checking result; if the verification result is exceeded, determining a first running log according to a preset mapping relation table of the error reporting instruction of the container engine and the running log, generating a running log checking instruction according to the error reporting instruction of the container engine and the first running log, and sending the running log checking instruction to a user side; if a checking error-free instruction sent by a user side is received, determining that the first running log is the running log, and continuously executing the step of acquiring all the running logs of the container engine corresponding to the error-free instruction of the container engine; if a correction running log instruction sent by the user side is received, determining that the running log specified in the correction running log instruction is the running log, and continuing to execute the step of acquiring all the running logs of the container engine corresponding to the error reporting instruction of the container engine.
In step 12 of the embodiment of the present invention, all variables in the running log are removed to obtain a first log template, see fig. 2, fig. 2 is a schematic diagram of generating the log template, the log content in fig. 2 may be equivalent to the running log in the embodiment of the present invention, the log template may be equivalent to the first log template in the embodiment of the present invention, and the log analysis may be equivalent to removing all variables in the running log in the embodiment of the present invention to obtain the first log template. Specifically, the running log of the container engine is typically composed of unstructured parts and structured parts, and after log parsing, the system log is extracted as a log template. The structured portion represents constant information in the log, the unstructured represents a variable, and after log parsing, the original log is parsed into a log template. Each log template is assigned a unique template number so that each log can be represented in place of its corresponding template number. As shown in fig. 2, the 1 st, 3 rd and 4 th logs may be represented by the same template numbers.
In step 13 of the embodiment of the present invention, after the first log template is obtained in step 12, feature extraction is performed on the first log template, that is, weighting processing is performed on different log information templates. Specifically, for each log template, its reverse text frequency is calculated, i.e., it is believed that a log template that frequently occurs in multiple log sequences will have a lower degree of discrimination than a log template that rarely occurs in log sequences. Reverse text frequency The formula of (2) is as follows:
。
wherein N is the total number of log sequences,for the number of log sequences that contain the existence of the log template t.
Further, referring to fig. 3, fig. 3 is a schematic diagram illustrating conversion of a vector sequence, where the template sequence may be equivalent to a first template sequence obtained by converting a first log template, each element in the first template sequence, that is, a field number corresponding to each field, and feature extraction is equivalent to a conversion process, and in this example, five different first log templates appear as vectorization examples in four log sequences. The first log sequence contains five templates, the second log sequence does not contain a No. 2 template, a No. 1 template does not exist in the third log sequence, and the fourth log sequence does not contain a No. 2 template and a No. 3 template (namely, the conversion of the first log template into the first vector sequence is equivalent to the conversion of the first log template into the first vector sequence in the embodiment of the invention, and the elements in the first vector sequence represent the reverse text frequency of each field in the first log template in a one-to-one correspondence).
In the embodiment of the invention, the training method of the running log abnormality diagnosis model comprises the following steps: acquiring a plurality of historical abnormal operation logs of a container engine; removing all variables in the historical abnormal operation logs to obtain an abnormal log template; converting the abnormal log template into a vector sequence, wherein elements in the vector sequence represent reverse text frequency of each field in the abnormal log template in a one-to-one correspondence manner; labeling the vector sequence with an abnormal type label, and combining all the labeled vector sequences to obtain a target training set; and training a diagnosis model by adopting a target training set to obtain an operation log abnormal diagnosis model. The training process is explained below in connection with specific examples:
Referring to fig. 2, fig. 2 is a schematic diagram illustrating generation of a log template, in this example, the log content in fig. 2 may be equivalent to a historical abnormal operation log in an embodiment of the present invention, the log template may be equivalent to an abnormal log template in an embodiment of the present invention, and log analysis may be equivalent to removing all variables in the historical abnormal operation log in an embodiment of the present invention to obtain an abnormal log template. Specifically, the running log of the container engine is typically composed of unstructured parts and structured parts, and after log parsing, the system log is extracted as a log template. The structured portion represents constant information in the log, the unstructured represents a variable, and after log parsing, the original log is parsed into a log template. Each log template is assigned a unique template number so that each log can be represented in place of its corresponding template number. As shown in fig. 2, the 1 st, 3 rd and 4 th logs may be represented by the same template numbers.
After log analysis, the different log information templates are weighted through feature extraction. Specifically, for each log template, its reverse text frequency is calculated, i.e., it is believed that a log template that frequently occurs in multiple log sequences will have a lower degree of discrimination than a log template that rarely occurs in log sequences. Reverse text frequency The formula of (2) is as follows:
。
wherein N is the total number of log sequences,for the number of log sequences that contain the existence of the log template t.
Further, referring to fig. 3, fig. 3 is a schematic diagram illustrating conversion of a vector sequence, in this example, the template sequence in fig. 3 corresponds to an abnormal log template, each element in the abnormal log template corresponds to a field number corresponding to each field in the abnormal log template, and feature extraction corresponds to a conversion process, in this example, five different log templates appear as vectorization examples in four log sequences. The first log sequence contains five templates, the second log sequence does not contain a No. 2 template, a No. 1 template does not exist in the third log sequence, and the fourth log sequence does not contain a No. 2 template and a No. 3 template (namely, the abnormal log template is converted into a vector sequence, and elements in the vector sequence are in one-to-one correspondence with reverse text frequencies of fields in the abnormal log template).
After the log sequence is obtained, the vector sequence can be marked by combining the prior knowledge enriched by the operation and maintenance personnel and the testers (namely, the vector sequence is marked with an abnormal type label, the marked vector sequences are combined to obtain a target training set), and different multi-classification algorithms are selected to train the fault diagnosis model (namely, the target training set is adopted to train the diagnosis model to obtain the operation log abnormal diagnosis model). The classification algorithm commonly used in machine learning is numerous, and the model in the example is efficient in selecting and convenient to implement, namely, a random forest. Random forests use an ensemble learning technique for multiple classification, which builds multiple decision trees during training, taking the mode of the individual tree output class as the final classification result of the algorithm.
The running log abnormality diagnosis model obtained by training by the training method can determine an abnormality diagnosis result with high accuracy according to the first vector sequence.
In the embodiment of the invention, all running logs of the container engine corresponding to the container engine error reporting instruction are obtained by receiving the container engine error reporting instruction; removing all variables in the operation log to obtain a first log template; converting the first log template into a first vector sequence, wherein elements in the first vector sequence represent reverse text frequencies of fields in the first log template in a one-to-one correspondence manner; the first vector sequence is input into a pre-trained running log abnormality diagnosis model to obtain an abnormality diagnosis result, and the embodiment of the invention avoids manual detection by operation and maintenance personnel and can realize high-efficiency diagnosis on a container engine; in addition, the embodiment of the invention adopts the operation log abnormality diagnosis model to realize abnormality diagnosis, and can keep high-accuracy fault diagnosis results for a long time.
In some embodiments of the invention, the method, optionally,
converting the first log template into a first vector sequence, comprising:
determining a field number corresponding to each field in the first log template, and converting the first log template into a first template sequence represented by the field number;
Generating a first template sequence checking instruction according to the first template sequence, and sending the first template sequence checking instruction to an interaction end associated with a user;
and receiving a first template sequence correct instruction sent by the interaction end, and converting the first template sequence into a first vector sequence.
The following is described in connection with examples:
for each log template, its reverse text frequency is calculated, i.e. it is believed that a log template that frequently occurs in multiple log sequences will have a lower degree of discrimination than a log template that rarely occurs in log sequences. Reverse text frequencyThe formula of (2) is as follows:
。
wherein N is the total number of log sequences,for containing dayThe number of log sequences in which the log template t exists.
Further, referring to fig. 3, fig. 3 is a schematic diagram illustrating conversion of a vector sequence, where the template sequence may be equivalent to a first template sequence obtained by converting a first log template, each element in the first template sequence, that is, a field number corresponding to each field, and feature extraction is equivalent to a conversion process, and in this example, five different first log templates are represented in vectorization examples in four log sequences. The first log sequence contains five templates, the second log sequence does not contain a No. 2 template, a No. 1 template does not exist in the third log sequence, and the fourth log sequence does not contain a No. 2 template and a No. 3 template (namely, the conversion of the first log template into the first vector sequence is equivalent to the conversion of the first log template into the first vector sequence in the embodiment of the invention, and the elements in the first vector sequence represent the reverse text frequency of each field in the first log template in a one-to-one correspondence).
In the embodiment of the invention, after a first template sequence is obtained, a first template sequence verification instruction is generated according to the first template sequence, and the first template sequence verification instruction is sent to an interaction end associated with a user; and receiving a first template sequence correct instruction sent by the interaction end, and converting the first template sequence into a first vector sequence. Because the first log template is converted into the first template sequence represented by the field number, the user can observe and confirm errors in the first log template more easily and intuitively, and the verification efficiency is improved. The embodiment of the invention ensures that the first vector sequence for abnormality diagnosis in the subsequent step 14 has high accuracy through verification, and ensures that the abnormality diagnosis result has high accuracy.
The embodiment of the invention provides a training method of an abnormality diagnosis model of a container engine, referring to fig. 4, fig. 4 is a flow diagram of the training method of the abnormality diagnosis model of the container engine according to the embodiment of the invention, and the training method comprises the following steps:
step 21: acquiring a plurality of historical abnormal operation logs of a container engine;
step 22: removing all variables in the historical abnormal operation logs to obtain an abnormal log template; converting the abnormal log template into a vector sequence, wherein elements in the vector sequence represent reverse text frequency of each field in the abnormal log template in a one-to-one correspondence manner;
Step 23: labeling the vector sequence with an abnormal type label, and combining all the labeled vector sequences to obtain a target training set; and training a diagnosis model by adopting a target training set to obtain an operation log abnormal diagnosis model.
In practical application, the historical abnormal operation log includes: a first abnormal operation log generated by the container engine operation and a second abnormal operation log generated by each container operation associated with the container engine.
The model training method of the embodiment of the invention is described below with reference to specific examples:
referring to fig. 2, fig. 2 is a schematic diagram illustrating generation of a log template, in this example, the log content in fig. 2 may be equivalent to a historical abnormal operation log in an embodiment of the present invention, the log template may be equivalent to an abnormal log template in an embodiment of the present invention, and log analysis may be equivalent to removing all variables in the historical abnormal operation log in an embodiment of the present invention to obtain an abnormal log template. Specifically, the running log of the container engine is typically composed of unstructured parts and structured parts, and after log parsing, the system log is extracted as a log template. The structured portion represents constant information in the log, the unstructured represents a variable, and after log parsing, the original log is parsed into a log template. Each log template is assigned a unique template number so that each log can be represented in place of its corresponding template number. As shown in fig. 2, the 1 st, 3 rd and 4 th logs may be represented by the same template numbers.
After log analysis, the different log information templates are weighted through feature extraction. Specifically, for each log template, its reverse text frequency is calculated, i.e., it is believed that a log template that frequently occurs in multiple log sequences will have a lower degree of discrimination than a log template that rarely occurs in log sequences. Reverse text frequencyThe formula of (2) is as follows:
。
wherein N is the total number of log sequences,for the number of log sequences that contain the existence of the log template t.
Further, referring to fig. 3, fig. 3 is a schematic diagram illustrating conversion of a vector sequence, in this example, the template sequence in fig. 3 corresponds to an abnormal log template, each element in the abnormal log template corresponds to a field number corresponding to each field in the abnormal log template, and feature extraction corresponds to a conversion process, in this example, five different log templates appear as vectorization examples in four log sequences. The first log sequence contains five templates, the second log sequence does not contain a No. 2 template, a No. 1 template does not exist in the third log sequence, and the fourth log sequence does not contain a No. 2 template and a No. 3 template (namely, the abnormal log template is converted into a vector sequence, and elements in the vector sequence are in one-to-one correspondence with reverse text frequencies of fields in the abnormal log template).
After the log sequence is obtained, the vector sequence can be marked by combining the prior knowledge enriched by the operation and maintenance personnel and the testers (namely, the vector sequence is marked with an abnormal type label, the marked vector sequences are combined to obtain a target training set), and different multi-classification algorithms are selected to train the fault diagnosis model (namely, the target training set is adopted to train the diagnosis model to obtain the operation log abnormal diagnosis model). The classification algorithm commonly used in machine learning is numerous, and the model in the example is efficient in selecting and convenient to implement, namely, a random forest. Random forests use an ensemble learning technique for multiple classification, which builds multiple decision trees during training, taking the mode of the individual tree output class as the final classification result of the algorithm.
The running log abnormality diagnosis model trained by the training method can determine an abnormality diagnosis result with high accuracy according to the first vector sequence.
In the embodiment of the invention, the abnormal log template is obtained by removing all variables in the historical abnormal operation log, the abnormal log template which can truly reflect the characteristics of the abnormal operation log is obtained, the abnormal log template is converted into a vector sequence, and then the vector sequence is marked with an abnormal type label, so that further characteristic extraction and marking are realized, and the operation log abnormal diagnosis model which can perform fault diagnosis with high accuracy is ensured to be obtained through training.
In some embodiments of the present invention, optionally, converting the exception log template into a vector sequence includes:
determining field numbers corresponding to each field in the abnormal log template, and converting the abnormal log template into a template sequence represented by the field numbers;
generating a template sequence checking instruction according to the template sequence, and sending the template sequence checking instruction to an interaction end associated with a user;
and receiving a template sequence correct instruction sent by the interaction end, and converting the template sequence into a vector sequence.
The following is described in connection with examples:
for each log template, its reverse text frequency is calculated, i.e. it is believed that a log template that frequently occurs in multiple log sequences will have a lower degree of discrimination than a log template that rarely occurs in log sequences. Reverse text frequencyThe formula of (2) is as follows:
。
wherein N is the total number of log sequences,for the number of log sequences that contain the existence of the log template t.
Further, referring to fig. 3, fig. 3 is a schematic diagram illustrating conversion of a vector sequence, where the template sequence may be equivalent to a template sequence obtained by converting an abnormal log template, each element in the template sequence, that is, a field number corresponding to each field, and feature extraction is equivalent to a conversion process, and in this example, five different abnormal log templates appear in vectorization examples in four log sequences. The first log sequence contains five templates, the second log sequence does not contain a No. 2 template, a No. 1 template does not exist in the third log sequence, and the fourth log sequence does not contain a No. 2 template and a No. 3 template (namely, the conversion of the abnormal log template into a vector sequence is equivalent to the conversion of the abnormal log template in the embodiment of the invention, and the elements in the vector sequence represent the reverse text frequency of each field in the abnormal log template in a one-to-one correspondence manner).
In the embodiment of the invention, after a template sequence is obtained, a template sequence verification instruction is generated according to the template sequence, and the template sequence verification instruction is sent to an interaction end associated with a user; and receiving a template sequence correct instruction sent by the interaction end, and converting the template sequence into a vector sequence. Because the abnormal log template is converted into the template sequence represented by the field number, the user can observe and confirm errors in the abnormal log template more easily and intuitively, and the verification efficiency is improved. The embodiment of the invention ensures that the vector sequence used for training in the step 23 has high accuracy through verification, avoids the problem of low accuracy of the model obtained through training caused by error and leakage of vector sequence conversion on abnormal diagnosis, and avoids the problem of increased training rounds caused by repeated reworking training due to low accuracy of model diagnosis, thereby improving training efficiency.
In some embodiments of the invention, the diagnostic model is optionally a random forest model.
In the embodiment of the invention, the diagnosis model is a random forest model, and the random forest is a model with high efficiency and convenient realization. Random forests use an ensemble learning technique for multiple classification, which builds multiple decision trees during training, taking the mode of the individual tree output class as the final classification result of the algorithm. According to the embodiment of the invention, under the condition that the random forest is ensured to meet the fault diagnosis requirement, the model complexity and the training difficulty are reduced by adopting the random forest, and the training efficiency can be effectively improved.
In some embodiments of the present invention, optionally, the historical abnormal operation log includes: a first abnormal operation log generated by the container engine operation and a second abnormal operation log generated by each container operation associated with the container engine. According to the embodiment of the invention, the first abnormal operation log generated by the operation of the container engine and the second abnormal operation log generated by the operation of each container associated with the container engine are used for model training, so that the abnormal diagnosis model obtained by training can be ensured to cover all directions in which the container engine is easy to report errors, and the abnormal diagnosis model obtained by training can be ensured to obtain a diagnosis result with high accuracy.
Referring to fig. 5, fig. 5 is a schematic block diagram of an abnormality diagnosis device for a container engine according to an embodiment of the present invention, and an abnormality diagnosis device 50 includes:
the receiving module 51 is configured to receive a container engine error reporting instruction, and obtain all running logs of the container engine corresponding to the container engine error reporting instruction;
the execution module 52 is configured to reject all variables in the running log to obtain a first log template;
the execution module 52 is further configured to convert the first log template into a first vector sequence, where elements in the first vector sequence represent reverse text frequencies of fields in the first log template in a one-to-one correspondence;
The execution module 52 is further configured to input the first vector sequence into a pre-trained log anomaly diagnosis model, to obtain an anomaly diagnosis result.
In some embodiments of the invention, the method, optionally,
the execution module 52 is further configured to determine a field number corresponding to each field in the first log template, and convert the first log template into a first template sequence represented by the field number;
the execution module 52 is further configured to generate a first template sequence verification instruction according to the first template sequence, and send the first template sequence verification instruction to an interaction end associated with a user;
the execution module 52 is further configured to receive a first template sequence correct instruction sent by the interaction end, and convert the first template sequence into the first vector sequence.
The abnormality diagnosis device for a container engine provided in the embodiment of the present application can implement each process implemented by the embodiments of the methods of fig. 1 to 3, and achieve the same technical effects, and for avoiding repetition, a detailed description is omitted herein.
Referring to fig. 6, fig. 6 is a schematic block diagram of a training device for an abnormality diagnosis model of a container engine according to an embodiment of the present invention, and a training device 60 includes:
An acquisition module 61 for acquiring a plurality of historical abnormal operation logs of the container engine;
the conversion module 62 is configured to reject all variables in the historical abnormal operation log for each historical abnormal operation log to obtain an abnormal log template; converting the abnormal log template into a vector sequence, wherein elements in the vector sequence represent reverse text frequencies of fields in the abnormal log template in a one-to-one correspondence manner;
the training module 63 is configured to label the vector sequence with an anomaly type label, and combine the completely labeled vector sequences to obtain a target training set; and training a diagnosis model by adopting the target training set to obtain an operation log abnormal diagnosis model.
In some embodiments of the invention, the method, optionally,
the conversion module 62 is further configured to determine a field number corresponding to each field in the abnormal log template, and convert the abnormal log template into a template sequence represented by the field number;
the conversion module 62 is further configured to generate a template sequence verification instruction according to the template sequence, and send the template sequence verification instruction to an interaction end associated with a user;
the conversion module 62 is further configured to receive a template sequence correct instruction sent by the interaction end, and convert the template sequence into the vector sequence.
In some embodiments of the invention, the method, optionally,
the diagnostic model is a random forest model.
In some embodiments of the invention, the method, optionally,
the history abnormal operation log includes: the first abnormal operation log generated by the container engine operation and the second abnormal operation log generated by each container operation associated with the container engine.
The training device for the abnormality diagnosis model of the container engine provided in the embodiment of the present application can implement each process implemented by the method embodiments of fig. 2 to fig. 4, and achieve the same technical effects, and in order to avoid repetition, a detailed description is omitted here.
An embodiment of the present invention provides an electronic device 70, referring to fig. 7, and fig. 7 is a schematic block diagram of the electronic device 70 according to an embodiment of the present invention, including a processor 71, a memory 72, and a program or an instruction stored in the memory 72 and executable on the processor 71, where the program or the instruction implements steps in the abnormality diagnosis method of any container engine of the present invention or implements steps in the training method of the abnormality diagnosis model of any container engine of the present invention when executed by the processor.
The embodiment of the present invention provides a readable storage medium, on which a program or an instruction is stored, where the program or the instruction, when executed by a processor, implements each process of the embodiment of the abnormality diagnosis method of the container engine according to any one of the above, or implements each process of the embodiment of the training method of the abnormality diagnosis model of the container engine according to any one of the above, and the same technical effects can be achieved, and for avoiding repetition, will not be described here again.
Wherein the readable storage medium is selected from Read-Only Memory (ROM), random access Memory (Random Access Memory RAM), magnetic disk or optical disk.
The embodiments of the present invention further provide a computer program product, which includes computer instructions, where the computer instructions, when executed by a processor, implement each process of the embodiment of the abnormality diagnosis method of the container engine according to any one of the foregoing, or implement each process of the embodiment of the training method of the abnormality diagnosis model of the container engine according to any one of the foregoing, and achieve the same technical effect, and for avoiding repetition, a detailed description is omitted herein.
The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present invention and the scope of the claims, which are to be protected by the present invention.
Claims (11)
1. A method for diagnosing an abnormality of a container engine, comprising:
Receiving a container engine error reporting instruction, and acquiring all operation logs of the container engine corresponding to the container engine error reporting instruction;
removing all variables in the operation log to obtain a first log template;
converting the first log template into a first vector sequence, wherein elements in the first vector sequence represent reverse text frequencies of fields in the first log template in a one-to-one correspondence manner;
and inputting the first vector sequence into a pre-trained running log abnormality diagnosis model to obtain an abnormality diagnosis result.
2. The abnormality diagnosis method of a container engine according to claim 1, characterized in that:
converting the first log template into a first vector sequence, comprising:
determining a field number corresponding to each field in the first log template, and converting the first log template into a first template sequence represented by the field number;
generating a first template sequence checking instruction according to the first template sequence, and sending the first template sequence checking instruction to an interaction end associated with a user;
and receiving a first template sequence correct instruction sent by the interaction end, and converting the first template sequence into the first vector sequence.
3. A method of training an anomaly diagnostic model for a container engine, comprising:
acquiring a plurality of historical abnormal operation logs of a container engine;
removing all variables in each historical abnormal operation log to obtain an abnormal log template; converting the abnormal log template into a vector sequence, wherein elements in the vector sequence represent reverse text frequencies of fields in the abnormal log template in a one-to-one correspondence manner;
labeling the vector sequence with an abnormal type label, and combining the vector sequences which are completely labeled to obtain a target training set; and training a diagnosis model by adopting the target training set to obtain an operation log abnormal diagnosis model.
4. A method of training an anomaly diagnostic model for a container engine according to claim 3, wherein:
converting the exception log template into a vector sequence, comprising:
determining field numbers corresponding to all fields in the abnormal log template, and converting the abnormal log template into a template sequence represented by the field numbers;
generating a template sequence checking instruction according to the template sequence, and sending the template sequence checking instruction to an interaction end associated with a user;
And receiving a template sequence correct instruction sent by the interaction end, and converting the template sequence into the vector sequence.
5. A method of training an anomaly diagnostic model for a container engine according to claim 3, wherein:
the diagnostic model is a random forest model.
6. A method of training an anomaly diagnostic model for a container engine according to claim 3, wherein:
the history abnormal operation log includes: the first abnormal operation log generated by the container engine operation and the second abnormal operation log generated by each container operation associated with the container engine.
7. An abnormality diagnosis device for a container engine, comprising:
the receiving module is used for receiving a container engine error reporting instruction and acquiring all running logs of the container engine corresponding to the container engine error reporting instruction;
the execution module is used for eliminating all variables in the running log to obtain a first log template;
the execution module is further used for converting the first log template into a first vector sequence, and elements in the first vector sequence represent reverse text frequencies of fields in the first log template in a one-to-one correspondence manner;
The execution module is further configured to input the first vector sequence into a running log anomaly diagnosis model that is trained in advance, so as to obtain an anomaly diagnosis result.
8. A training apparatus for an abnormality diagnostic model of a container engine, comprising:
the acquisition module is used for acquiring a plurality of historical abnormal operation logs of the container engine;
the conversion module is used for removing all variables in each historical abnormal operation log to obtain an abnormal log template; converting the abnormal log template into a vector sequence, wherein elements in the vector sequence represent reverse text frequencies of fields in the abnormal log template in a one-to-one correspondence manner;
the training module is used for labeling the abnormal type labels on the vector sequences, and combining the vector sequences which are completely labeled to obtain a target training set; and training a diagnosis model by adopting the target training set to obtain an operation log abnormal diagnosis model.
9. An electronic device, characterized in that: comprising a processor, a memory and a program or instructions stored on the memory and executable on the processor, which program or instructions when executed by the processor implement the steps in the method for diagnosing anomalies in a container engine according to any one of claims 1 to 2, or the steps in the method for training an anomaly diagnostic model for a container engine according to any one of claims 3 to 6.
10. A readable storage medium, characterized by: the readable storage medium has stored thereon a program or instructions which, when executed by a processor, implement the steps in the abnormality diagnosis method of the container engine according to any one of claims 1 to 2, or the steps in the training method of the abnormality diagnosis model of the container engine according to any one of claims 3 to 6.
11. A computer program product comprising computer instructions which, when executed by a processor, implement the steps in the method of diagnosing anomalies in a container engine according to any one of claims 1 to 2, or the steps in the method of training an anomaly diagnostic model for a container engine according to any one of claims 3 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410247168.6A CN117827620B (en) | 2024-03-05 | Abnormality diagnosis method, training device, training equipment, and recording medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410247168.6A CN117827620B (en) | 2024-03-05 | Abnormality diagnosis method, training device, training equipment, and recording medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117827620A true CN117827620A (en) | 2024-04-05 |
| CN117827620B CN117827620B (en) | 2024-05-10 |
Family
ID=
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113704201A (en) * | 2021-09-02 | 2021-11-26 | 国家电网有限公司信息通信分公司 | Log anomaly detection method and device and server |
| CN114647525A (en) * | 2020-12-21 | 2022-06-21 | 中兴通讯股份有限公司 | Diagnostic method, diagnostic device, terminal and storage medium |
| CN114844797A (en) * | 2022-05-27 | 2022-08-02 | 中国银行股份有限公司 | Call chain log and monitoring log association method and micro-service anomaly detection method |
| CN114968633A (en) * | 2022-04-14 | 2022-08-30 | 阿里巴巴（中国）有限公司 | Abnormal log detection method and device |
| US20220405592A1 (en) * | 2022-03-10 | 2022-12-22 | University Of Electronic Science And Technology Of China | Multi-feature log anomaly detection method and system based on log full semantics |
| CN115509797A (en) * | 2022-11-22 | 2022-12-23 | 北京优特捷信息技术有限公司 | Method, device, equipment and medium for determining fault category |
| CN115658360A (en) * | 2022-10-21 | 2023-01-31 | 华南理工大学 | Cloud system fault diagnosis method based on log data |
| CN115758183A (en) * | 2022-10-31 | 2023-03-07 | 通号城市轨道交通技术有限公司 | Training method and device for log anomaly detection model |
| CN115828180A (en) * | 2022-12-29 | 2023-03-21 | 北京邮电大学 | Log anomaly detection method based on analytic optimization and time sequence convolution network |
| CN116107834A (en) * | 2022-12-12 | 2023-05-12 | 浪潮通信信息系统有限公司 | Log abnormality detection method, device, equipment and storage medium |
| CN116167370A (en) * | 2023-02-08 | 2023-05-26 | 云南大学 | Log space-time characteristic analysis-based distributed system anomaly detection method |
| CN116244146A (en) * | 2023-02-28 | 2023-06-09 | 京东科技信息技术有限公司 | Log abnormality detection method, training method and device of log abnormality detection model |
| CN116502147A (en) * | 2023-04-23 | 2023-07-28 | 京东科技信息技术有限公司 | Training method of anomaly detection model and related equipment |
| CN117421595A (en) * | 2023-10-25 | 2024-01-19 | 广东技术师范大学 | System log anomaly detection method and system based on deep learning technology |
| CN117608889A (en) * | 2023-10-30 | 2024-02-27 | 北京邮电大学 | Log semantic based anomaly detection method and related equipment |
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114647525A (en) * | 2020-12-21 | 2022-06-21 | 中兴通讯股份有限公司 | Diagnostic method, diagnostic device, terminal and storage medium |
| CN113704201A (en) * | 2021-09-02 | 2021-11-26 | 国家电网有限公司信息通信分公司 | Log anomaly detection method and device and server |
| US20220405592A1 (en) * | 2022-03-10 | 2022-12-22 | University Of Electronic Science And Technology Of China | Multi-feature log anomaly detection method and system based on log full semantics |
| CN114968633A (en) * | 2022-04-14 | 2022-08-30 | 阿里巴巴（中国）有限公司 | Abnormal log detection method and device |
| CN114844797A (en) * | 2022-05-27 | 2022-08-02 | 中国银行股份有限公司 | Call chain log and monitoring log association method and micro-service anomaly detection method |
| CN115658360A (en) * | 2022-10-21 | 2023-01-31 | 华南理工大学 | Cloud system fault diagnosis method based on log data |
| CN115758183A (en) * | 2022-10-31 | 2023-03-07 | 通号城市轨道交通技术有限公司 | Training method and device for log anomaly detection model |
| CN115509797A (en) * | 2022-11-22 | 2022-12-23 | 北京优特捷信息技术有限公司 | Method, device, equipment and medium for determining fault category |
| CN116107834A (en) * | 2022-12-12 | 2023-05-12 | 浪潮通信信息系统有限公司 | Log abnormality detection method, device, equipment and storage medium |
| CN115828180A (en) * | 2022-12-29 | 2023-03-21 | 北京邮电大学 | Log anomaly detection method based on analytic optimization and time sequence convolution network |
| CN116167370A (en) * | 2023-02-08 | 2023-05-26 | 云南大学 | Log space-time characteristic analysis-based distributed system anomaly detection method |
| CN116244146A (en) * | 2023-02-28 | 2023-06-09 | 京东科技信息技术有限公司 | Log abnormality detection method, training method and device of log abnormality detection model |
| CN116502147A (en) * | 2023-04-23 | 2023-07-28 | 京东科技信息技术有限公司 | Training method of anomaly detection model and related equipment |
| CN117421595A (en) * | 2023-10-25 | 2024-01-19 | 广东技术师范大学 | System log anomaly detection method and system based on deep learning technology |
| CN117608889A (en) * | 2023-10-30 | 2024-02-27 | 北京邮电大学 | Log semantic based anomaly detection method and related equipment |
Non-Patent Citations (2)
| Title |
|---|
| JIN WANG ET AL.: "LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things", SENSORS, 26 April 2020 (2020-04-26), pages 5 - 10 * |
| XU ZHANG ET AL.: "Robust Log-Based Anomaly Detection on Unstable Log Data", PROCEEDINGS OF THE 2019 27TH ACM JOINT MEETING ON EUROPEAN SOFTWARE ENGINEERING CONFERENCE AND SYMPOSIUM ON THE FOUNDATIONS OF SOFTWARE ENGINEERING, 31 December 2019 (2019-12-31), pages 807 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102339252B (en) | Static state detecting system based on XML (Extensive Makeup Language) middle model and defect mode matching | |
| CN107862327B (en) | Security defect identification system and method based on multiple features | |
| CN111045927A (en) | Performance test evaluation method and device, computer equipment and readable storage medium | |
| CN113779590B (en) | Source code vulnerability detection method based on multidimensional characterization | |
| CN117827620B (en) | Abnormality diagnosis method, training device, training equipment, and recording medium | |
| CN111398703B (en) | Automatic test method and device for voltage analog-to-digital converter and storage medium | |
| US10152407B1 (en) | Optimization of analysis of automated test results | |
| CN117827620A (en) | Abnormality diagnosis method, training device, training equipment, and recording medium | |
| Huangfu et al. | System failure detection using deep learning models integrating timestamps with nonuniform intervals | |
| US20230351158A1 (en) | Apparatus, system and method for detecting anomalies in a grid | |
| CN115859191A (en) | Fault diagnosis method and device, computer readable storage medium and computer equipment | |
| CN113377962B (en) | Intelligent process simulation method based on image recognition and natural language processing | |
| CN115828264A (en) | Intelligent contract vulnerability detection method and system and electronic equipment | |
| CN114416417A (en) | System abnormity monitoring method, device, equipment and storage medium | |
| CN110929786B (en) | Data augmentation method and electronic equipment | |
| CN112464237A (en) | Static code safety diagnosis method and device | |
| CN110968518A (en) | Analysis method and device for automatic test log file | |
| CN113568662B (en) | Code change influence range analysis method and system based on calling relation | |
| CN117435511B (en) | Flow monitoring software testing method and device based on graphic neural network and storage medium | |
| Samarin et al. | Preventing SQL injection attacks by automatic parameterizing of raw queries using lexical and semantic analysis methods | |
| CN113726576B (en) | Method, device, equipment and storage medium for constructing network adaptation framework | |
| CN110764783A (en) | Method, device, equipment and storage medium for generating information acquisition tool | |
| CN116976443B (en) | Performance index debugging method, terminal equipment and computer storage medium | |
| CN115169164B (en) | Information processing method, device and equipment of industrial model | |
| CN116842128B (en) | Text relation extraction method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |