CN112084105A - Log file monitoring and early warning method, device, equipment and storage medium - Google Patents
Log file monitoring and early warning method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112084105A CN112084105A CN201910514720.2A CN201910514720A CN112084105A CN 112084105 A CN112084105 A CN 112084105A CN 201910514720 A CN201910514720 A CN 201910514720A CN 112084105 A CN112084105 A CN 112084105A
- Authority
- CN
- China
- Prior art keywords
- fault
- log file
- early warning
- feature information
- effective
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3604—Software analysis for verifying properties of programs
- G06F11/3612—Software analysis for verifying properties of programs by runtime analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/362—Software debugging
- G06F11/366—Software debugging using diagnostics
Abstract
The invention discloses a log file monitoring and early warning method, a device, equipment and a storage medium, wherein the method comprises the steps of obtaining a log file generated by the operation of a software product, reading effective characteristic information of the log file, vectorizing the effective characteristic information, judging problem characteristic attributes of the effective characteristic information based on the vectorized effective characteristic information, confirming a fault type and calculating a probability value of the fault, and providing a fault processing mode corresponding to the fault type and the probability value for a user. The invention realizes early warning of software product risks, can reduce the running risks of software products, reduces unnecessary loss of users and improves user experience.
Description
Technical Field
The invention relates to a monitoring and early warning method, in particular to a log file monitoring and early warning method, a device, equipment and a storage medium for realizing the method.
Background
Most software products in the market at present, such as various application programs, cannot avoid faults of jamming, abnormal operation, program crash and the like during operation, and the software products lack early warning capability for the faults, so that the problems cannot be solved in time by users. The current solution is to take relevant measures to remedy by analyzing the log files after error reporting. Since the software lacks the ability to predict these problems, it is difficult to avoid them from happening again, and it is a very poor experience, passive way for the user, and it cannot recover the user's losses that result from this. Therefore, how to timely and comprehensively warn users and give coping strategies when some faults are about to occur in software products becomes a technical problem to be solved at present.
Disclosure of Invention
In view of the above technical problems, the present invention provides a method, an apparatus, a device and a storage medium for monitoring and warning a log file, which at least solve the above-mentioned technical problems.
A log file monitoring and early warning method comprises the following steps: the method comprises the steps of obtaining a log file generated by software product operation, reading effective characteristic information of the log file, vectorizing the effective characteristic information, judging problem characteristic attributes of the effective characteristic information based on the vectorized effective characteristic information, confirming fault types, calculating probability values of faults, and providing a log file monitoring and early warning device corresponding to the fault types and the probability values for users, wherein the log file monitoring and early warning device comprises the following modules: the system comprises a text feature extraction module, a neural network model module and a scheme processing setting module, wherein the text feature extraction module is used for acquiring a log file generated by software product operation, reading effective feature information of the log file, vectorizing the effective feature information, judging problem feature attributes of the effective feature information based on the vectorized effective feature information, confirming fault types and calculating probability values of faults, and the scheme processing setting module is used for providing fault processing modes corresponding to the fault types and the probability values for users.
An electronic device comprising a processor, a memory for storing processor-executable instructions, wherein the processor is configured to perform the above method.
A tangible computer readable storage medium having instructions which, when executed by a processor, cause the processor to perform the above method.
The established neural network model establishes a set of text effective characteristic information of log files when various faults occur through training and learning, real-time monitoring of the log files in a software product can be realized by using the model, the type, probability and corresponding processing mode of the faults can be obtained by matching the extracted effective characteristic information of the log texts with the set in the neural network model, early warning of software product risks is realized, the running risk of the software product can be reduced, unnecessary loss of a user is reduced, and user experience is provided.
Drawings
Fig. 1 is a schematic flowchart of a log file monitoring and early warning method in an embodiment.
FIG. 2 is a flow diagram that illustrates extraction of valid feature information for a log file, under an embodiment.
FIG. 3 is a flow diagram of neural network model training learning, under an embodiment.
Fig. 4 is a flowchart illustrating an early warning information processing method according to an embodiment.
Fig. 5 is a block diagram of a log file monitoring and warning device in an embodiment.
FIG. 6 is a block flow diagram of a manner in which a scenario processing module processes in one embodiment.
FIG. 7 is a diagram illustrating the internal architecture of an electronic device in one embodiment.
Detailed Description
For the purpose of making the purpose and technical solutions of the present invention clearer, the present invention will be described in further detail with reference to the accompanying drawings and examples, it being understood that the specific examples described herein are only for better understanding of the present invention and are not intended to limit the present invention. The electronic device to which the present invention is applied includes a mobile phone, a tablet computer, a wearable device, and the like, which is not limited in this respect.
In one embodiment, as shown in fig. 1, a log file monitoring and early warning method is provided, which may include the following steps.
Step S101, starting the software product to run, and acquiring the log file generated by the running of the software product.
Step S101 is to obtain a log file generated during the operation of the software product in real time when the software product is started to operate.
In an embodiment, the step S101 may further include presetting a fault handling manner, and setting a fault handling manner list to correspond one fault type to one or more fault handling manners. Besides the default processing mode, the user can customize and extend the content of the processing mode according to the actual requirement.
The software product runs and outputs the log file, and if the output log file is a text file, the log file is firstly converted into a binary stream file. And then writing the converted log file into a cache region in a memory of the electronic equipment in a sectional reading mode until the log file cached in the cache region reaches a saturation state, and otherwise, continuously reading the output log file. In some embodiments, if the output log file is a binary stream file, the output log file is directly written into a cache area in the memory of the electronic device in a segmented reading manner without file conversion until the log file cached in the cache area reaches a saturation state, otherwise, the output log file is continuously read.
Step S102, extracting effective characteristic information of the log file, and vectorizing the extracted effective characteristic information.
The log file contains more information, and some invalid information, such as messy codes, special symbols and the like, is also contained in the log file. In order to improve the efficiency of text feature extraction, it is necessary to filter and remove the stream codes corresponding to the invalid information from the stream codes corresponding to the source log file, extract valid feature information from the filtered log file, and then extract valid feature information of the log file according to the feature structure.
In one embodiment, the feature structure defined in the present invention is composed of content, content length, and content value. The content in the feature structure may include, but is not limited to, memory occupancy, CPU occupancy, cycle markers, timestamp information, and the like. The length of the content is usually variable, and the value of the content can be a numerical value, a single word, a mark symbol and the like, and the value content is processed as a character string object. And extracting the content meeting the definition from the log file stream according to the characteristic structure body, packaging the content into a corresponding data structure object, and converting the data structure object into a vector object by using model tools such as Word2vec, FastText and the like. The feature structure body supports an extensible format because effective feature information of log files extracted by different fault types is different. Table 1 is an example of an expandable feature structure:
| content providing method and apparatus | Length of | Value taking |
| Time stamp | Variable | Time string |
| Valid feature information 1 | Variable | Content character string |
| Valid feature information 2 | Variable | Content character string |
| …… | Variable | Content character string |
| Significant feature information n | Variable | Content character string |
Step S103, in an embodiment, the number of the subsets of the corresponding fault feature set may be one or multiple, so that the calculated probability value may be one or multiple, and the probability value is a reference standard for determining whether a fault occurs, and is a predicted probability of the fault. If the valid feature information represented by the vector does not have the corresponding subset of the fault feature set, step S106 is executed.
Step S104, after receiving the data list of the fault types and the corresponding probability values returned by the neural network model, if the fault occurrence probability value of one or more items is greater than a first preset threshold (for example, 0, in other embodiments, the preset threshold may be dynamically adjusted or changed according to the actual situation), then step S105 is executed. If the probability value of one or more faults is equal to or smaller than the second preset threshold, step S106 is executed. It should be noted that the first preset threshold may be greater than the second preset threshold, and the first preset threshold may also be equal to the second preset threshold.
And S105, popping up an early warning prompt interface, displaying the fault type and the corresponding probability value, and providing a processing mode corresponding to the fault type and the probability value. If multiple fault types exist, the fault types are displayed in a table form. If the user selects one or more provided processing modes for processing aiming at the early warning prompt interface, the software product continues to run, and the processing is carried out according to the processing modes, and step S106 is executed after the processing is finished.
In an embodiment, the method may further include step S106, releasing the valid feature information represented by the vector from the buffer in the memory. This can improve the efficiency of extracting effective feature information.
In one embodiment, a method of extracting log file valid feature information is provided. As shown in fig. 2, according to an embodiment of the present invention, there is provided a method for extracting valid feature information from a log file, which includes the steps of:
step S201, obtaining the log text file, and unifying the encoding format of the text file, such as UTF-8 or GBK encoding format.
Step S202, the log text file with the unified coding format is converted into a binary stream file, so that the efficiency is improved when the effective characteristic information of the log file is extracted and the vector data is made.
And step S203, filtering invalid information in the converted binary stream file. Invalid information may include, but is not limited to, nonsense symbols, invalid words, uniformly encoded scrambling codes, non-timestamped information content, and the like. The filtering method may be a regular expression or the like, as long as the filtering method can perform a filtering function, and the present invention is not limited thereto.
And step S204, extracting effective characteristic information from the filtered binary stream file according to the characteristic structure body. The content value of the effective characteristic information can be a numerical value, a single word, a mark symbol and the like. The method of extraction adopts a general text processing method, such as regular expression, which is not limited by the present invention. The extracted content is encapsulated into corresponding data structure objects.
Step S205, converting the specific data structure object into a vector object by using model tools such as Word2vec, FastText and the like.
In one embodiment, as shown in FIG. 3, a method for neural network model training learning is provided, which may include the following steps.
Step S301, classifying the log files of different fault types, assuming that there are n types, and grouping the log files of the same fault type into a set, which is recorded as SnFor each set Si(i 1.. n) a training flag is added, which indicates that training is complete if the flag is 1, and indicates that training is not performed if the flag is 0.
Step S302, detecting SiIf the flag value is 0, go to step S303, if S isiIf the flag value of (i ═ 1.. n) is 1, step S307 is executed.
Step S303, extracting set SiOfIf log files exist and the number of the log files is recorded as q, each log file is recorded as Lk(k ═ 1.. q), and is LkAnd adding a training mark, wherein if the mark is 1, the training is finished, and if the mark is 0, the training is not finished.
Step S304, detecting LkIf the flag value is 0, step S305 is executed, and if the flag value is 1, step S302 is executed.
Step S305, initializing model parameters, inputting a vector object, namely vec-objs, executing a neural network model on the vec-objs, and calculating a loss value of a loss function.
Step S306, updating model parameters in a gradient descending mode and the like so as to minimize loss. Stopping model pair L when loss value reaches optimal solutionkAnd training of LkIs set to 1, step S304 is performed.
Step S307, the model training is stopped.
The Neural Network model is trained based on an RNN (Recurrent Neural Network) and a CNN (Convolutional Neural Network) in the AI field, and is circularly trained by inputting a training sample, executing an inference model on the training sample, calculating loss, adjusting model parameters and the like, and the training is stopped when the value of a loss function reaches the optimal solution. Therefore, the neural network model in the invention has the capability of identifying the fault type of the software product. Similarly, samples of multiple fault types are trained, and finally the neural network model in the invention has the capability of identifying multiple fault types.
In one embodiment, as shown in fig. 4, an early warning processing method is provided, which may include the following steps.
Step S401, reading a failure type result list transmitted by the neural network model, and popping up an early warning prompt box when the predicted occurrence probability of some failure (S) is greater than a preset threshold (e.g., 0, although in other embodiments, the preset threshold may be dynamically adjusted or changed according to actual conditions).
In step S402, the content prompted by the warning prompt box includes, but is not limited to, a fault type name, an occurrence probability, one or more processing methods, and an ignore button. When the ignore button is clicked (for example, the user clicks the ignore button), the prompt box is closed and all the contents of the pre-warning are skipped, otherwise, step S403 is executed. It should be noted that when the ignore button is not clicked and the warning prompt box is closed, all contents of the warning are also skipped.
And S403, when one or more processing modes are clicked, performing corresponding processing according to the content of the processing mode, and closing the early warning prompt box after the clicked one or more processing modes are completely executed.
In one embodiment, as shown in fig. 5, a flow diagram of a log file monitoring and warning device is provided, which may include, but is not limited to: the system comprises a text feature extraction module 501, a neural network model module 502, an early warning processing module 503 and a scheme processing setting module 504.
The text feature extraction module 501 is configured to extract effective feature information of the log file, and vectorize the effective feature information.
In one embodiment, the text feature extraction module 501 may convert the valid feature information into a vector object using a model tool such as Word2vec, FastText, and the like.
In one embodiment, further, the text feature extraction module 501 may also release the valid feature information represented by the vector from a buffer in a memory of an electronic device. This can improve the efficiency of extracting effective feature information.
The neural network model module 502 is configured to determine the problem feature attribute of the effective feature information by using a preset neural network model, determine the fault type, and calculate the probability value of the fault. The neural network model completes classification of different fault features through training and learning, establishes a fault feature set corresponding to the classification according to the completed classification, matches a subset of the fault feature set corresponding to effective feature information represented by a received vector by utilizing a forward propagation algorithm, and calculates a probability value of fault occurrence according to the coverage rate of the subset of the fault feature set if the subset of the fault feature set corresponding to the effective feature information represented by the vector exists.
The early warning processing module 503 is configured to pop up the early warning prompt interface to the user; the early warning prompt interface can be displayed on a display device of the electronic equipment.
The scenario processing setup module 504 is configured to provide a processing manner corresponding to the fault type and the probability value. It should be noted that, the electronic device or the corresponding server has a preset fault handling manner, and one fault type corresponds to one or more fault handling manners by setting a fault handling manner list. Besides the default processing mode, the user can customize and extend the content of the processing mode according to the actual requirement.
In one embodiment, as shown in fig. 6, a flow diagram of a processing manner of the solution processing module is provided, and specific default processing manners include, but are not limited to, forcibly stopping the software product, stopping the problem task, actively releasing the memory space, and disconnecting the network connection. The resolvable failure includes, but is not limited to, the problems of system product breakdown, memory leakage, system performance abnormity, network attack and the like. Because the fault types are various and the appropriate processing modes are different, the scheme processing module provided by the invention supports custom extension and increases the processing modes except default setting.
In one embodiment, as shown in fig. 7, an electronic device is provided, which includes a processor and a memory, wherein the processor is configured to implement the log file monitoring and warning method in any one of the above embodiments, and the memory is configured to store processor executable instructions.
It can be understood by those skilled in the art that the structure shown in fig. 7 is only a partial structural block diagram related to the present application, and does not constitute a limitation to the electronic device to which the present application is applied. A particular electronic device may include more components than shown, or combine certain components, or have a different arrangement of components.
In one embodiment, a computer-readable storage medium is also provided such that the steps in the above-described method embodiments may be implemented when the instructions in the storage medium are executed by a processor. The computer storage medium may be a Memory such as a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a magnetic Random Access Memory (FRAM), a Flash Memory (Flash Memory), a magnetic surface Memory, an optical Disc, or a Compact Disc Read-Only Memory (CD-ROM); and may be various electronic devices such as mobile phones, computers, tablet devices, personal digital assistants, etc., including one or any combination of the above-mentioned memories.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, recitation of an element by the phrase "comprising an … …" does not exclude the presence of other like elements in the process, method, article, or apparatus that comprises the element. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only. Therefore, the protection scope of the present invention should be subject to the appended claims.
Claims (10)
1. A log file monitoring and early warning method comprises the following steps:
acquiring a log file generated by the operation of a software product, extracting effective characteristic information of the log file, and vectorizing the effective characteristic information;
judging the problem feature attribute of the effective feature information based on the vectorized effective feature information, and confirming the corresponding fault type and calculating the probability value of the fault;
popping up the early warning prompt interface to a user;
and outputting a fault processing mode corresponding to the fault type and the probability value.
2. The method of claim 1, wherein extracting valid feature information of the log file comprises:
converting the log file into a binary stream file;
filtering invalid contents in the binary stream file to obtain a filter file;
extracting effective characteristic information of the filter file according to a preset characteristic structure;
and obtaining effective characteristic information of the log file.
3. The method according to claim 1, wherein the determining the problem feature attribute of the valid feature information based on the vectorized valid feature information includes determining by using a preset neural network model in combination with the vectorized valid feature information, and specifically includes:
training the neural network model to complete classification of different fault characteristics, and establishing a fault characteristic set corresponding to the classification according to the completed classification;
matching a subset in the fault feature set corresponding to the effective feature information;
and when the matching is successful, calculating the probability value of the fault according to the coverage rate of the subset, or when the matching is unsuccessful, releasing the effective characteristic information.
4. The method of claim 1, further comprising:
popping up an early warning prompt interface on a display device, and displaying the fault type, the probability value and the processing mode on the prompt interface;
and when the prompt interface is selected to be ignored or closed, the software product continues to run.
5. A log file monitoring and early warning device is characterized by comprising:
the text feature extraction module is used for acquiring a log file generated by software product operation, reading effective feature information of the log file and vectorizing the effective feature information;
the fault judgment module is used for judging the problem characteristic attribute of the effective characteristic information based on the vectorized effective characteristic information, confirming the fault type and calculating the probability value of the fault;
and the scheme processing and setting module is used for providing a fault processing mode corresponding to the fault type and the probability value for the user.
6. The apparatus according to claim 5, wherein the text feature extraction module is further configured to convert the log file into a binary stream file, filter invalid contents in the binary stream file to obtain a filter file, and extract valid feature information of the filter file according to a preset feature structure.
7. The apparatus according to claim 5, wherein the fault determining module includes a neural network model, the neural network model is trained to complete classification of different fault features, a fault feature set corresponding to the classification is established according to the completed classification, a subset of the fault feature set corresponding to the valid feature information is matched, if the matching is successful, a probability value of occurrence of a fault is calculated according to a coverage rate of the subset, and if the matching is unsuccessful, the valid feature information is released.
8. The device of claim 5, further comprising an early warning processing module, configured to pop up an early warning prompt interface to a user, display the fault type, the probability value, and the processing manner on the prompt interface, and when the prompt interface is selected to be ignored or turned off, the software product continues to run.
9. An electronic device comprising a processor and a memory for storing processor-executable instructions, wherein the processor is configured to perform the steps of the method of any one of claims 1 to 4.
10. A computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor, cause the processor to perform the steps of the method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910514720.2A CN112084105A (en) | 2019-06-13 | 2019-06-13 | Log file monitoring and early warning method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910514720.2A CN112084105A (en) | 2019-06-13 | 2019-06-13 | Log file monitoring and early warning method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112084105A true CN112084105A (en) | 2020-12-15 |
Family
ID=73733854
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910514720.2A Pending CN112084105A (en) | 2019-06-13 | 2019-06-13 | Log file monitoring and early warning method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112084105A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113220537A (en) * | 2021-06-02 | 2021-08-06 | 上海节卡机器人科技有限公司 | Software monitoring method, device, equipment and readable storage medium |
| CN113282433A (en) * | 2021-06-10 | 2021-08-20 | 中国电信股份有限公司 | Cluster anomaly detection method and device and related equipment |
-
2019
- 2019-06-13 CN CN201910514720.2A patent/CN112084105A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113220537A (en) * | 2021-06-02 | 2021-08-06 | 上海节卡机器人科技有限公司 | Software monitoring method, device, equipment and readable storage medium |
| CN113282433A (en) * | 2021-06-10 | 2021-08-20 | 中国电信股份有限公司 | Cluster anomaly detection method and device and related equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20190163742A1 (en) | Method and apparatus for generating information | |
| CN107315810B (en) | Internet of things equipment behavior portrait method | |
| CN109284371B (en) | Anti-fraud method, electronic device, and computer-readable storage medium | |
| CN115618371B (en) | Non-text data desensitization method, device and storage medium | |
| CN112765003B (en) | Risk prediction method based on APP behavior log | |
| CN112084105A (en) | Log file monitoring and early warning method, device, equipment and storage medium | |
| CN115438650B (en) | Contract text error correction method, system, equipment and medium fusing multi-source characteristics | |
| CN111145202B (en) | Model generation method, image processing method, device, equipment and storage medium | |
| CN107992508B (en) | Chinese mail signature extraction method and system based on machine learning | |
| CN115858776B (en) | Variant text classification recognition method, system, storage medium and electronic equipment | |
| CN107045497A (en) | A kind of quick newsletter archive content sentiment analysis system and method | |
| CN116644183A (en) | Text classification method, device and storage medium | |
| CN110598115A (en) | Sensitive webpage identification method and system based on artificial intelligence multi-engine | |
| CN110727743A (en) | Data identification method and device, computer equipment and storage medium | |
| CN115270789A (en) | Abnormal voice data detection method and device and related equipment | |
| CN115080745A (en) | Multi-scene text classification method, device, equipment and medium based on artificial intelligence | |
| CN113688240A (en) | Threat element extraction method, device, equipment and storage medium | |
| JP2018132965A (en) | Fault analysis program, fault analysis device, and fault analysis method | |
| CN113590436A (en) | Alarm method and device | |
| CN112766052A (en) | CTC-based image character recognition method and device | |
| CN115718696B (en) | Source code cryptography misuse detection method and device, electronic equipment and storage medium | |
| CN113872852B (en) | Outgoing mail monitoring method and device, electronic equipment and storage medium | |
| CN117082021B (en) | Mail intervention method, device, equipment and medium | |
| CN109408795B (en) | Text recognition method, text recognition equipment, computer readable storage medium and device | |
| CN116257856A (en) | Source code detection method, source code detection device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |