CN113176978B - Monitoring method, system, equipment and readable storage medium based on log file - Google Patents
Monitoring method, system, equipment and readable storage medium based on log file Download PDFInfo
- Publication number
- CN113176978B CN113176978B CN202110483490.5A CN202110483490A CN113176978B CN 113176978 B CN113176978 B CN 113176978B CN 202110483490 A CN202110483490 A CN 202110483490A CN 113176978 B CN113176978 B CN 113176978B
- Authority
- CN
- China
- Prior art keywords
- log file
- target
- monitoring
- log
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a monitoring method based on log files, which comprises the following steps: collecting log files of each terminal in a target terminal cluster in real time and storing the log files in a message queue; sequentially pulling a target log file conforming to a target field from the message queue, and extracting a content field in the target log file; matching the content fields with a preset monitoring rule to screen out matched target log files, so as to obtain a target log file set; and analyzing the target log file set according to a preset monitoring rule, and generating a monitoring result according to an analysis result. The invention also discloses a monitoring system, equipment and readable storage medium based on the log file. The embodiment of the invention can flexibly monitor the terminal cluster.
Description
Technical Field
The present invention relates to the field of log monitoring technologies, and in particular, to a log file-based monitoring method, system, device, and readable storage medium.
Background
With the maturity of computer technology, although the application program running on the intelligent device is more and more perfect, the situation of execution error is unavoidable in the actual running process. Therefore, a developer usually needs to record the running state and the operation content of the application program through a log system so as to be convenient for a person to check and take as the basis of debugging equipment. The log records various running states and operation information of the application program and generates a log file. However, when a problem occurs in the running process of the application program, a long time is often required to find the cause of the problem from the log file, so that a monitoring system for monitoring the log file is added, and a good monitoring system can accurately find the problem according to the log file when the problem occurs in the application program and give an alarm in real time to remind a developer.
In the prior art, the monitoring system monitors based on the host level, that is, a monitoring program is directly implanted into a monitored terminal, and then the monitoring program executes the monitoring of the log file of the terminal and returns result data; in addition, if the monitored object is a cluster including a plurality of terminals, monitoring needs to be performed on each terminal in the cluster in turn, and when the monitoring rule of the monitoring program needs to be adjusted, the monitoring program on each terminal needs to be updated one by one. Therefore, the existing monitoring mode is inconvenient for self-help configuration and is difficult to maintain in practical application.
Disclosure of Invention
The invention provides a monitoring method, a system, equipment and a readable storage medium based on a log file, which can solve the problems that log monitoring cannot be flexibly set and is low in efficiency in the prior art.
In order to achieve the above object, the present invention provides a log file-based monitoring method, which includes:
collecting log files of each terminal in a target terminal cluster in real time and storing the log files in a message queue;
sequentially pulling a target log file conforming to a target field from the message queue, and extracting a content field in the target log file;
matching the content fields with a preset monitoring rule to screen out matched target log files, so as to obtain a target log file set;
and analyzing the target log file set according to a preset monitoring rule, and generating a monitoring result according to an analysis result.
Further, before collecting log file data of each terminal in the target terminal cluster in real time and storing the log file data in the message queue, the method includes:
the control monitoring program collects log files generated by application programs on each terminal according to preset data acquisition rules, wherein the data acquisition rules comprise real-time acquisition rules;
the log file is stored in each terminal in the target terminal cluster.
Further, the collecting, in real time, the log file of each terminal in the target terminal cluster and storing the log file in the message queue includes:
a log acquisition instruction is sent to the monitoring program so as to control the monitoring program to acquire log files on each terminal according to the log acquisition instruction;
and storing the collected log file into a preset message queue.
Further, the analyzing the target log file set according to the preset monitoring rule, and generating the monitoring result according to the analysis result includes:
acquiring log files with the same log file names in the target log file set;
summing up and calculating the occurrence times of the log files with the same log file name;
and if the occurrence times of the log files meet preset monitoring rules, generating monitoring results of the terminals corresponding to the log files.
Further, the matching the content field with the preset monitoring rule to screen out a matched target log file, and obtaining a target log file set includes:
fuzzy matching is carried out on the content fields of each target log file based on a preset regular expression so as to extract candidate fields;
and identifying the candidate fields through a sensitive information identification rule, screening out target log files corresponding to the matched candidate fields, and obtaining a target log file set, wherein the preset monitoring rule comprises the preset regular expression and the sensitive information identification rule.
Further, after analyzing the target log file set according to a preset monitoring rule and generating a monitoring result according to the analysis result, the method further includes:
storing the analysis result in a time sequence database;
and sending the analysis result generation report to the target user for checking through the time sequence database.
Further, after analyzing the target log file set according to a preset monitoring rule and generating a monitoring result according to the analysis result, the method further includes:
acquiring a message channel of a target user from a configuration management center;
and sending the monitoring result to the target user for viewing through the message channel.
To achieve the above object, the present invention provides a log file-based monitoring system, the system comprising:
the acquisition module is used for acquiring log files of each terminal in the target terminal cluster in real time and storing the log files into the message queue;
the extraction module is used for sequentially extracting the log files conforming to the target fields from the message queue and extracting the content fields in the log files;
the matching module is used for matching the content fields with a preset monitoring rule so as to screen out matched log files and obtain a target log file set;
and the analysis module is used for analyzing the target log file set according to a preset monitoring rule and generating a monitoring result according to an analysis result.
To achieve the above object, the present invention provides a computer device including a memory, a processor, on which a computer program is stored which is executable on the processor, the computer program, when executed by the processor, implementing the steps of the log file based monitoring method as described above.
To achieve the above object, the present invention provides a computer-readable storage medium having stored therein a computer program executable by at least one processor to cause the at least one processor to perform the steps of the log file-based monitoring method as described above.
Compared with the prior art, the log file-based monitoring method, system, equipment and readable storage medium can collect log files of the terminal cluster, store the log files in the message queue for shunting treatment, screen the log files in the message queue, and screen the target log files including the matching of the content fields of the log files and the analysis of preset monitoring rules so as to monitor the terminal through the analysis of the log files; the log files of the terminal clusters are stored through the message queues, and then the target logs are processed through the distributed computing framework, so that the efficiency of log file monitoring is improved, and the log files with large data level can be monitored.
Drawings
Fig. 1 is a flowchart of a first embodiment of a log file-based monitoring method according to the present invention.
Fig. 2 is a schematic program module diagram of a second embodiment of the log file-based monitoring system according to the present invention.
Fig. 3 is a schematic diagram of a hardware structure of a third embodiment of the computer device of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that the description of "first", "second", etc. in this disclosure is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implying an indication of the number of technical features being indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present invention.
Example 1
Referring to fig. 1, a flowchart illustrating steps of a log file-based monitoring method according to a first embodiment of the present invention is shown. It will be appreciated that the flow charts in the method embodiments are not intended to limit the order in which the steps are performed. An exemplary description will be made below with the computer device 2 as an execution subject. Specifically, the following is described.
Step S100, collecting log files of each terminal in the target terminal cluster in real time and storing the log files in a message queue.
Specifically, the log collecting client collects log files of a plurality of terminals, the collected log files comprise terminal names, log file paths, log contents and the like of each terminal, and the log files are stored in the message queue, so that subsequent data calculation and data screening are facilitated. The message queue is a preset storage unit for storing the log files on each terminal in a sequence according to the time sequence. The message queue is a distributed publish-subscribe message system Kafka, log file data can be stored and consumed in real time, the storage time is set to be 2 hours, when the log file is consumed, namely, the log file disappears from the message queue, and the log file which is not consumed can be stored in the message queue for two hours; when the data volume is too large, the kafka system can expand capacity, and the transverse data storage volume and the longitudinal data storage time can be increased according to actual needs. The log files collected by the log collection client are transferred to the message queue of the kafka system in real time.
Illustratively, before the step S100, the method further includes:
the control monitoring program collects log files generated by application programs on each terminal according to preset data acquisition rules, wherein the data acquisition rules comprise real-time acquisition rules; the log file is stored in each terminal in the target terminal cluster.
Specifically, the log collection client is in communication connection with each terminal in the target terminal cluster, an application is installed on the terminal, and then a monitoring program can be set in each terminal in advance. The monitoring program may collect log files generated by the application program on the terminal according to a pre-configured data collection rule, for example, the data collection rule is a timing collection or a periodic collection. The monitor may be a log collection client. Then, the log collection client acquires the log file collected by the monitoring program from each terminal and stores the log file in a preset message queue.
Illustratively, the step S100 includes:
step S101, a log acquisition instruction is sent to the monitoring program so as to control the monitoring program to acquire log files on each terminal according to the log acquisition instruction; step S102, storing the collected log file into a preset message queue.
Specifically, a log collection instruction is sent to a monitoring program of a target terminal cluster through a monitoring platform so as to collect log files generated by application programs on each terminal in real time and obtain log files collected in a historical time period from the terminal.
Step S120, sequentially pulling the target log file conforming to the target field from the message queue, and extracting the content field in the target log file.
Specifically, a log file meeting a target field in a message queue is obtained, the target field is a target application name of an application to be monitored and a target log file path of the log file to be monitored, the application is installed on a target terminal cluster, the target log file path can be arranged on a memory of each terminal, and the target log file path stored in the log file can be queried through the application name. And pulling the target log file of the application needing to be monitored in the information queue, and screening out the target log file through the target field. And extracting the content field of the log file and screening again. The content field is any field in the log file, and may be all the fields of each log file.
Illustratively, the target field may be pre-stored in a database, which may be a mysql database, that is acquired first when pulling the log file in the information queue.
Step S140, matching the content fields with a preset monitoring rule to screen out matched target log files, and obtaining a target log file set.
Specifically, the preset monitoring rules may be key fields and white lists, where the key fields are sensitive information that needs to be monitored, for example: private sensitive information such as an identity card number, a mobile phone number, a bank card number and the like, and a preset field or characters are in a white list. The log files in the information queue can be pulled through the distributed computing framework to be screened, the distributed computing framework comprises a plurality of servers, huge amounts of data can be processed at the same time, the distributed computing framework applies monitoring rules to match real-time log files in the information queue to generate a target log file set, the target log file set is composed of target log files screened by the monitoring rules, the log files can be classified according to the monitoring rules, the operation amount is reduced, huge amounts of data above one hundred million levels are filtered by the monitoring rules to reduce the data amount, and centralized management of the log files is facilitated.
Illustratively, the step S140 includes:
step S141, fuzzy matching is carried out on the content field of each target log file based on a preset regular expression so as to extract candidate fields; step S142, identifying the candidate fields through a sensitive information identification rule, screening out target log files corresponding to the matched candidate fields, and obtaining a target log file set, wherein the preset monitoring rule comprises the preset regular expression and the sensitive information identification rule.
Specifically, screening content fields of target log files in target log data sets, screening candidate target log file sets corresponding to the content fields containing key fields based on regular expressions, and screening target log files which do not contain fields or characters in a white list in the candidate target log file sets based on the white list to serve as the target log file sets. For example: screening out the key field of the identity card number in the content field to serve as a candidate target log file set, and then checking whether the candidate target log file set contains the white list field of error, filtering if the candidate target log file set contains the white list field of error, and taking the candidate target log file set as a target log data set if the candidate target log file set does not contain the white list field of error. The monitoring rules are not limited to preset regular expressions and sensitive information identification rules.
Step S160, analyzing the target log file set according to a preset monitoring rule, and generating a monitoring result according to an analysis result.
Specifically, the target log file set may be stored in a redis database, a monitoring program is set in the redis database in advance to monitor the redis database, whether the target log files in the target log file set meet a preset condition is monitored, when the preset condition is met, an alarm instruction is triggered, alarm information is generated according to the alarm instruction, and the alarm information is a monitoring result. And sending the alarm information to the target user. The monitoring program presets preset conditions, the preset conditions are limiting values of the occurrence times of the same log file, the preset conditions are met, namely, the occurrence times reach the limiting values, an alarm is triggered, alarm information is generated, for example, the preset conditions are that the occurrence times within 10 minutes are limiting values of 20 times, and if the occurrence times are met, the alarm is triggered, and the alarm information is generated.
Illustratively, the step S160 includes:
step S161, obtaining log files with the same log file names in the target log file set; step S162, summing up and calculating the occurrence times of the log files with the same log file names; step S163, if the occurrence number of the log file meets the preset monitoring rule, generating a monitoring result of the terminal corresponding to the log file.
Specifically, the monitoring rules comprise fault alarm monitoring rules, wherein the fault alarm monitoring rules are fault contents set according to monitoring requirements, and the function of the fault alarm monitoring rules is to screen out log files containing fault information meeting the fault alarm monitoring rules, namely preset conditions, from the log files. If the log file does not contain the fault content set in the fault monitoring rule, the log file is a normal log file, otherwise, the log file is a fault file, and the number of times of occurrence of one fault file is increased by one, or the log file can be directly summed. The monitoring result comprises an alarm signal sent by an alarm program and a log file containing sensitive information, wherein the alarm signal content contains the fault reporting times of faults and the fault log file information, and the log file information contains information such as the name of a target terminal, the occurrence position of the log file and the like.
Illustratively, after the step S160, the method further includes:
step S171, storing the analysis result in a time sequence database; and step S172, sending the analysis result generation report to the target user for viewing through the time sequence database.
Specifically, the analysis result can be stored in a time sequence type database, the time sequence type database is preferably an influxdb database, and a report generation program of the influxdb database generates a sensitive information report from a log file containing sensitive information, so that report display is performed, and convenience is brought to users to check problems.
Illustratively, after the step S160, the method further includes:
step S181, obtaining a message channel of a target user from a configuration management center; and step S182, the monitoring result is sent to the target user for viewing through the message channel.
Specifically, a message channel sent to a target user is prestored in the Cmdb database, and alarm information is sent to the target user through the message channel, so that the purpose of monitoring the application through the log file is achieved. The target user may be queried via CMDB (configuration management database, configuration Management Database, CMDB) and the alert information may be sent to the target user. When sending an alarm signal to a target user, the alarm signal can be sent by mail or short message; and recording the alarm signal notification to the abnormal data report. The database of the monitoring platform is provided with a statistical program, and the statistical program generates daily reports, weekly reports and monthly reports according to the abnormal data report and the sensitive information report and counts the historical error reporting amount and the current error reporting amount. And the automatic management background selectively sends the abnormal data report and the sensitive information report to different users, so that the users with opposite business can master related log problems.
Example two
With continued reference to fig. 2, a schematic program module of a second embodiment of the log file-based monitoring system of the present invention is shown. In this embodiment, the log file-based monitoring system 20 may include or be divided into one or more program modules, which are stored in a storage medium and executed by one or more processors to accomplish the present invention and implement the log file-based monitoring method described above. Program modules in accordance with embodiments of the present invention refer to a series of computer program instruction segments capable of performing particular functions, and are more suitable than programs themselves for describing the execution of the log file-based monitoring system 20 in a storage medium. The following description will specifically describe functions of each program module of the present embodiment:
and the acquisition module 200 is used for acquiring the log file of each terminal in the target terminal cluster in real time and storing the log file in the message queue.
Specifically, the log collecting client collects log files of a plurality of terminals, the collected log files comprise terminal names, log file paths, log contents and the like of each terminal, and the log files are stored in the message queue, so that subsequent data calculation and data screening are facilitated. The message queue is a preset storage unit for storing the log files on each terminal in a sequence according to the time sequence. The message queue is a distributed publish-subscribe message system Kafka, log file data can be stored and consumed in real time, the storage time is set to be 2 hours, when the log file is consumed, namely, the log file disappears from the message queue, and the log file which is not consumed can be stored in the message queue for two hours; when the data volume is too large, the kafka system can expand capacity, and the transverse data storage volume and the longitudinal data storage time can be increased according to actual needs. The log files collected by the log collection client are transferred to the message queue of the kafka system in real time.
Illustratively, the acquisition module 200 is further configured to:
a log acquisition instruction is sent to the monitoring program so as to control the monitoring program to acquire log files on each terminal according to the log acquisition instruction; and storing the collected log file into a preset message queue.
Specifically, a log acquisition instruction is sent to a monitoring program of a target terminal cluster through a monitoring platform, log files generated by application programs on each terminal are acquired in real time, and log files acquired in a historical time period are acquired from the terminal.
And the extracting module 202 is used for sequentially extracting the target log files conforming to the target fields from the message queue and extracting the content fields in the target log files.
Specifically, a log file meeting a target field in a message queue is obtained, the target field is a target application name of an application to be monitored and a target log file path of the log file to be monitored, the application is installed on a target terminal cluster, the target log file path can be arranged on a memory of each terminal, and the target log file path stored in the log file can be queried through the application name. And pulling the target log file of the application needing to be monitored in the information queue, and screening out the target log file through the target field. And extracting the content field of the log file and screening again. The content field is any field in the log file, and may be all the fields of each log file.
Illustratively, the target field may be pre-stored in a database, which may be a mysql database, that is acquired first when pulling the log file in the information queue.
And the matching module 204 is configured to match the content field with a preset monitoring rule, so as to screen out a matched target log file, and obtain a target log file set.
Specifically, the preset monitoring rules may be key fields and white lists, where the key fields are sensitive information that needs to be monitored, for example: private sensitive information such as an identity card number, a mobile phone number, a bank card number and the like, and a preset field or characters are in a white list. The log files in the information queue can be pulled through the distributed computing framework to be screened, the distributed computing framework comprises a plurality of servers, huge amounts of data can be processed at the same time, the distributed computing framework applies monitoring rules to match real-time log files in the information queue to generate a target log file set, the target log file set is composed of target log files screened by the monitoring rules, the log files can be classified according to the monitoring rules, the operation amount is reduced, huge amounts of data above one hundred million levels are filtered by the monitoring rules to reduce the data amount, and centralized management of the log files is facilitated.
Illustratively, the matching module 204 is further configured to:
fuzzy matching is carried out on the content fields of each target log file based on a preset regular expression so as to extract candidate fields; and identifying the candidate fields through a sensitive information identification rule, screening out target log files corresponding to the matched candidate fields, and obtaining a target log file set, wherein the preset monitoring rule comprises the preset regular expression and the sensitive information identification rule.
Specifically, screening content fields of target log files in target log data sets, screening candidate target log file sets corresponding to the content fields containing key fields based on regular expressions, and screening target log files which do not contain fields or characters in a white list in the candidate target log file sets based on the white list to serve as the target log file sets. For example: screening out the key field of the identity card number in the content field to serve as a candidate target log file set, and then checking whether the candidate target log file set contains the white list field of error, filtering if the candidate target log file set contains the white list field of error, and taking the candidate target log file set as a target log data set if the candidate target log file set does not contain the white list field of error. The monitoring rules are not limited to preset regular expressions and sensitive information identification rules.
And the analysis module 206 is configured to analyze the target log file set according to a preset monitoring rule, and generate a monitoring result according to an analysis result.
Specifically, the target log file set may be stored in a redis database, a monitoring program is set in the redis database in advance to monitor the redis database, whether the target log files in the target log file set meet a preset condition is monitored, when the preset condition is met, an alarm instruction is triggered, alarm information is generated according to the alarm instruction, and the alarm information is a monitoring result. And sending the alarm information to the target user. The monitoring program presets preset conditions, the preset conditions are limiting values of the occurrence times of the same log file, the preset conditions are met, namely, the occurrence times reach the limiting values, an alarm is triggered, alarm information is generated, for example, the preset conditions are that the occurrence times within 10 minutes are limiting values of 20 times, and if the occurrence times are met, the alarm is triggered, and the alarm information is generated.
Illustratively, the analysis module 206 is further configured to:
acquiring log files with the same log file names in the target log file set; summing up and calculating the occurrence times of the log files with the same log file name; and if the occurrence times of the log files meet preset monitoring rules, generating monitoring results of the terminals corresponding to the log files.
Specifically, the monitoring rules comprise fault alarm monitoring rules, wherein the fault alarm monitoring rules are fault contents set according to monitoring requirements, and the function of the fault alarm monitoring rules is to screen out log files containing fault information meeting the fault alarm monitoring rules, namely preset conditions, from the log files. If the log file does not contain the fault content set in the fault monitoring rule, the log file is a normal log file, otherwise, the log file is a fault file, and the number of times of occurrence of one fault file is increased by one, or the log file can be directly summed. The monitoring result comprises an alarm signal sent by an alarm program and a log file containing sensitive information, wherein the alarm signal content contains the fault reporting times of faults and the fault log file information, and the log file information contains information such as the name of a target terminal, the occurrence position of the log file and the like.
Example III
Referring to fig. 3, a hardware architecture diagram of a computer device according to a third embodiment of the present invention is shown. In this embodiment, the computer device 2 is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction. The computer device 2 may be a rack server, a blade server, a tower server, or a rack server (including a stand-alone server, or a server cluster made up of multiple servers), or the like. As shown in fig. 3, the computer device 2 includes, but is not limited to, at least a memory 21, a processor 22, a network interface 23, and a log file-based monitoring system 20 that are communicatively coupled to each other via a system bus. Wherein:
in this embodiment, the memory 21 includes at least one type of computer-readable storage medium including flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 21 may be an internal storage unit of the computer device 2, such as a hard disk or a memory of the computer device 2. In other embodiments, the memory 21 may also be an external storage device of the computer device 2, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the computer device 2. Of course, the memory 21 may also include both internal storage units of the computer device 2 and external storage devices. In this embodiment, the memory 21 is typically used to store an operating system and various types of application software installed on the computer device 2, such as program codes of the log file-based monitoring system 20 of the second embodiment. Further, the memory 21 may be used to temporarily store various types of data that have been output or are to be output.
The processor 22 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 22 is typically used to control the overall operation of the computer device 2. In this embodiment, the processor 22 is configured to execute the program code or process data stored in the memory 21, for example, execute the log file-based monitoring system 20, so as to implement the log file-based monitoring method of the second embodiment.
The network interface 23 may comprise a wireless network interface or a wired network interface, which network interface 23 is typically used for establishing a communication connection between the server 2 and other electronic devices. For example, the network interface 23 is used to connect the server 2 to an external terminal through a network, establish a data transmission channel and a communication connection between the server 2 and the external terminal, and the like. The network may be an Intranet (Intranet), the Internet (Internet), a global system for mobile communications (Global System of Mobile communication, GSM), wideband code division multiple access (Wideband Code Division Multiple Access, WCDMA), a 4G network, a 5G network, bluetooth (Bluetooth), wi-Fi, or other wireless or wired network. It is noted that fig. 3 only shows a computer device 2 having components 20-23, but it is understood that not all of the illustrated components are required to be implemented, and that more or fewer components may alternatively be implemented. In this embodiment, the log file-based monitoring system 20 stored in the memory 21 may be further divided into one or more program modules, which are stored in the memory 21 and executed by one or more processors (the processor 22 in this embodiment) to complete the present invention.
For example, fig. 2 shows a schematic program module diagram of a second embodiment of the log file based monitoring system 20, where the log file based monitoring system 20 may be divided into the acquisition module 200, the extraction module 202, the matching module 204, and the analysis module 206. Program modules in the present invention are understood to mean a series of computer program instruction segments capable of performing a specific function, more preferably than a program, describing the execution of the log file based monitoring system 20 in the computer device 2. The specific functions of the program modules 200-206 are described in detail in the second embodiment, and are not described herein.
Example IV
The present embodiment also provides a computer-readable storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, which when executed by a processor, performs the corresponding functions. The computer readable storage medium of the present embodiment is used for a computer program, which when executed by a processor implements the log file-based monitoring method of the first embodiment.
It will be apparent to those skilled in the art that the modules or steps of the embodiments of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may alternatively be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than what is shown or described, or they may be separately fabricated into individual integrated circuit modules, or a plurality of modules or steps in them may be fabricated into a single integrated circuit module. Thus, embodiments of the invention are not limited to any specific combination of hardware and software.
The foregoing description is only the preferred embodiments of the present invention, and is not intended to limit the scope of the embodiments of the present invention, but rather the equivalent structures or equivalent flow changes made by the descriptions of the embodiments of the present invention and the contents of the drawings, or the direct or indirect application in other related technical fields, are all included in the scope of the embodiments of the present invention.
Claims (8)
1. A log file-based monitoring method, the method comprising:
collecting log files of each terminal in a target terminal cluster in real time and storing the log files in a message queue;
sequentially pulling a target log file conforming to a target field from the message queue, and extracting a content field in the target log file;
matching the content fields with a preset monitoring rule to screen out matched target log files, so as to obtain a target log file set;
analyzing the target log file set according to a preset monitoring rule, and generating a monitoring result according to an analysis result;
matching the content fields with a preset monitoring rule to screen out matched target log files, wherein the step of obtaining the target log file set comprises the following steps:
fuzzy matching is carried out on the content fields of each target log file based on a preset regular expression so as to extract candidate fields;
identifying the candidate fields through a sensitive information identification rule, screening out target log files corresponding to the matched candidate fields, and obtaining a target log file set, wherein the preset monitoring rule comprises the preset regular expression and the sensitive information identification rule;
the analyzing the target log file set according to the preset monitoring rule, and generating the monitoring result according to the analysis result comprises the following steps:
acquiring log files with the same log file names in the target log file set;
summing up and calculating the occurrence times of the log files with the same log file name;
and if the occurrence times of the log files meet preset monitoring rules, generating monitoring results of the terminals corresponding to the log files.
2. The log file based monitoring method as set forth in claim 1, wherein before collecting log file data of each terminal in the target terminal cluster in real time and storing the log file data in the message queue, the method comprises:
the control monitoring program collects log files generated by application programs on each terminal according to preset data acquisition rules, wherein the data acquisition rules comprise real-time acquisition rules;
the log file is stored in each terminal in the target terminal cluster.
3. The log file based monitoring method as set forth in claim 2, wherein the collecting the log file of each terminal in the target terminal cluster in real time and storing the log file in the message queue comprises:
a log acquisition instruction is sent to the monitoring program so as to control the monitoring program to acquire log files on each terminal according to the log acquisition instruction;
and storing the collected log file into a preset message queue.
4. The log file-based monitoring method as set forth in claim 1, wherein after analyzing the target log file set according to a preset listening rule and generating a monitoring result according to the analysis result, further comprising:
storing the analysis result in a time sequence database;
and sending the analysis result generation report to a target user for checking through the time sequence database.
5. The log file based monitoring method as set forth in claim 4, wherein after analyzing the target log file set according to a preset listening rule and generating a monitoring result according to the analysis result, further comprising:
acquiring a message channel of a target user from a configuration management center;
and sending the monitoring result to the target user for viewing through the message channel.
6. A log file based monitoring system, the system comprising:
the acquisition module is used for acquiring log files of each terminal in the target terminal cluster in real time and storing the log files into the message queue;
the extraction module is used for sequentially extracting the log files conforming to the target fields from the message queue and extracting the content fields in the log files;
the matching module is used for matching the content fields with a preset monitoring rule so as to screen out matched log files and obtain a target log file set;
the analysis module is used for analyzing the target log file set according to a preset monitoring rule and generating a monitoring result according to an analysis result;
the matching module is also used for:
fuzzy matching is carried out on the content fields of each target log file based on a preset regular expression so as to extract candidate fields; identifying the candidate fields through a sensitive information identification rule, screening out target log files corresponding to the matched candidate fields, and obtaining a target log file set, wherein the preset monitoring rule comprises the preset regular expression and the sensitive information identification rule;
the analysis module is also configured to:
acquiring log files with the same log file names in the target log file set; summing up and calculating the occurrence times of the log files with the same log file name; and if the occurrence times of the log files meet preset monitoring rules, generating monitoring results of the terminals corresponding to the log files.
7. A computer device, characterized in that it comprises a memory, a processor, on which a computer program is stored which is executable on the processor, the computer program, when being executed by the processor, implementing the steps of the log file based monitoring method according to any of claims 1-5.
8. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program executable by at least one processor to cause the at least one processor to perform the steps of the log file based monitoring method according to any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110483490.5A CN113176978B (en) | 2021-04-30 | 2021-04-30 | Monitoring method, system, equipment and readable storage medium based on log file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110483490.5A CN113176978B (en) | 2021-04-30 | 2021-04-30 | Monitoring method, system, equipment and readable storage medium based on log file |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113176978A CN113176978A (en) | 2021-07-27 |
| CN113176978B true CN113176978B (en) | 2023-07-21 |
Family
ID=76925720
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110483490.5A Active CN113176978B (en) | 2021-04-30 | 2021-04-30 | Monitoring method, system, equipment and readable storage medium based on log file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113176978B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113672484A (en) * | 2021-08-09 | 2021-11-19 | 深圳市猿人创新科技有限公司 | Equipment monitoring method, device, equipment and medium based on log information |
| CN113760683B (en) * | 2021-08-31 | 2024-02-20 | 银清科技有限公司 | Log acquisition method and device |
| CN114785676A (en) * | 2022-03-02 | 2022-07-22 | 深圳市优必选科技股份有限公司 | Method, device and equipment for extracting log and storage medium |
| CN114866276A (en) * | 2022-03-21 | 2022-08-05 | 杭州薮猫科技有限公司 | Terminal detection method and device for abnormal transmission file, storage medium and equipment |
| CN115514687A (en) * | 2022-06-14 | 2022-12-23 | 鸬鹚科技(深圳)有限公司 | Multi-cloud application gateway flow monitoring method and device, computer equipment and medium |
| CN115080355B (en) * | 2022-07-20 | 2022-11-29 | 北京未来智安科技有限公司 | Method and device for generating monitoring log |
| CN116974856B (en) * | 2023-08-25 | 2024-03-19 | 北京优特捷信息技术有限公司 | Method, device, equipment and medium for automatically acquiring log file |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105389507A (en) * | 2015-11-13 | 2016-03-09 | 小米科技有限责任公司 | Method and apparatus for monitoring files of system partition |
| CN110096408A (en) * | 2019-03-11 | 2019-08-06 | 中国平安人寿保险股份有限公司 | Alarm-monitor method, apparatus, electronic equipment and computer readable storage medium |
| CN110224865A (en) * | 2019-05-30 | 2019-09-10 | 宝付网络科技（上海）有限公司 | A kind of log warning system based on Stream Processing |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140325670A1 (en) * | 2013-04-25 | 2014-10-30 | Rivendale Software Solution Private Limited | System and method for providing risk score based on sensitive information inside user device |
| JP6561212B2 (en) * | 2016-09-01 | 2019-08-14 | 株式会社日立製作所 | Inquiry handling system and method |
| CN109144817A (en) * | 2018-08-03 | 2019-01-04 | 江苏满运软件科技有限公司 | A kind of daily record data monitoring system and method |
| CN110166290A (en) * | 2019-05-16 | 2019-08-23 | 平安科技（深圳）有限公司 | Alarm method and device based on journal file |
| CN110990178A (en) * | 2019-11-27 | 2020-04-10 | 深圳前海微众银行股份有限公司 | Keyword collection monitoring method, device and equipment and computer storage medium |
-
2021
- 2021-04-30 CN CN202110483490.5A patent/CN113176978B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105389507A (en) * | 2015-11-13 | 2016-03-09 | 小米科技有限责任公司 | Method and apparatus for monitoring files of system partition |
| CN110096408A (en) * | 2019-03-11 | 2019-08-06 | 中国平安人寿保险股份有限公司 | Alarm-monitor method, apparatus, electronic equipment and computer readable storage medium |
| CN110224865A (en) * | 2019-05-30 | 2019-09-10 | 宝付网络科技（上海）有限公司 | A kind of log warning system based on Stream Processing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113176978A (en) | 2021-07-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113176978B (en) | Monitoring method, system, equipment and readable storage medium based on log file | |
| CN110661659B (en) | Alarm method, device and system and electronic equipment | |
| CN111143163B (en) | Data monitoring method, device, computer equipment and storage medium | |
| CN112311617A (en) | Configured data monitoring and alarming method and system | |
| CN109460343A (en) | System exception monitoring method, device, equipment and storage medium based on log | |
| CN109495291B (en) | Calling abnormity positioning method and device and server | |
| CN111078513A (en) | Log processing method, device, equipment, storage medium and log alarm system | |
| CN111694718A (en) | Method and device for identifying abnormal behavior of intranet user, computer equipment and readable storage medium | |
| CN107465652B (en) | Operation behavior detection method, server and system | |
| CN111062503B (en) | Power grid monitoring alarm processing method, system, terminal and storage medium | |
| CN111427959B (en) | Data storage method and device | |
| CN110069382B (en) | Software monitoring method, server, terminal device, computer device and medium | |
| CN109558300B (en) | Whole cabinet alarm processing method and device, terminal and storage medium | |
| CN116737765A (en) | Service alarm information processing method and device, electronic equipment and storage medium | |
| CN113835961B (en) | Alarm information monitoring method, device, server and storage medium | |
| CN112131090B (en) | Service system performance monitoring method, device, equipment and medium | |
| CN114356722A (en) | Monitoring alarm method, system, equipment and storage medium for server cluster | |
| CN116416764A (en) | Alarm threshold generation method and device, electronic equipment and storage medium | |
| CN112612679A (en) | System running state monitoring method and device, computer equipment and storage medium | |
| CN112134760A (en) | Link state monitoring method, device, equipment and computer readable storage medium | |
| CN110995500A (en) | Node log management and control method, system and related components | |
| CN110990237A (en) | Information collection system, method and storage medium | |
| CN110677271A (en) | Big data alarm method, device, equipment and storage medium based on ELK | |
| CN112054913B (en) | Data monitoring system and method | |
| CN116127149B (en) | Quantification method and system for health degree of graph database cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |