CN110990218B - Visualization and alarm method and device based on massive logs and computer equipment - Google Patents
Visualization and alarm method and device based on massive logs and computer equipment Download PDFInfo
- Publication number
- CN110990218B CN110990218B CN201911155971.2A CN201911155971A CN110990218B CN 110990218 B CN110990218 B CN 110990218B CN 201911155971 A CN201911155971 A CN 201911155971A CN 110990218 B CN110990218 B CN 110990218B
- Authority
- CN
- China
- Prior art keywords
- information
- log
- preset
- index
- alarm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000012800 visualization Methods 0.000 title claims abstract description 35
- 230000000007 visual effect Effects 0.000 claims abstract description 31
- 238000004590 computer program Methods 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 9
- 238000013079 data visualisation Methods 0.000 claims description 7
- 238000012216 screening Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 5
- 238000001914 filtration Methods 0.000 abstract description 9
- 230000001629 suppression Effects 0.000 abstract description 6
- 230000000694 effects Effects 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000013459 approach Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000005192 partition Methods 0.000 description 3
- 238000003306 harvesting Methods 0.000 description 2
- 238000007794 visualization technique Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3086—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves the use of self describing data formats, i.e. metadata, markup languages, human readable formats
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/323—Visualisation of programs or trace data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Computational Linguistics (AREA)
- Library & Information Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method, a device, computer equipment and a storage medium for visualization and alarm based on mass logs. The method comprises the steps of receiving collected log data information, and caching the log data information to a target cache area to obtain cache information; filtering, assembling and splitting the cache information to obtain information to be stored; the information to be stored is sent and stored to a target storage area to obtain database information; periodically reading the database information according to a reading mode; judging whether the key words which are the same as the key words included in the key word list exist in the log index of the database information; if the keyword which is the same as the keyword included in the keyword list exists in the log index, generating alarm information according to the alarm template and sending the alarm information to the receiving end; and reading the log index of the database information, and correspondingly generating an index display graph according to the log index according to the query statement. The method enriches the log visual graph display effect and the alarm receiving mode, and realizes the alarm suppression function.
Description
Technical Field
The present invention relates to the field of log visualization and alarm, and in particular, to a method, an apparatus, a computer device, and a storage medium for visualization and alarm based on a massive log.
Background
The current method for log management in the industry is generally a log-stack+elastic search+Kibana (ELK) management mode, wherein log is collected by log-stack, log data is stored and indexed by elastic search, and the Kibana visualizes the data. The ELK approach makes log management very simple, but this approach also has technical drawbacks and disadvantages.
The log management of the ELK mode realizes the visualization of the log through the display function of the Kibana, but the original Kibana graph has less plug-ins, general effect, single display content and lack of expansibility; the log alarm is generally customized through scripting, the alarm is characterized in that keywords are filtered by customized scripts, the expansibility is poor, the log alarm is inflexible, and the log alarm has no function of alarm inhibition.
Disclosure of Invention
The embodiment of the invention provides a method, a device and computer equipment for visualization and alarm based on massive logs, and aims to solve the problems that in the prior art, the visualization display content of log management is single, the alarm can not be inhibited, and the visualization of the logs and the expansibility of the alarm are poor.
In a first aspect, an embodiment of the present invention provides a method for visualizing and alerting based on massive logs, including:
receiving the collected log data information;
caching the log data information to a preset target cache area to obtain cache information;
filtering, assembling and splitting the data information to obtain information to be stored;
the information to be stored is sent and stored to a preset target storage area, and database information is obtained;
reading the database information according to a preset reading mode period;
judging whether the key words which are the same as the key words included in a preset key word list exist in the log index of the database information or not;
if the key words which are the same as the key words included in the key word list exist in the log index, generating alarm information according to a preset alarm template, and sending the alarm information to a preset receiving end;
and reading the log index of the database information, and correspondingly generating an index display graph according to a preset query statement.
In a second aspect, an embodiment of the present invention provides a device for visualizing and alerting based on massive logs, including:
the receiving unit is used for receiving the collected log data information;
the caching unit is used for caching the log data information to a preset target caching area to obtain caching information;
the processing unit is used for carrying out data information filtering, data information assembling and data information splitting on the cache information to obtain information to be stored;
the storage unit is used for sending and storing the information to be stored to a preset target storage area to obtain database information;
the reading unit is used for periodically reading the database information according to a preset reading mode;
a judging unit for judging whether the same keywords as the keywords included in the preset keyword list exist in the log index of the database information;
the alarm unit is used for generating alarm information according to a preset alarm template and sending the alarm information to a preset receiving end;
and the graph display unit is used for correspondingly generating an index display graph according to the log index according to the preset query statement.
In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements the method for visualizing and alarming based on a mass log according to the first aspect when the processor executes the computer program.
In a fourth aspect, an embodiment of the present invention further provides a computer readable storage medium, where the computer readable storage medium stores a computer program, where the computer program when executed by a processor causes the processor to perform the method for visualizing and alerting based on mass logs according to the first aspect.
The embodiment of the invention provides a method, a device, computer equipment and a storage medium for visualization and alarm based on massive logs, wherein the method comprises the steps of receiving collected log data information; caching the log data information to a preset target cache area to obtain cache information; filtering, assembling and splitting the data information to obtain information to be stored; the information to be stored is sent and stored to a preset target storage area, and database information is obtained; reading the database information according to a preset reading mode period; judging whether the key words which are the same as the key words included in a preset key word list exist in the log index of the database information or not; if the key words which are the same as the key words included in the key word list exist in the log index, generating alarm information according to a preset alarm template, and sending the alarm information to a preset receiving end; and reading the log index of the database information, and correspondingly generating an index display graph according to a preset query statement.
The method provides the customized display of rich graphics and the alarm suppression function, realizes the clear, attractive and readable visual display effect in log management, and can also approach real-time alarm, avoid alarm flooding and enrich alarm receiving modes.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic view of an application scenario of a method for visualizing and alarming based on massive logs provided in an embodiment of the present invention;
FIG. 2 is a flow chart of a method for visualizing and alerting based on massive logs according to an embodiment of the present invention;
FIG. 3 is a schematic sub-flowchart of a method for visualizing and alerting based on massive logs according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of another sub-flow of the method for visualizing and alerting based on massive logs provided in an embodiment of the present invention;
FIG. 5 is a schematic block diagram of a device for visualizing and alerting based on massive logs provided by an embodiment of the invention;
FIG. 6 is a schematic block diagram of a subunit of a device for visualizing and alerting based on massive logs provided by an embodiment of the present invention;
FIG. 7 is a schematic block diagram of another subunit of a device for visualizing and alerting based on massive logs provided by an embodiment of the present invention;
fig. 8 is a schematic block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be understood that the terms "comprises" and "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
Referring to fig. 1 and fig. 2, fig. 1 is a schematic view of an application scenario of a method for visualizing and alarming based on massive logs according to an embodiment of the present invention; fig. 2 is a flow chart of a method for visualizing and alarming based on a massive log, which is provided by the embodiment of the invention, and is applied to a server, and the method is executed by application software installed in the server.
As shown in fig. 2, the method includes steps S110 to S180.
S110, receiving the collected log data information.
In this embodiment, a specific application scenario is based on management work of massive logs, and needs to manage log data information of a daily server, and implement timely alarm and graphic display of log data information visualization on error information therein. The file is a log data collector of a local file, monitors a log directory or a specific log file, collects log data information at a server side, and starts one or more searchers to check a local path designated by a log administrator for the log file when the file is started. The survey component Proselector and the harvesting component Harvest work cooperatively to monitor event data changes of the read log catalog or specific log files and send the event data changes to an output system preset for the filebean by a log manager. When the method is specifically applied, a log data collector is called to collect log data information at a corresponding target server, and the collected log data information is received according to preset settings.
S120, caching the collected log data information.
In this embodiment, in order to avoid a performance bottleneck of the log collection management tool logstack caused by receiving massive log file data information collected by the log data collector filebat at the server, the log file information collected by the log data collector filebat is sent to the message type Topic of the distributed publishing and subscribing message system Kafka according to a preset target cache area of a log administrator after being received, so as to cache the collected log data information, obtain cache information, and gradually let the log collection management tool logstack read.
And caching the log data information to Kafka is to write the log data information to Topic so that the log data information can be gradually read from Topic by the log collection management tool logstack. The distributed publish-subscribe message system Kafka is a high-throughput distributed publish-subscribe message system and has the advantages of high performance, persistence, multi-copy backup and lateral expansion capability. One Topic actually consists of a plurality of partitions, the order of messages is guaranteed in a single partitions, and when the distributed publish-subscribe message system Kafka caches log data information and encounters a bottleneck, the number of partitions can be increased in the system to perform transverse capacity expansion.
And S130, carrying out data information filtering, data information assembling and data information splitting on the cache information to obtain information to be stored.
When the log data information is cached to a preset target cache area, after the cache information is obtained, the cache information is gradually read by the log collection management tool logstack, so that the cache information is subjected to data information filtering, data information assembling and data information splitting to obtain information to be stored.
The log collection management tool logstack is a tool for collecting logs, and the cache information is collected, parsed and filtered through Agent agents deployed on application program running nodes. The method comprises the steps that a log collection management tool logstack data processing process sets plug-in Inputs, plug-in Filters, plug-in Outputs and plug-in codes which need to be used by defining a Pipeline configuration file so as to realize specific data collection, data processing and data output of log data information; further, obtaining the cache information through an input plug-in, wherein the plug-in Inputs are used for obtaining the cache information; and then sending the cache information to a filter plug-in unit, screening the cache information through screening conditions preset in the filter plug-in unit by plug-in Filters to obtain screened information, assembling the screened information through a preset data assembly format to obtain assembled information, and cutting and splitting the assembled information through a preset regular expression to obtain split information. Wherein a regular expression is a text pattern that describes one or more strings to be matched when searching text, thereby expressing a filtering logic for the strings. Forming information to be stored by each piece of information included in the split information; and the information to be stored is sent to a system target storage area elastic search according to a preset path through the plug-in units output.
And S140, the information to be stored is sent and stored to a preset target storage area, and database information is obtained.
In this embodiment, the log collection management tool logstack filters, assembles and splits the data information from the cache information to form the information to be stored, where the information to be stored is sent to the target storage area elastic search according to a preset setting, so as to store the information to be stored to obtain the database information.
The target storage area elastic search is a search server based on Lucene, provides a distributed multi-user-capability full-text search engine based on a RESTful web interface, can realize distributed, high-expansion and high-real-time search and data analysis, and generates a log index of the database information to the database information stored in the elastic search through HTTP.
In one embodiment, as shown in fig. 3, step S140 further includes:
s141, generating a corresponding index name of the database information according to the writing event time of the database information.
In this embodiment, when the information to be stored is sent to and stored in a preset target storage area, after obtaining database information, the elastic search as a search server may automatically create an index according to the database information, where after the database information is sent to the target storage area, the server may create a corresponding index name for generating the database information according to the writing event time of the database information.
S142, mapping the database information into the corresponding target index according to the index name.
In this embodiment, after generating the corresponding index name of the database information according to the writing event time of the database information, the elastic search provides an index template, and the database information is mapped to the corresponding target index according to the index name. The target storage area elastic search is used as a search server, and a database of the target storage area elastic search can be used as a data source to search and analyze the database information in real time to create a log index for generating the database information.
S150, reading the database information according to a preset reading mode period.
In this embodiment, after the information to be stored is sent and stored to the target storage area elastic search to obtain the database information, the database information is read every 10s by using an Elastaler as an alarm background according to a preset reading mode. The Elastaler also provides an alarm suppression function, and alarm flooding is avoided. The method uses the Elastaler as an alarm background, so that the alarm is more timely, the rule configuration is flexible, the alarm receiving mode is enriched, the alarm suppression is provided, and the working efficiency of a log manager and operation and maintenance developers is improved.
S160, judging whether the same keywords as the keywords included in the preset keyword list exist in the log index of the database information.
In this embodiment, the alarm background Elastaler reads the database information with a period of 10s according to a preset reading mode, the elastiscearch creates a log index for generating the database information by taking the database information as a data source, compares the log index of the database information with a keyword list preset by a log manager, and judges whether the keyword which is the same as the keyword included in the preset keyword list exists in the log index of the database information.
S170, if the key words which are the same as the key words included in the key word list exist in the log index, generating alarm information according to a preset alarm template, and sending the alarm information to a preset receiving end.
Judging whether the keyword which is the same as the keyword which is included in a preset keyword list exists in the log index of the database information, if the keyword which is the same as the keyword which is included in the keyword list exists in the log index, triggering an alarm by an alarm background Elastaler, generating alarm information according to a preset alarm template, sending the alarm information to a preset receiving end, namely generating the alarm information according to an alarm type which corresponds to the alarm template preset by a log manager by the alarm background Elastaler and sending the alarm information to the log manager or an operation and maintenance developer. Various alarm types exist in the Elastaler to realize customized alarms and alarm receiving modes, such as realizing mail alarms, customized WeChat push alarms and the like.
If the log index of the database information does not have the same keywords as the keywords included in the preset keyword list, ending the comparison and judgment of the log index of the database information and the preset keyword list of the log manager.
S180, reading the log index of the database information, and correspondingly generating an index display graph according to a preset query statement.
In order to realize rich visual effects and stronger customization of graphic display of log data information, the method calls open source data visual tool Grafana to realize visual graphic display of the log data information, and after the information to be stored is sent and stored to a preset target storage area to obtain database information, the open source data visual tool Grafana reads a log index of the database information and generates corresponding graphic display according to a customized query statement preset by a log administrator.
The open source data visualization tool Grafana is an open source application written in go language and is used for visual presentation of large-scale index data and simultaneously supports a plurality of different data sources, including the target storage area elastsearch, so that graphic rich and customized presentation can be provided for the database information by using the open source data visualization tool Grafana.
In one embodiment, as shown in fig. 4, step S180 further includes:
s181, calling a visual query statement corresponding to the visual graph generation.
In an embodiment, the database information stored in the elastic search is required to be visually displayed, and the dashboard of Grafana is required to be displayed based on the data source, so that the visualization tool Grafana should be operated to configure the database information of the elastic search as the data source first, and a visualization query statement corresponding to the generation of the visualization graph is called to query the database information.
S182, reading the log index of the database information through the visual query statement to obtain target index information in the log index.
In this embodiment, after the database information of the elastic search is configured for the visualization tool Grafana as a data source, the log index of the database information is read through the visual query statement, and the visualization tool Grafana sets a template variable according to the actual requirement of a log manager to perform custom screening according to the target index, so as to obtain target index information in the log index.
S183, calling a visual graph generation statement, and correspondingly generating an index display graph by the target index information.
In this embodiment, a visual graphic generation statement is called, and an index display graphic is generated corresponding to the target index information. The method comprises the steps of configuring a data source of an open source data visualization tool, presetting a visualization mode of Grafana and adjusting corresponding variable data according to the requirement of a log manager on log graphical display, and establishing a corresponding instrument panel to realize visualization of customized massive log data information. There are various visualization methods of visualization tools Grafana, such as Graph, table, pie chart, and the like.
The embodiment of the invention also provides a device for visualizing and alarming based on the massive logs, which is used for executing any embodiment of the method for visualizing and alarming based on the massive logs. In particular, referring to fig. 5, fig. 5 is a schematic block diagram of an apparatus for visualizing and alarming based on massive logs according to an embodiment of the present invention. The massive log based visualization and alerting device 100 may be configured in a server.
As shown in fig. 5, the apparatus 100 for visualizing and alerting based on a mass log includes a receiving unit 110, a buffering unit 120, a processing unit 130, a storage unit 140, a reading unit 150, a judging unit 160, an alerting unit 170, and a graphic display unit 180.
And a receiving unit 110 for receiving the collected log data information.
In this embodiment, a specific application scenario is based on management work of massive logs, and needs to manage log data information of a daily server, and implement timely alarm and graphic display of log data information visualization on error information therein. And collecting the log data information in the corresponding target server by calling a log data collector, and then receiving the log data information collected by the log data collector Filebeat. The log data collector filecoat collects daily massive log data information at the server side, and monitors event data change of a read log directory or a specific log file.
And the caching unit 120 is configured to cache the log data information to a preset target cache area, so as to obtain cache information.
In this embodiment, after the log data information collected by the log data collector filebat is received, the log data information is sent and stored to a pre-configured target cache area Kafka, and the log data information is written into a message type Topic of the distributed publish-subscribe message system Kafka to obtain the cache information, so that the log collection management tool logstack can gradually read the cache information, thereby avoiding the performance bottleneck of the log collection management tool logstack caused by massive log data information collected by the log data collector filebat at a server side.
And the processing unit 130 is configured to perform data information filtering, data information assembling and data information splitting on the cache information to obtain information to be stored.
In this embodiment, after the log data information is cached in a preset target cache area to obtain the cache information, the log collection management tool logstack gradually reads the cache information from the Topic of the distributed publish-subscribe message system Kafka, so as to perform data information filtering, data information assembly and data information splitting on the cache information to obtain the information to be stored. And the information to be stored is sent to the target storage area elastic search through an output plug-in according to the preset setting of a log manager.
And the storage unit 140 is configured to send and store the information to be stored to a preset target storage area, so as to obtain database information.
In this embodiment, the log collection management tool logstack filters data information, assembles the data information, and splits the data information to form the information to be stored, and sends the information to be stored to the target storage area elastic search according to a preset setting of a log administrator through an output plug-in unit, so as to store the information to be stored to obtain the database information.
In one embodiment, as shown in FIG. 6, the memory unit 140 includes:
the index name creating unit 141 is configured to generate a corresponding index name of the database information according to the writing event time of the database information.
In this embodiment, when the information to be stored is sent to and stored in a preset target storage area, after obtaining database information, the elastic search as a search server may automatically create an index according to the database information, where after the database information is sent to the target storage area, the server may create a corresponding index name for generating the database information according to the writing event time of the database information.
And a mapping unit 142, configured to map the database information to a corresponding target index according to the index name.
In this embodiment, after generating the corresponding index name of the database information according to the writing event time of the database information, the elastic search provides an index template, and the database information is mapped to the corresponding target index according to the index name.
And a reading unit 150 for reading the database information in a preset reading mode period.
In this embodiment, after the information to be stored is sent and stored to the target storage area elastic search to obtain the database information, the database information is read every 10s by using an Elastaler as an alarm background according to a preset reading mode. The Elastaler also provides an alarm suppression function, and alarm flooding is avoided. The method uses the Elastaler as an alarm background, so that the alarm is more timely, the rule configuration is flexible, the alarm receiving mode is enriched, the alarm suppression is provided, and the working efficiency of a log manager and operation and maintenance developers is improved.
And a judging unit 160, configured to judge whether a keyword identical to a keyword included in a preset keyword list exists in a log index of the database information.
In this embodiment, the alarm background Elastaler reads the database information with a period of 10s according to a preset reading mode, the elastiscearch creates a log index for generating the database information by taking the database information as a data source, compares the log index of the database information with a keyword list preset by a log manager, and judges whether the keyword which is the same as the keyword included in the preset keyword list exists in the log index of the database information.
And the alarm unit 170 is configured to generate alarm information according to a preset alarm template, and send the alarm information to a preset receiving end.
In this embodiment, the database information is read by the alarm background Elastaler with the period of 10s, the log index of the database information is compared with a keyword list preset by a log manager, whether the keyword which is the same as the keyword included in the preset keyword list exists in the log index of the database information is judged, if the keyword which is the same as the keyword included in the keyword list exists in the log index, the alarm background Elastaler triggers an alarm, alarm information is generated according to a preset alarm template, and the alarm information is sent to a preset receiving end;
if the log index of the database information does not have the same keywords as the keywords included in the preset keyword list, ending the comparison and judgment of the log index of the database information and the preset keyword list of the log manager.
And the graph display unit 180 is configured to generate an index display graph corresponding to the log index according to a preset query statement.
In the embodiment, in order to enable massive log data information to realize rich visual graphic display, the method calls open source data visualization tool Grafana to realize visual graphic display of the log data information. After the information to be stored is sent and stored to a preset target storage area to obtain database information, the open source data visualization tool Grafana reads the log index of the database information of the elastic search and generates corresponding graphic display according to a customized query statement preset by a log manager.
In one embodiment, as shown in fig. 7, the graphic display unit 180 includes:
181. and the first calling unit is used for calling the visual query statement corresponding to the visual graph generation.
In an embodiment, the database information stored in the elastic search is required to be visually displayed, and the dashboard of Grafana is required to be displayed based on the data source, so that the visualization tool Grafana should be operated to configure the database information of the elastic search as the data source first, and a visualization query statement corresponding to the generation of the visualization graph is called to query the database information.
182. And the query unit is used for reading the log index of the database information through the visual query statement to obtain target index information in the log index.
In this embodiment, after the database information of the elastic search is configured for the visualization tool Grafana as a data source, the log index of the database information is read through the visual query statement, and the visualization tool Grafana sets a template variable according to the actual requirement of a log manager to perform custom screening according to the target index, so as to obtain target index information in the log index.
183. And the second calling unit is used for calling the visual graph generation statement and correspondingly generating the index display graph by the target index information.
In this embodiment, a visual graphic generation statement is called, and an index display graphic is generated corresponding to the target index information. And configuring a data source of the open source data visualization tool Grafana, presetting a visualization mode of Grafana and adjusting corresponding variable data according to the requirement of a log manager on log graphical display, and establishing a corresponding instrument panel to realize customized graphical display of massive log data information. There are various visualization methods of visualization tools Grafana, such as Graph, table, pie chart, and the like.
The above-described mass log based visualization and alerting means may be implemented in the form of a computer program which may be run on a computer device as shown in fig. 8.
Referring to fig. 8, fig. 8 is a schematic block diagram of a computer device according to an embodiment of the present invention. The computer device 500 is a server, and the server may be a stand-alone server or a server cluster formed by a plurality of servers.
With reference to FIG. 8, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032, when executed, may cause the processor 502 to perform a method of visualization and alerting based on a massive log.
The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.
The internal memory 504 provides an environment for the execution of a computer program 5032 in the non-volatile storage medium 503, which computer program 5032, when executed by the processor 502, causes the processor 502 to perform a method of visualizing and alerting based on mass logs.
The network interface 505 is used for network communication, such as providing for transmission of data information, etc. It will be appreciated by those skilled in the art that the architecture shown in fig. 8 is merely a block diagram of some of the architecture relevant to the present inventive arrangements and is not limiting of the computer device 500 to which the present inventive arrangements may be implemented, as a particular computer device 500 may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
The processor 502 is configured to execute a computer program 5032 stored in a memory, so as to implement the method for visualizing and alarming based on massive logs disclosed in the embodiment of the present invention.
Those skilled in the art will appreciate that the embodiment of the computer device shown in fig. 8 is not limiting of the specific construction of the computer device, and in other embodiments, the computer device may include more or less components than those shown, or certain components may be combined, or a different arrangement of components. For example, in some embodiments, the computer device may include only a memory and a processor, and in such embodiments, the structure and function of the memory and the processor are consistent with the embodiment shown in fig. 8, and will not be described again.
It should be appreciated that in embodiments of the present invention, the processor 502 may be a central processing unit (Central Processing Unit, CPU), the processor 502 may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf Programmable gate arrays (FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. Wherein the general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
In another embodiment of the invention, a computer-readable storage medium is provided. The computer readable storage medium may be a non-volatile computer readable storage medium. The computer readable storage medium stores a computer program, wherein the computer program when executed by a processor implements the method for visualizing and alarming based on massive logs disclosed in the embodiment of the invention.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, specific working procedures of the apparatus, device and unit described above may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein. Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, in computer software, or in a combination of the two, and that the elements and steps of the examples have been generally described in terms of function in the foregoing description to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus, device and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, and for example, the division of the units is merely a logical function division, there may be another division manner in actual implementation, or units having the same function may be integrated into one unit, for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. In addition, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices, or elements, or may be an electrical, mechanical, or other form of connection.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present invention.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units may be stored in a storage medium if implemented in the form of software functional units and sold or used as stand-alone products. Based on such understanding, the technical solution of the present invention is essentially or a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, an optical disk, or other various media capable of storing program codes.
While the invention has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.
Claims (5)
1. The method for visualizing and alarming based on the massive logs is characterized by comprising the following steps:
the method comprises the steps of receiving collected log data information, wherein a filebat is called to collect the log data information at a corresponding target server, the collected log data information is received according to preset settings, the filebat is a log data collector of a local file, and when the filebat is started, one or more finders are started to check a local path appointed by a log administrator for the log file, so that the log data information is collected;
caching the log data information to a preset target cache area to obtain cache information;
acquiring the cache information through an input plug-in;
the input plug-in unit sends the cache information to a filter plug-in unit, and the cache information is screened through screening conditions preset in the filter plug-in unit to obtain screened information;
assembling the screened information through a preset data assembly format to obtain assembled information;
cutting and splitting the assembled information through a preset regular expression to obtain split information, and forming information to be stored by each piece of information included in the split information;
the information to be stored is sent and stored to a preset target storage area, and database information is obtained;
generating a corresponding index name of the database information according to the writing event time of the database information;
mapping the database information into a corresponding target index according to the index name;
reading the database information according to a preset reading mode period;
judging whether the key words which are the same as the key words included in a preset key word list exist in the log index of the database information or not;
if the key words which are the same as the key words included in the key word list exist in the log index, generating alarm information according to a preset alarm template, and sending the alarm information to a preset receiving end;
invoking a visual query statement corresponding to the visual graph generation;
reading a log index of the database information through the visual query statement to obtain target index information in the log index;
and calling a visual graph generation statement, and correspondingly generating an index display graph from the target index information, wherein a data source of an open source data visual tool is configured, a visual mode of Grafana is preset according to the requirement of a log manager on log graphical display, corresponding variable data is adjusted, and a corresponding instrument panel is established so as to customize the visualization of massive log data information.
2. The method for visualizing and alerting based on a massive journal as in claim 1, further comprising, prior to said receiving the collected journal data information:
and calling a plurality of log data collectors, and respectively sending the plurality of log data collectors to corresponding target servers to collect log data information.
3. A device for visualizing and alerting based on massive logs, comprising:
the receiving unit is used for receiving the collected log data information, wherein the collected log data information is collected by calling a filebat at a corresponding target server, the collected log data information is received according to preset settings, the filebat is a log data collector of a local file, and when the filebat is started, one or more finders are started to check a local path appointed by a log manager for the log file, so that the log data information is collected;
the caching unit is used for caching the log data information to a preset target caching area to obtain caching information;
the processing unit is used for acquiring the cache information through an input plug-in; the input plug-in unit sends the cache information to a filter plug-in unit, and the cache information is screened through screening conditions preset in the filter plug-in unit to obtain screened information; assembling the screened information through a preset data assembly format to obtain assembled information; cutting and splitting the assembled information through a preset regular expression to obtain split information, and forming information to be stored by each piece of information included in the split information;
the storage unit is used for sending and storing the information to be stored to a preset target storage area to obtain database information;
the index name creation unit is used for generating a corresponding index name of the database information according to the writing event time of the database information;
the mapping unit is used for mapping the database information into the corresponding target index according to the index name;
the reading unit is used for periodically reading the database information according to a preset reading mode;
a judging unit for judging whether the same keywords as the keywords included in the preset keyword list exist in the log index of the database information;
the alarm unit is used for generating alarm information according to a preset alarm template and sending the alarm information to a preset receiving end;
the first calling unit is used for calling a visual query statement corresponding to the visual graph generation;
the query unit is used for reading the log index of the database information through the visual query statement to obtain target index information in the log index;
and the second calling unit is used for calling the visual graph generation statement and correspondingly generating the index display graph from the target index information, wherein a data source of the open source data visualization tool is configured, a visualization mode of Grafana is preset according to the requirement of a log manager on log graphical display, corresponding variable data is adjusted, and a corresponding instrument panel is established so as to customize visualization of massive log data information.
4. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of mass log based visualization and alerting of any one of claims 1 to 2 when the computer program is executed.
5. A computer readable storage medium, characterized in that it stores a computer program which, when executed by a processor, causes the processor to perform the method of massive log based visualization and alerting according to any one of claims 1 to 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911155971.2A CN110990218B (en) | 2019-11-22 | 2019-11-22 | Visualization and alarm method and device based on massive logs and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911155971.2A CN110990218B (en) | 2019-11-22 | 2019-11-22 | Visualization and alarm method and device based on massive logs and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110990218A CN110990218A (en) | 2020-04-10 |
| CN110990218B true CN110990218B (en) | 2023-12-26 |
Family
ID=70085919
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911155971.2A Active CN110990218B (en) | 2019-11-22 | 2019-11-22 | Visualization and alarm method and device based on massive logs and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110990218B (en) |
Families Citing this family (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111581054B (en) * | 2020-04-30 | 2024-04-09 | 重庆富民银行股份有限公司 | Log embedded point service analysis alarm system and method based on ELK |
| CN111881011A (en) * | 2020-07-31 | 2020-11-03 | 网易（杭州）网络有限公司 | Log management method, platform, server and storage medium |
| CN111737091B (en) * | 2020-08-27 | 2020-12-08 | 北京安帝科技有限公司 | Log processing method and device and readable medium |
| CN114625595B (en) * | 2020-12-14 | 2024-07-09 | 网联清算有限公司 | Method, device and system for rechecking dynamic configuration information of service system |
| CN112631885A (en) * | 2020-12-18 | 2021-04-09 | 武汉市烽视威科技有限公司 | Method and system for pre-judging fault in advance and automatically repairing fault |
| CN112667574A (en) * | 2020-12-23 | 2021-04-16 | 国网宁夏电力有限公司信息通信公司 | Method and system for screening mass log data |
| CN112685376A (en) * | 2020-12-23 | 2021-04-20 | 国网宁夏电力有限公司信息通信公司 | Massive log data analysis method and system |
| CN112632070B (en) * | 2020-12-23 | 2022-11-25 | 中国人民解放军63921部队 | Method and device for storing and copying massive diversified complex spacecraft simulation data |
| CN112732663A (en) * | 2020-12-30 | 2021-04-30 | 浙江大华技术股份有限公司 | Log information processing method and device |
| CN112767636A (en) * | 2021-01-14 | 2021-05-07 | 广州穗能通能源科技有限责任公司 | Fire alarm method, fire alarm device, computer equipment and storage medium |
| CN113138896A (en) * | 2021-04-25 | 2021-07-20 | 中国工商银行股份有限公司 | Application running condition monitoring method, device and equipment |
| CN113904913B (en) * | 2021-08-19 | 2024-10-18 | 济南浪潮数据技术有限公司 | Method, device, equipment and storage medium for alarm processing based on pipeline |
| CN114143178B (en) * | 2021-12-03 | 2024-06-04 | 中电信数智科技有限公司 | Alarm root-cause positioning visualization method and device combined with TR069 protocol |
| CN114500255B (en) * | 2022-03-01 | 2024-03-15 | 北京百度网讯科技有限公司 | Log data reporting method, device, equipment and storage medium |
| CN115348161A (en) * | 2022-08-16 | 2022-11-15 | 中国电信股份有限公司 | Log alarm information generation method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107800592A (en) * | 2017-11-09 | 2018-03-13 | 郑州云海信息技术有限公司 | A kind of server test results acquisition method |
| CN108733538A (en) * | 2018-06-26 | 2018-11-02 | 郑州云海信息技术有限公司 | A kind of visualization daily record warning system and method based on ElastAlert |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
| CN110191005A (en) * | 2019-06-25 | 2019-08-30 | 北京九章云极科技有限公司 | A kind of alarm log processing method and system |
| CN110362544A (en) * | 2019-05-27 | 2019-10-22 | 中国平安人寿保险股份有限公司 | Log processing system, log processing method, terminal and storage medium |
| CN110457178A (en) * | 2019-07-29 | 2019-11-15 | 江苏艾佳家居用品有限公司 | A kind of full link monitoring alarm method based on log collection analysis |
-
2019
- 2019-11-22 CN CN201911155971.2A patent/CN110990218B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107800592A (en) * | 2017-11-09 | 2018-03-13 | 郑州云海信息技术有限公司 | A kind of server test results acquisition method |
| CN108733538A (en) * | 2018-06-26 | 2018-11-02 | 郑州云海信息技术有限公司 | A kind of visualization daily record warning system and method based on ElastAlert |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
| CN110362544A (en) * | 2019-05-27 | 2019-10-22 | 中国平安人寿保险股份有限公司 | Log processing system, log processing method, terminal and storage medium |
| CN110191005A (en) * | 2019-06-25 | 2019-08-30 | 北京九章云极科技有限公司 | A kind of alarm log processing method and system |
| CN110457178A (en) * | 2019-07-29 | 2019-11-15 | 江苏艾佳家居用品有限公司 | A kind of full link monitoring alarm method based on log collection analysis |
Non-Patent Citations (2)
| Title |
|---|
| ELK实现日志监控告警;yeweiouyang;《CSDN博客》;20170209;1-8 * |
| elk引入redis kafka;badudd;《CSDN博客》;20191006;1-3 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110990218A (en) | 2020-04-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110990218B (en) | Visualization and alarm method and device based on massive logs and computer equipment | |
| US12079255B1 (en) | Systems and methods for updating a status indication in a system providing dynamic indexer discovery | |
| CN111241078B (en) | Data analysis system, data analysis method and device | |
| US11928118B2 (en) | Generating a correlation search | |
| US11614856B2 (en) | Row-based event subset display based on field metrics | |
| US20210042306A1 (en) | Querying an archive for a data store | |
| US11144528B2 (en) | Event time selection output techniques | |
| US11405301B1 (en) | Service analyzer interface with composite machine scores | |
| US10909151B2 (en) | Distribution of index settings in a machine data processing system | |
| US10523521B2 (en) | Managing ephemeral event streams generated from captured network data | |
| US10572863B2 (en) | Systems and methods for managing allocation of machine data storage | |
| US10366101B2 (en) | Bidirectional linking of ephemeral event streams to creators of the ephemeral event streams | |
| US8533193B2 (en) | Managing log entries | |
| US20150026167A1 (en) | Discovering fields to filter data returned in response to a search | |
| US20190303385A1 (en) | Bidirectional linking of ephemeral event streams to creators of the ephemeral event streams | |
| CN113360554B (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| US12120170B1 (en) | Presenting un-deployed features of an application | |
| US20090204949A1 (en) | System, method and program product for dynamically adjusting trace buffer capacity based on execution history | |
| CN111651751B (en) | Security event analysis report generation method and device, storage medium and equipment | |
| US10778710B2 (en) | User configurable alert notifications applicable to search query results | |
| US8073946B1 (en) | Monitoring of metrics to identify abnormalities in a large scale distributed computing environment | |
| US20180225343A1 (en) | Configuring Alerts Related to Performance Problems or Security Issues in an Information Technology Environment | |
| Barberis et al. | ATLAS Eventlndex monitoring system using the Kibana analytics and visualization platform | |
| Vainio | Implementation of Centralized Logging and Log Analysis in Cloud Transition | |
| US11907227B1 (en) | System and method for changepoint detection in streaming data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |