CN110347716B - Log data processing method, device, terminal equipment and storage medium - Google Patents
Log data processing method, device, terminal equipment and storage medium Download PDFInfo
- Publication number
- CN110347716B CN110347716B CN201910447654.1A CN201910447654A CN110347716B CN 110347716 B CN110347716 B CN 110347716B CN 201910447654 A CN201910447654 A CN 201910447654A CN 110347716 B CN110347716 B CN 110347716B
- Authority
- CN
- China
- Prior art keywords
- log data
- cluster
- log
- real
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 34
- 238000003860 storage Methods 0.000 title claims abstract description 18
- 238000012545 processing Methods 0.000 claims abstract description 162
- 238000004458 analytical method Methods 0.000 claims description 28
- 238000004590 computer program Methods 0.000 claims description 13
- 238000000034 method Methods 0.000 claims description 11
- 238000007619 statistical method Methods 0.000 claims description 10
- 230000002452 interceptive effect Effects 0.000 claims description 9
- 230000006399 behavior Effects 0.000 claims description 5
- 238000007621 cluster analysis Methods 0.000 claims description 3
- 230000009467 reduction Effects 0.000 claims description 3
- 238000011084 recovery Methods 0.000 claims 1
- 238000012550 audit Methods 0.000 abstract description 5
- 230000006870 function Effects 0.000 description 9
- 241001178520 Stomatepia mongo Species 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 239000000523 sample Substances 0.000 description 3
- 238000003306 harvesting Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000008719 thickening Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1464—Management of the backup or restore process for networked environments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/338—Presentation of query results
Abstract
The embodiment of the invention provides a log data processing method, which comprises the following steps: acquiring log data; receiving a processing instruction for the log data; judging the processing instruction to be a log data real-time processing instruction or a log data offline processing instruction; when the processing instruction is the log data real-time processing instruction, the log data is processed in real time through an elastic search cluster; and when the processing instruction is the log data offline processing instruction, performing offline processing on the log data through the HBase cluster. The embodiment of the invention also provides a log data processing device, terminal equipment and a computer readable storage medium. By utilizing the embodiment of the invention, the mass log data can be efficiently analyzed and stored, and the efficiency of utilizing the log data to carry out security audit is improved.
Description
Technical Field
The present invention relates to the field of log processing technologies, and in particular, to a log data processing method, a log data processing device, a terminal device, and a computer readable storage medium.
Background
At present, the number and types of threats to key information resources in a network environment are rapidly increased, and how to actively respond to network attack behaviors in time is a research hotspot in the network security field in recent years. The assessment of network security situations by analyzing log data has gained increasing acceptance. With the development of computers and networks, the data processing amount of log data is increasing, and the data magnitude of log data is usually more than one million, even more than one million and one trillion. For such a huge log data system, higher requirements are first put forward on the processing of log data. However, the current log data processing system is generally composed of a log collection agent and an analysis management system, and can perform security analysis on logs with smaller data volume, but facing massive log files in a large and complex network, the log files cannot better perform collection and analysis tasks in a tool form working mode, the data are isolated and dispersed, cannot be associated, cannot extract commonalities, and cannot make the network into a whole to cope with security events due to the lack of comprehensive analysis on the whole log data.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a log data processing method, a log data processing apparatus, a terminal device, and a computer-readable storage medium, which are capable of efficiently analyzing and storing massive log data, and improving the efficiency of security audit using log data.
An embodiment of the present invention provides a log data processing method, where the log data processing method includes:
acquiring log data;
receiving a processing instruction for the log data;
judging the processing instruction to be a log data real-time processing instruction or a log data offline processing instruction;
when the processing instruction is the log data real-time processing instruction, the log data is processed in real time through an elastic search cluster;
and when the processing instruction is the log data offline processing instruction, performing offline processing on the log data through the HBase cluster.
Further, in the above log data processing method provided by the embodiment of the present invention, the real-time processing of the log data by the elastic search cluster includes:
searching the log data in real time according to a keyword searching mode, and displaying a searching result in a preset mode;
real-time alarming is carried out on the log data according to preset alarming rules, wherein the preset alarming rules comprise one or more of the following combinations: event alarms, field statistics alarms, continuous statistics alarms, baseline comparison alarms and threshold statistics alarms;
and carrying out rule matching on the log data according to a preset statistical rule, and carrying out real-time statistics on the log data meeting the preset statistical rule.
Further, in the above log data processing method provided by the embodiment of the present invention, the threshold statistics alarm includes:
carrying out statistics on the log data according to a statistics rule to obtain a statistics analysis result;
checking a preset output index in the statistical analysis result according to a preset index threshold value, and judging whether the preset output index in the statistical analysis result exceeds the preset index threshold value;
and if the preset output index in the statistical analysis result exceeds the preset index threshold, outputting a preset alarm prompt to a preset application responsible person.
Further, in the above log data processing method provided by the embodiment of the present invention, the offline processing of the log data by the HBase cluster includes one or more of the following combinations:
performing offline analysis on the log data through the HBase cluster, wherein the offline analysis comprises offline log data cluster analysis and user behavior analysis;
carrying out log backup on the log data through the HBase cluster;
and carrying out log restoration on the log data through the HBase cluster.
Further, in the above log data processing method provided by the embodiment of the present invention, the performing, by the HBase cluster, log backup on the log data includes:
reading index information in the elastic search cluster through a TCP protocol;
acquiring log data in the elastic search cluster according to the index information;
and writing the log data in the elastic search cluster into the HBase cluster for log backup.
Further, in the above log data processing method provided by the embodiment of the present invention, the performing log restoration on the log data by using the HBase cluster includes:
reading log data in the HBase cluster;
and writing the read log data in the HBase cluster back to the elastic search cluster in a Bluetooth API mode in the elastic search cluster to perform log reduction.
Further, in the above log data processing method provided by the embodiment of the present invention, after the log data is obtained, the method further includes:
shunting the log data through a Kafka cluster to obtain real-time log data and non-real-time log data;
inputting the real-time log data into the elastiscearch cluster;
and inputting the non-real-time log data into the HBase cluster.
The second aspect of the embodiment of the present invention further provides a log data processing device, where the log data processing device includes:
the log acquisition module is used for acquiring log data;
the instruction receiving module is used for receiving a processing instruction aiming at the log data;
the instruction judging module is used for judging whether the processing instruction is a log data real-time processing instruction or a log data offline processing instruction;
the real-time processing module is used for processing the log data in real time through an elastic search cluster when the processing instruction is the log data real-time processing instruction;
and the offline processing module is used for performing offline processing on the log data through the HBase cluster when the processing instruction is the log data offline processing instruction.
A third aspect of the embodiment of the present invention further provides a terminal device, where the terminal device includes a processor, and the processor is configured to implement any one of the log data processing methods described above when executing a computer program stored in a memory.
A fourth aspect of the embodiment of the present invention further provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the log data processing method of any one of the above.
The embodiment of the invention provides a log data processing method, a log data processing device, terminal equipment and a computer readable storage medium, which are used for acquiring log data; receiving a processing instruction for the log data; judging the processing instruction to be a log data real-time processing instruction or a log data offline processing instruction; when the processing instruction is the log data real-time processing instruction, the log data is processed in real time through an elastic search cluster; and when the processing instruction is the log data offline processing instruction, performing offline processing on the log data through the HBase cluster. By utilizing the embodiment of the invention, the mass log data can be efficiently analyzed and stored, and the efficiency of utilizing the log data to carry out security audit is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a log data processing method according to a first embodiment of the present invention.
Fig. 2 is a schematic structural diagram of a terminal device according to an embodiment of the present invention.
Fig. 3 is an exemplary functional block diagram of the terminal device shown in fig. 2.
Description of the main reference signs
| Terminal equipment | 1 |
| Memory device | 10 |
| Display screen | 20 |
| Processor and method for controlling the same | 30 |
| Log data processing device | 100 |
| Log acquisition module | 101 |
| Instruction receiving module | 102 |
| Instruction judging module | 103 |
| Real-time processing module | 104 |
| Offline processing module | 105 |
The following detailed description will further illustrate embodiments of the invention in conjunction with the above-described drawings.
Detailed Description
In order that the above-recited objects, features and advantages of embodiments of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description. In addition, features in the embodiments of the present application may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the invention, and the described embodiments are merely some, rather than all, of the embodiments of the invention. All other embodiments, based on the embodiments of the invention, which are obtained by a person of ordinary skill in the art without making any inventive effort, are within the scope of the embodiments of the invention.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which embodiments of the invention belong. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
Fig. 1 is a flowchart of a log data processing method according to a first embodiment of the present invention. The log data processing method may be applied to the terminal device 1, and the terminal device 1 may be a smart device such as a smart phone, a notebook computer, a desktop/tablet computer, a smart watch, and a personal digital assistant (Personal Digital Assistant, PDA). As shown in fig. 1, the log data processing method may include the steps of:
s101: log data is obtained.
In this embodiment, the log data obtained by the log obtaining module from a preset source database may include user behavior data, application state data or device state data, and the preset source database may be preset by a system operator, which does not limit the content and source of the log data. The log acquisition module may use filebat to perform log data acquisition (hereinafter referred to as a filebat log acquisition module), where filebat is a log data collector. The Filebeat log acquisition module supports customization of various log data senders, and is used for acquiring log data and outputting the log data to various log data receivers. Specifically, the filebed log collection module starts one or more probes (probes) to detect a specified log directory or file; for each log file found by the probe, the filebed log acquisition module initiates a harvesting process (harvester); each of the harvests reads the new content of one log file and sends the new content of the log file to a handler (spooner), which gathers the log data, and finally the filebed log acquisition module sends the gathered log data to a designated location. It may be appreciated that after the log data is obtained, the log data may be further converted according to a preset structure, and specifically, the preset structure of the log data may include a log time, a log level, a log output class, a log content, and the like.
In this embodiment, after the log data is obtained, the method further includes: the log data is shunted through a Kafka cluster (the Kafka cluster is a distributed message cache middleware and has the characteristic of high throughput (even if very common hardware is used, the Kafka can support hundreds of thousands of messages per second) and is used for caching massive data, and the data is distributed and controlled in a message queue mode), so that real-time log data and non-real-time log data are obtained; inputting the real-time log data into the elastiscearch cluster; and inputting the non-real-time log data into the HBase cluster. The shunting of the log data through the Kafka cluster includes adopting a Strom stream type computing framework to analyze and process the log data cached in the Kafka message queue, so as to obtain real-time log data and non-real-time log data. In other embodiments, the log data may be classified by a Zookeeper (Zookeeper is a distributed, open source, distributed application coordination service) cluster to obtain real-time log data and non-real-time log data. It is to be appreciated that prior to entering the real-time log data into the elastiscearch cluster, the method further comprises: receiving real-time log data in different topics cached in a Kafka message queue; and analyzing the real-time log data according to a preset analysis rule by using a Logstar log analysis module. Analyzing the real-time log data through the logstar log analysis module according to a preset analysis rule comprises cleaning and processing the real-time log data through the logstar log analysis module, and structuring the real-time log data into different fields. And analyzing the log file through a Logstar log analysis module, identifying useful information in the first shunt log data to be processed, and filtering out junk data. The Logstar log analysis module is configured with analysis files of all log sources, and the preset analysis rules are rules set in the analysis files.
Before entering the non-real-time log data into the HBase cluster, the method further comprises: reading a preset analysis rule; and analyzing the non-real-time log data through the Spark cluster according to a preset analysis rule, analyzing the non-real-time log data into an HBase data table format, and storing the analyzed HBase data table format into the HBase cluster. The predetermined parsing rule may be preset by a system developer, and the predetermined parsing rule may include one or more of regular expression, keyValue parsing, field value splitting (for example, splitting by using a split function), string type conversion into numerical values, json parsing, URL decoding, timestamp identification, and UserAgent parsing.
S102: and receiving a processing instruction aiming at the log data.
In this embodiment, a processing instruction for the log data is received, where the processing instruction for the log data includes a log data real-time processing instruction and a log data offline processing instruction, the log data real-time processing instruction includes a real-time search instruction, a real-time alarm instruction, and an online statistics instruction, and the log data offline processing instruction includes an offline analysis instruction, a log backup instruction, and a log restoration instruction. The embodiment of the invention provides an interactive interface, and a corresponding touch area is arranged on the interactive interface aiming at each log data processing instruction. And obtaining a processing instruction aiming at the log data by receiving a preset operation (such as mouse click or finger touch) output in the corresponding touch area.
S103: and judging the processing instruction to be a log data real-time processing instruction or a log data offline processing instruction.
In this embodiment, after receiving the processing instruction for the log data, it is determined that the processing instruction is a log data real-time processing instruction or a log data offline processing instruction, and when the processing instruction is the log data real-time processing instruction, step S104 is executed; when the processing instruction is the log data offline processing instruction, step S105 is executed.
S104: and processing the log data in real time through an elastic search cluster.
In this embodiment, when the processing instruction is the log data real-time processing instruction, the log data is processed in real time by the elastic search cluster. The real-time processing of the log data by the elastic search cluster comprises: searching the log data in real time according to a keyword searching mode, and displaying a searching result in a preset mode; the preset mode comprises thickening and highlighting the detection result. For log data containing keywords, it is also supported to view the context of printing of the log data containing the log keywords.
Or, carrying out real-time alarm on the log data according to preset alarm rules, wherein the preset alarm rules comprise one or more of the following combinations: event alarms, field statistics alarms, continuous statistics alarms, baseline comparison alarms and threshold statistics alarms; and for the event alarm rule, creating alarm triggering conditions based on the search result of log data, for example, setting a preset threshold number for triggering alarms within a preset time range, and if the number of actually triggered alarms is greater than the preset threshold number, performing alarm prompt. And providing alarm setting for field contents for the field statistical alarm rule, wherein the field contents can be filled in the trigger condition, and the statistical mode can be selected in a drop-down box of the interactive interface, wherein the drop-down box comprises a criterion (independent count), sum (summation), avg (average value), max (maximum value) and min (minimum value). And providing continuous triggering alarm setting for the continuous statistical alarm rule, setting alarm conditions, and triggering an alarm when the continuous triggering times of the alarm conditions in a preset time period reach a preset threshold value. For the baseline contrast alert rule, the threshold may be set to a statistical baseline value (which may vary over time) and the time range for baseline generation is selected. Meanwhile, the baseline contrast alarm provides a more flexible trigger range setting mode, for example, the trigger range setting mode can be selected to be larger than, smaller than, in the interval and out of the interval in a drop-down frame. For a threshold statistical alert rule, the threshold statistical alert includes: carrying out statistics on the log data according to a statistics rule to obtain a statistics analysis result; checking a preset output index in the statistical analysis result according to a preset index threshold value, and judging whether the preset output index in the statistical analysis result exceeds the preset index threshold value; and if the preset output index in the statistical analysis result exceeds the preset index threshold, outputting a preset alarm prompt to a preset application responsible person. The statistical rule includes a preset statistical item including field information (for example, clientip, requestURL and other field information) specified in advance to be counted, and a preset output index. The preset output index includes an output value of the preset statistical item (for example, the preset output index is a count (count)), and the count may include a statistical number of the pre-specified field information to be counted. The preset index threshold is a value preset by a user of the terminal equipment. The preset application is responsible for an application responsible person preset by a terminal device user.
Or performing rule matching on the log data according to a preset statistical rule, and performing real-time statistics on the log data meeting the preset statistical rule. The rule matching of the log data according to the preset statistical rule, and the real-time statistics of the log data meeting the preset statistical rule comprise the following steps: and carrying out rule matching on the log data according to the received preset statistical rule, carrying out statistics on the information to be counted which meets the preset statistical rule, and outputting a statistical result. The statistical result can be displayed in the forms of broken lines, tables, bars, cakes and the like. The preset statistical rules can support operations such as adding, modifying, deleting, searching and storing on the interactive interface.
S105: and performing offline processing on the log data through the HBase cluster.
In this embodiment, when the processing instruction is the log data offline processing instruction, the log data is offline processed by the HBase cluster. The offline processing of the log data by the HBase cluster includes one or more of the following combinations: performing offline analysis on the log data through the HBase cluster, wherein the offline analysis comprises offline log data cluster analysis and user behavior analysis; carrying out log backup on the log data through the HBase cluster; and carrying out log restoration on the log data through the HBase cluster.
The performing log backup on the log data through the HBase cluster includes: reading index information in the elastic search cluster through a TCP protocol; acquiring log data in the elastic search cluster according to the index information; and writing the log data in the elastic search cluster into the HBase cluster for log backup. The performing log restoration on the log data through the HBase cluster includes: reading log data in the HBase cluster; and writing the read log data in the HBase cluster back to the elastic search cluster for log reduction by means of a Bluetooth API (Bluetooth interface for containing a plurality of index operations in one interface call) in the elastic search cluster.
In this embodiment, after the log data is processed in real time by the elastic search cluster, a real-time processing result is output; and after the log data is subjected to offline processing through the HBase cluster, outputting an offline processing result. The real-time processing result and the offline processing result can be displayed through a result display module in the Web client. The embodiment of the invention also provides a Mysql database, a Mongo database and a Web application program. The Web application program is connected with the Mysql database and the Mongo database. The Mysql database is a relational database management system of open source codes, and mainly stores resource configuration related data. The Mongo database is a database based on distributed file storage, and aims to provide an extensible high-performance data storage solution for WEB applications, and the Mongo database mainly stores statistical analysis results of log data.
The Web application program is also connected with the Web server, the Web server is used for receiving the interactive data which is uploaded by the Web client and used for carrying out data interaction with the Web application program, outputting the interactive data to the Web application program through an interface, processing the interactive data by the Web application program to obtain a processing result, feeding back the processing result to the Web server, feeding back the processing result to the Web client through the Web server, and displaying the result through a result display module in the Web client.
The embodiment of the invention provides a log data processing method, which is used for acquiring log data; receiving a processing instruction for the log data; judging the processing instruction to be a log data real-time processing instruction or a log data offline processing instruction; when the processing instruction is the log data real-time processing instruction, the log data is processed in real time through an elastic search cluster; and when the processing instruction is the log data offline processing instruction, performing offline processing on the log data through the HBase cluster. By utilizing the embodiment of the invention, the mass log data can be efficiently analyzed and stored, and the efficiency of utilizing the log data to carry out security audit is improved.
Fig. 2 is a schematic structural diagram of a terminal device 1 according to an embodiment of the present invention, and as shown in fig. 2, the terminal device 1 includes a memory 10, and a log data processing apparatus 100 is stored in the memory 10. The terminal device 1 may be a mobile phone, a tablet computer, a personal digital assistant, or the like, which has an application display function. The log data processing apparatus 100 may acquire log data; receiving a processing instruction for the log data; judging the processing instruction to be a log data real-time processing instruction or a log data offline processing instruction; when the processing instruction is the log data real-time processing instruction, the log data is processed in real time through an elastic search cluster; and when the processing instruction is the log data offline processing instruction, performing offline processing on the log data through the HBase cluster. By utilizing the embodiment of the invention, the mass log data can be efficiently analyzed and stored, and the efficiency of utilizing the log data to carry out security audit is improved.
In this embodiment, the terminal device 1 may further include a display 20 and a processor 30. The memory 10 and the display 20 may be electrically connected to the processor 30, respectively.
The memory 10 may be a different type of storage device for storing various types of data. For example, the memory and the internal memory of the terminal device 1 may be used, and a memory Card such as a flash memory, an SM Card (Smart Media Card), an SD Card (Secure Digital Card ) or the like may be used as the memory Card which can be externally connected to the terminal device 1. In addition, memory 10 may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card), at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 10 is used for storing various types of data, such as various Applications (Applications) installed in the terminal device 1, data set and acquired by applying the above log data processing method, and the like.
A display screen 20 is mounted to the terminal device 1 for displaying information.
The processor 30 is configured to execute the log data processing method and various software installed in the terminal device 1, such as an operating system and application display software. The processor 30 includes, but is not limited to, a processor (Central Processing Unit, CPU), a micro control unit (Micro Controller Unit, MCU), etc., for interpreting the computer and processing the data in the computer software.
The log data processing apparatus 100 may include one or more modules stored in the memory 10 of the terminal device 1 and configured to be executed by one or more processors (one processor 30 in this embodiment) to complete the embodiment of the present invention. For example, referring to fig. 3, the log data processing apparatus 100 may include a log obtaining module 101, an instruction receiving module 102, an instruction judging module 103, a real-time processing module 104, and an offline processing module 105. Modules may be referred to in the embodiments of the present invention as program segments, which perform a particular function, more favorably than programs for describing the execution of software in a processor.
It will be appreciated that, corresponding to each of the above embodiments of the log data processing method, the terminal device 1 may include some or all of the functional modules shown in fig. 3, and the functions of each module will be described in detail below. It should be noted that the same noun related nouns and their specific explanations in the above embodiments of the log data processing method may also be applied to the following functional descriptions of the modules. For the sake of space saving and repetition avoidance, the description is omitted.
The log acquisition module 101 may be used to acquire log data.
The instruction receiving module 102 may be configured to receive processing instructions for the log data.
The instruction determining module 103 may be configured to determine that the processing instruction is a log data real-time processing instruction or a log data offline processing instruction.
The real-time processing module 104 may be configured to process the log data in real time through an elastic search cluster when the processing instruction is the log data real-time processing instruction.
The offline processing module 105 may be configured to perform offline processing on the log data through an HBase cluster when the processing instruction is the log data offline processing instruction.
The embodiment of the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the log data processing method in any of the above embodiments.
The modules/units of the log data processing apparatus 100/terminal device 1/computer device integration may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as a separate product. Based on such understanding, the present invention may implement all or part of the flow of the method of the foregoing embodiment, or may be implemented by instructing related hardware by a computer program, where the computer program may be stored on a computer readable storage medium, and the computer program may implement the steps of each of the method embodiments described above when executed by a processor. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable storage medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth.
The processor 30 may be a central processing unit (Central Processing Unit, CPU), other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, and the processor 30 is a control center of the log data processing apparatus 100/terminal device 1, and connects the respective parts of the whole log data processing apparatus 100/terminal device 1 using various interfaces and lines.
The memory 10 is used for storing the computer program and/or the module, and the processor 30 implements various functions of the log data processing apparatus 100/terminal device 1 by running or executing the computer program and/or the module stored in the memory and calling the data stored in the memory 10. The memory 10 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data (such as audio data, phonebook, etc.) created according to the use of the handset, etc.
In the several embodiments provided in the present invention, it should be understood that the disclosed terminal device and method may be implemented in other manners. For example, the system embodiments described above are merely illustrative, e.g., the division of the modules is merely a logical function division, and other manners of division may be implemented in practice.
It will be evident to those skilled in the art that the embodiments of the invention are not limited to the details of the foregoing illustrative embodiments, and that the embodiments of the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of embodiments being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. A plurality of units, modules or means recited in the claims can also be implemented by means of software or hardware by means of one and the same unit, module or means.
The foregoing embodiments are merely for illustrating the technical solution of the embodiment of the present invention, but not for limiting the same, although the embodiment of the present invention has been described in detail with reference to the foregoing preferred embodiments, it will be understood by those skilled in the art that modifications and equivalent substitutions may be made to the technical solution of the embodiment of the present invention without departing from the spirit and scope of the technical solution of the embodiment of the present invention.
Claims (9)
1. The log data processing method is applied to the terminal equipment and is characterized by comprising the following steps of:
acquiring log data;
receiving a processing instruction for the log data;
judging the processing instruction to be a log data real-time processing instruction or a log data offline processing instruction;
when the processing instruction is the log data real-time processing instruction, the log data is processed in real time through an elastic search cluster;
when the processing instruction is the log data offline processing instruction, performing offline processing on the log data through an HBase cluster, including: user behavior analysis is carried out on the log data through the HBase cluster, the processing instructions are obtained by receiving preset operation triggers carried out on an interactive interface of the terminal equipment, and each processing instruction is provided with a corresponding touch area on the interactive interface; or alternatively
And shunting the log data through a Kafka cluster to obtain real-time log data and non-real-time log data, inputting the real-time log data into the elastic search cluster, analyzing the non-real-time log data into an HBase data table format through a Spark cluster according to a preset analysis rule, and inputting the analyzed HBase data table format into the HBase cluster.
2. The log data processing method as set forth in claim 1, wherein the real-time processing of the log data by the elastic search cluster comprises:
searching the log data in real time according to a keyword searching mode, and displaying a searching result in a preset mode;
real-time alarming is carried out on the log data according to preset alarming rules, wherein the preset alarming rules comprise one or more of the following combinations: event alarms, field statistics alarms, continuous statistics alarms, baseline comparison alarms and threshold statistics alarms;
and carrying out rule matching on the log data according to a preset statistical rule, and carrying out real-time statistics on the log data meeting the preset statistical rule.
3. The log data processing method of claim 2 wherein the threshold statistical alert comprises:
carrying out statistics on the log data according to a statistics rule to obtain a statistics analysis result;
checking a preset output index in the statistical analysis result according to a preset index threshold value, and judging whether the preset output index in the statistical analysis result exceeds the preset index threshold value;
and if the preset output index in the statistical analysis result exceeds the preset index threshold, outputting a preset alarm prompt to a preset application responsible person.
4. The log data processing method of claim 1, wherein the offline processing of the log data by the HBase cluster further comprises one or more of the following combinations:
performing offline log data cluster analysis on the log data through the HBase cluster;
carrying out log backup on the log data through the HBase cluster;
and carrying out log restoration on the log data through the HBase cluster.
5. The method according to claim 4, wherein the performing log backup on the log data by the HBase cluster includes:
reading index information in the elastic search cluster through a TCP protocol;
acquiring log data in the elastic search cluster according to the index information;
and writing the log data in the elastic search cluster into the HBase cluster for log backup.
6. The log data processing method of claim 5, wherein the log recovery of the log data by the HBase cluster comprises:
reading log data in the HBase cluster;
and writing the read log data in the HBase cluster back to the elastic search cluster in a Bluetooth API mode in the elastic search cluster to perform log reduction.
7. A log data processing apparatus for implementing the log data processing method according to any one of claims 1 to 6, characterized in that the log data processing apparatus comprises:
the log acquisition module is used for acquiring log data;
the instruction receiving module is used for receiving a processing instruction aiming at the log data;
the instruction judging module is used for judging whether the processing instruction is a log data real-time processing instruction or a log data offline processing instruction;
the real-time processing module is used for processing the log data in real time through an elastic search cluster when the processing instruction is the log data real-time processing instruction;
and the offline processing module is used for performing offline processing on the log data through the HBase cluster when the processing instruction is the log data offline processing instruction.
8. A terminal device, characterized in that the terminal device comprises a processor for implementing the log data processing method according to any of claims 1-6 when executing a computer program stored in a memory.
9. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the log data processing method according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910447654.1A CN110347716B (en) | 2019-05-27 | 2019-05-27 | Log data processing method, device, terminal equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910447654.1A CN110347716B (en) | 2019-05-27 | 2019-05-27 | Log data processing method, device, terminal equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110347716A CN110347716A (en) | 2019-10-18 |
| CN110347716B true CN110347716B (en) | 2024-04-02 |
Family
ID=68173983
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910447654.1A Active CN110347716B (en) | 2019-05-27 | 2019-05-27 | Log data processing method, device, terminal equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110347716B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111125042A (en) * | 2019-11-13 | 2020-05-08 | 中国建设银行股份有限公司 | Method and device for determining risk operation event |
| CN111404909B (en) * | 2020-03-10 | 2022-05-31 | 上海豌豆信息技术有限公司 | Safety detection system and method based on log analysis |
| CN112131283A (en) * | 2020-09-30 | 2020-12-25 | 重庆市海普软件产业有限公司 | Intelligent acquisition system capable of being flexibly expanded |
| CN113283884A (en) * | 2020-12-31 | 2021-08-20 | 深圳怡化电脑股份有限公司 | Log processing method and device |
| CN113221033A (en) * | 2021-04-24 | 2021-08-06 | 上海钢银科技发展有限公司 | Buried point acquisition and statistical analysis method, system, equipment and storage medium |
| CN113238912B (en) * | 2021-05-08 | 2022-12-06 | 国家计算机网络与信息安全管理中心 | Aggregation processing method for network security log data |
| CN113411206B (en) * | 2021-05-26 | 2022-09-06 | 北京沃东天骏信息技术有限公司 | Log auditing method, device, equipment and computer storage medium |
| CN113312353A (en) * | 2021-06-10 | 2021-08-27 | 中国民航信息网络股份有限公司 | Storage method and system for tracking journal |
| CN113783849B (en) * | 2021-08-25 | 2023-07-11 | 福建天泉教育科技有限公司 | Sensitive information detection method and terminal |
| CN116991661A (en) * | 2023-07-20 | 2023-11-03 | 北京直客通科技有限公司 | Problem alarm system and method for software system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106790718A (en) * | 2017-03-16 | 2017-05-31 | 北京搜狐新媒体信息技术有限公司 | Service call link analysis method and system |
| CN107294801A (en) * | 2016-12-30 | 2017-10-24 | 江苏号百信息服务有限公司 | Stream Processing method and system based on magnanimity real-time Internet DPI data |
| CN107577588A (en) * | 2017-09-26 | 2018-01-12 | 北京中安智达科技有限公司 | A kind of massive logs data intelligence operational system |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170169078A1 (en) * | 2015-12-14 | 2017-06-15 | Siemens Aktiengesellschaft | Log Mining with Big Data |
| US10474513B2 (en) * | 2016-10-11 | 2019-11-12 | Oracle International Corporation | Cluster-based processing of unstructured log messages |
-
2019
- 2019-05-27 CN CN201910447654.1A patent/CN110347716B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107294801A (en) * | 2016-12-30 | 2017-10-24 | 江苏号百信息服务有限公司 | Stream Processing method and system based on magnanimity real-time Internet DPI data |
| CN106790718A (en) * | 2017-03-16 | 2017-05-31 | 北京搜狐新媒体信息技术有限公司 | Service call link analysis method and system |
| CN107577588A (en) * | 2017-09-26 | 2018-01-12 | 北京中安智达科技有限公司 | A kind of massive logs data intelligence operational system |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110347716A (en) | 2019-10-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110347716B (en) | Log data processing method, device, terminal equipment and storage medium | |
| CN110362544B (en) | Log processing system, log processing method, terminal and storage medium | |
| US11586972B2 (en) | Tool-specific alerting rules based on abnormal and normal patterns obtained from history logs | |
| CN109034993B (en) | Account checking method, account checking equipment, account checking system and computer readable storage medium | |
| US20170192872A1 (en) | Interactive detection of system anomalies | |
| JP7373611B2 (en) | Log auditing methods, equipment, electronic equipment, media and computer programs | |
| CN109669795B (en) | Crash information processing method and device | |
| CN105743730A (en) | Method and system used for providing real-time monitoring for webpage service of mobile terminal | |
| CN110928934A (en) | Data processing method and device for business analysis | |
| CN111400361A (en) | Data real-time storage method and device, computer equipment and storage medium | |
| CN113254320A (en) | Method and device for recording user webpage operation behaviors | |
| CN114091704B (en) | Alarm suppression method and device | |
| CN115033876A (en) | Log processing method, log processing device, computer device and storage medium | |
| US11568344B2 (en) | Systems and methods for automated pattern detection in service tickets | |
| CN112948223A (en) | Method and device for monitoring operation condition | |
| CN110442439B (en) | Task process processing method and device and computer equipment | |
| CN116781568A (en) | Data monitoring alarm method, device, equipment and storage medium | |
| CN115964392A (en) | Real-time monitoring method, device and equipment based on flink and readable storage medium | |
| CN112015623B (en) | Report data processing method, device, equipment and readable storage medium | |
| CN110677271B (en) | Big data alarm method, device, equipment and storage medium based on ELK | |
| CN110457089B (en) | Data acquisition method, data acquisition device, computer readable storage medium and computer equipment | |
| CN112052398A (en) | Media information recommendation method and device, electronic equipment and storage medium | |
| CN110619541B (en) | Application program management method, device, computer equipment and storage medium | |
| CN111552674A (en) | Log processing method and device | |
| CN113590447B (en) | Buried point processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |