CN110362544B - Log processing system, log processing method, terminal and storage medium - Google Patents
Log processing system, log processing method, terminal and storage medium Download PDFInfo
- Publication number
- CN110362544B CN110362544B CN201910447683.8A CN201910447683A CN110362544B CN 110362544 B CN110362544 B CN 110362544B CN 201910447683 A CN201910447683 A CN 201910447683A CN 110362544 B CN110362544 B CN 110362544B
- Authority
- CN
- China
- Prior art keywords
- log
- log data
- processing
- result
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 35
- 238000003860 storage Methods 0.000 title claims abstract description 19
- 238000009826 distribution Methods 0.000 claims abstract description 19
- 238000004458 analytical method Methods 0.000 claims description 31
- 238000000034 method Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 15
- 230000006870 function Effects 0.000 description 9
- 241001178520 Stomatepia mongo Species 0.000 description 7
- 230000002452 interceptive effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 239000000523 sample Substances 0.000 description 3
- 238000001914 filtration Methods 0.000 description 2
- 238000003306 harvesting Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention provides a log processing system, which comprises a log acquisition module, a log processing module and a log processing module, wherein the log acquisition module is used for acquiring log data; the Kafka log distribution cluster is used for distributing log data to obtain first distributed log data and second distributed log data; the elastic search cluster is used for performing first processing on the first shunting log data to obtain a first result; the HBase cluster is used for obtaining a second result after performing second processing on the second shunt log data; and the result display module is used for displaying the first result and/or the second result. The embodiment of the invention also provides a log processing method, a terminal and a computer readable storage medium. By using the embodiment of the invention, the elastic search cluster stores short-term log data, log real-time processing is carried out, and the HBase cluster is mainly responsible for offline log data processing, so that the log processing efficiency is improved.
Description
Technical Field
The invention relates to the technical field of log generation process optimization, in particular to a log processing system, a log processing method, a terminal and a computer readable storage medium.
Background
With the development of computers and networks, the data processing amount of log data is increasing, and the data magnitude of log data is usually more than one million, even more than one million and one trillion. For such a huge log data system, higher requirements are first put forward on the processing of log data. In the prior art, a log system generally adopts two schemes, namely an ELK (electronic search and analysis engine) architecture, wherein the ELK is a basic architecture taking an elastic search (tool for collecting, analyzing and filtering logs), a Logstar (a Web-based graphical interface for searching, analyzing and visualizing log data stored in an elastic search index) and Kibana (a Web-based graphical interface) as core kits, and the method has good instantaneity and convenient query, but is not suitable for providing logs for the outside in a large scale because the elastic search query is an Http protocol; and the system is also based on a Hadoop architecture, so that logs can be gathered, and then log files are provided outside, but the real-time performance is poor, and the inquiry is not convenient enough.
Thus, there is a need for an improved method for log data processing.
Disclosure of Invention
In view of the foregoing, it is necessary to provide a log processing system, a log processing method, a terminal, and a computer readable storage medium, which can combine ELK ecology with Hadoop ecology, store short-term log data by an elastic search cluster, mainly take charge of real-time processing of log data, and take charge of offline log data processing by an HBase cluster, thereby improving log processing efficiency.
A first aspect of an embodiment of the present invention provides a log processing system, including:
the log acquisition module is used for acquiring log data;
the Kafka log distribution cluster is used for distributing log data to obtain first distributed log data and second distributed log data;
the elastic search cluster is used for performing first processing on the first shunting log data to obtain a first result;
the HBase cluster is used for obtaining a second result after performing second processing on the second shunt log data;
and the result display module is used for displaying the first result and/or the second result.
A second aspect of the present invention provides a log processing method for performing log processing by using the log processing system, where the log processing method includes:
acquiring log data;
shunting the log data to obtain first shunting log data and second shunting log data;
inputting the first split log data into the elastic search cluster for first processing to obtain a first result;
inputting the second shunt log data into the HBase cluster for second processing to obtain a second result;
the first result and/or the second result are shown.
Further, in the above log processing method provided by the embodiment of the present invention, the splitting the log data to obtain first split log data and second split log data includes:
converting the acquired log data into a Kafka message queue through the Kafka log distribution cluster;
and carrying out shunting processing on the log data cached in the Kafka message queue, wherein the log data are divided into real-time log data and non-real-time log data, the first shunting log data are real-time log data, and the second shunting log data are non-real-time log data.
Further, in the log processing method provided by the embodiment of the present invention, before the first split log data is input into the elastic search cluster and subjected to the first processing to obtain a first result, the method further includes:
receiving real-time log data in different topics cached in the Kafka message queue;
and analyzing the real-time log data according to a preset analysis rule by using a Logstar log analysis module.
Further, in the log processing method provided by the embodiment of the present invention, the inputting the first split log data into the elastic search cluster to perform the first processing to obtain a first result includes:
storing the parsed first split log data through the elastic search cluster;
performing real-time log data processing on the first split log data to obtain a real-time log data processing result, wherein the real-time log data processing comprises one or more of the following combinations: real-time retrieval processing, real-time alarm processing and online statistics processing.
Further, in the log processing method provided by the embodiment of the present invention, before the second result is obtained after the second split log data is input into the HBase cluster and is subjected to the second processing, the method further includes:
reading a preset analysis rule;
and analyzing the second shunt log data according to a preset analysis rule through the Spark cluster.
Further, in the log processing method provided by the embodiment of the present invention, the inputting the second split log data into the HBase cluster to perform the second processing to obtain a second result includes:
the HBase cluster is used for storing the second shunt log data after the analysis processing;
performing offline log data processing on the second shunt log data to obtain an offline log data processing result, wherein the offline log data processing comprises one or more of the following steps: offline analysis processing, log backup processing and log restoration processing.
Further, in the log processing method provided by the embodiment of the present invention, the displaying the first result and/or the second result includes:
acquiring log data information currently being processed by the log processing system;
when the log data information currently being processed by the log processing system is first split log data, displaying the first result;
and when the log data information currently being processed by the log processing system is second shunt log data, displaying the second result.
The third aspect of the embodiment of the present invention further provides a terminal, where the terminal includes a processor, and the processor is configured to implement the log processing system or implement any one of the log processing methods when executing a computer program stored in a memory.
The fourth aspect of the embodiment of the present invention further provides a computer readable storage medium, where a computer program is stored, where the computer program when executed by a processor implements the log processing system or implements the log processing method according to any one of the above.
The embodiment of the invention provides a log processing system, a log processing method, a terminal and a computer readable storage medium, wherein the log processing system comprises: the log acquisition module is used for acquiring log data; the Kafka log distribution cluster is used for distributing log data to obtain first distributed log data and second distributed log data; the elastic search cluster is used for performing first processing on the first shunting log data to obtain a first result; the HBase cluster is used for obtaining a second result after performing second processing on the second shunt log data; and the result display module is used for displaying the first result and/or the second result. By utilizing the embodiment of the invention, ELK ecology and Hadoop ecology can be combined, the elastic search cluster stores short-term log data and is mainly responsible for real-time processing of the log data, and the HBase cluster is mainly responsible for offline log data processing, so that the performance and instantaneity of real-time processing are ensured, and the log processing efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a log processing method according to a first embodiment of the present invention.
Fig. 2 is a schematic structural diagram of a log processing system according to a first embodiment of the present invention.
Fig. 3 is a schematic structural view of a terminal according to an embodiment of the present invention.
Fig. 4 is an exemplary functional block diagram of the terminal shown in fig. 3.
Description of the main reference signs
The following detailed description will further illustrate embodiments of the invention in conjunction with the above-described drawings.
Detailed Description
In order that the above-recited objects, features and advantages of embodiments of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description. In addition, features in the embodiments of the present application may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the invention, and the described embodiments are merely some, rather than all, of the embodiments of the invention. All other embodiments, based on the embodiments of the invention, which are obtained by a person of ordinary skill in the art without making any inventive effort, are within the scope of the embodiments of the invention.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which embodiments of the invention belong. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
Fig. 1 is a flowchart of a log processing method according to a first embodiment of the present invention. The log processing method may be applied to the terminal 1, and the terminal 1 may be a smart device such as a smart phone, a notebook computer, a desktop/tablet computer, a smart watch, and a personal digital assistant (Personal Digital Assistant, PDA). As shown in fig. 1, the log processing method may include the steps of:
s101: log data is obtained.
In this embodiment, the log data obtained from the operation platform of the related application through the log collection module may include user behavior data, application state data or device state data, which is not limited herein in content and source of the log data. The log collection module may use filebat to collect log data (referred to as a filebat log collection module), where filebat is a log data collector. The filebed log acquisition module supports customizing senders of various log data in the log processing system 100, and is used for acquiring log data and outputting the log data to receivers of various log data. Specifically, the filebed log collection module starts one or more probes (probes) to detect a specified log directory or file; for each log file found by the probe, the filebed log acquisition module initiates a harvesting process (harvester); each of the harvests reads the new content of one log file and sends the new content of the log file to a handler (spooner), which gathers the log data, and finally the filebed log acquisition module sends the gathered log data to a designated location. It will be appreciated that after the log data is obtained, the method further comprises: the log data is converted according to a preset structure, and specifically, the preset structure of the log data can include log time, log level, log output class, log content and the like.
S102: and shunting the log data to obtain first shunting log data and second shunting log data.
In this embodiment, the log data is collected by the log collection module, and the log data is pushed to the Kafka log distribution cluster to be used as a buffer layer of the log data. The Kafka log distribution cluster is a distributed message cache middleware and has the characteristic of high throughput (even if very common hardware is used, kafka can support hundreds of thousands of messages per second) and is used for caching massive data, and data are distributed and controlled in a message queue mode. The Kafka log distribution cluster may convert received log data into a Kafka message queue. The Kafka log distribution cluster may perform splitting processing on the log data cached in the Kafka message queue, and the elastic search cluster and the HBase cluster are consumers of the Kafka log distribution cluster. That is, the Kafka log distribution cluster may output one log data to the elastic search cluster and one log data to the HBase cluster.
The step of shunting the log data to obtain first shunt log data and second shunt log data comprises the following steps: and shunting the log data through the Kafka log distribution cluster to divide the log data into real-time log data and non-real-time log data, wherein the first shunting log data is the real-time log data, and the second shunting data is the non-real-time log data. Outputting the first split log data to the elastic search cluster; and outputting the second shunt log data to the HBase cluster. The shunting of the log data through the Kafka log distribution cluster comprises adopting a Strom flow type computing framework to analyze and process the log data cached in the Kafka message queue to obtain real-time log data and non-real-time log data. In other embodiments, the log data may be classified by a Zookeeper (Zookeeper is a distributed, open source distributed application coordination service) log distribution cluster to obtain first split log data and second split log data.
S103: and inputting the first shunt log data into the elastic search cluster to perform first processing to obtain a first result.
In this embodiment, the first split log data is real-time log data, and before the first split log data is input into the elastic search cluster and subjected to the first processing to obtain a first result, the method further includes: receiving real-time log data in different topics cached in the Kafka message queue; and analyzing the real-time log data according to a preset analysis rule by using a Logstar log analysis module. Analyzing the real-time log data through the logstar log analysis module according to a preset analysis rule comprises cleaning and processing the first split log data through the logstar log analysis module, and structuring the first split log data into different fields. And analyzing the log file through a Logstar log analysis module, identifying useful information in the first shunt log data to be processed, and filtering out junk data. The Logstar log analysis module is configured with analysis files of all log sources, and the preset analysis rules are rules set in the analysis files.
And outputting the first split log data subjected to the analysis processing of the Logstar log analysis module to the elastic search cluster. The step of inputting the first split log data into the elastic search cluster to perform a first process to obtain a first result includes: storing the parsed first split log data through the elastic search cluster; performing real-time log data processing on the first split log data to obtain a real-time log data processing result, wherein the real-time log data processing comprises one or more of the following combinations: real-time retrieval processing, real-time alarm processing and online statistics processing. The method for storing the log data by the elastic search cluster is a distributed storage method, and the first shunt log data maps the key words with the log data in a reverse index mode. Wherein, the key words comprise time, field, key words and the like. And the index is segmented, different segments exist on different cluster nodes, log data can be backed up to prevent the files from being lost due to node damage, log data information can be displayed, and the needed information can be quickly searched in a mode of inputting the key words (such as time, fields and keywords).
S104: and inputting the second shunt log data into the HBase cluster for second processing to obtain a second result.
In this embodiment, the second split log data is non-real-time log data, and before the second split log data is input into the HBase cluster to perform the second processing to obtain the second result, the method further includes: reading a preset analysis rule; and analyzing the second split log data through the Spark cluster according to a preset analysis rule, analyzing the second split log data into an HBase data table format, and storing the analyzed HBase data table format into the HBase cluster. The predetermined parsing rule may be preset by a system developer, and the predetermined parsing rule may include one or more of regular expression, keyValue parsing, field value splitting (for example, splitting by using a split function), string type conversion into numerical values, json parsing, URL decoding, timestamp identification, and UserAgent parsing.
And outputting the analyzed second shunt log data to the HBase cluster. The step of inputting the second shunt log data into the HBase cluster to perform second processing to obtain a second result includes: the HBase cluster is used for storing the second shunt log data after the analysis processing; performing offline log data processing on the second shunt log data to obtain an offline log data processing result, wherein the offline log data processing comprises one or more of the following steps: offline analysis processing, log backup processing and log restoration processing.
S105: the first result and/or the second result are shown.
In this embodiment, the first result and/or the second result is displayed by a result display module, and the result display module is stored in the Web client. The displaying the first result and/or the second result comprises: acquiring log data information currently being processed by the log processing system 100; when the log data information currently being processed by the log processing system 100 is first split log data, displaying the first result; when the log data information currently being processed by the log processing system 100 is second split log data, the second result is presented.
The embodiment of the invention also provides a Mysql database, a Mongo database and a Web application program. The Web application program is connected with the Mysql database and the Mongo database. The Mysql database is a relational database management system of open source codes, and mainly stores resource configuration related data. The Mongo database is a database based on distributed file storage, and aims to provide an extensible high-performance data storage solution for WEB applications, and the Mongo database mainly stores statistical analysis results of log data.
The Web application program is also connected with the Web server, the Web server is used for receiving the interactive data which is uploaded by the Web client and used for carrying out data interaction with the Web application program, outputting the interactive data to the Web application program through an interface, processing the interactive data by the Web application program to obtain a processing result, feeding back the processing result to the Web server, feeding back the processing result to the client through the Web server, and displaying the result through a result display module in the client.
The embodiment of the invention provides a log processing method, which is used for acquiring log data; shunting the log data to obtain first shunting log data and second shunting log data; inputting the first split log data into the elastic search cluster for first processing to obtain a first result; inputting the second shunt log data into the HBase cluster for second processing to obtain a second result; the first result and/or the second result are shown. By utilizing the embodiment of the invention, ELK ecology and Hadoop ecology can be combined, the elastic search cluster stores short-term log data and is mainly responsible for real-time processing of the log data, the HBase cluster is mainly responsible for offline log data processing, and when some time-consuming offline analysis tasks are operated on the HBase cluster, real-time processing such as log inquiry and alarm can be simultaneously carried out in the elastic search cluster, so that the log processing efficiency is improved.
Fig. 2 is a schematic structural diagram of a log processing system according to a first embodiment of the present invention. As shown in fig. 2, the log processing system 100 includes a log collection module 101, a Kafka log distribution cluster 102, an elastic search cluster 103, an HBase cluster 104, and a result display module 105 (the result display module 105 is not shown in the figure, and the result display module is saved in a Web client, which is also not shown in the figure). Wherein the Log collection module 101 may be configured to obtain Log data (Log); the Kafka log distribution cluster 102 may be configured to split log data to obtain first split log data and second split log data; the first split log data is the real-time log data, and the second split log data is the non-real-time log data. Before the first split log data is output to the elastic search cluster, the log-analyzing module is required to analyze (i.e. clean and process the first split log data and structure the first split log data into different fields) the first split log data according to a preset analysis rule. Before the second split log data is output to the HBase cluster, the second split log data needs to be analyzed through a Spark cluster according to a predetermined analysis rule. The elastic search cluster can be used for obtaining a first result after performing first processing on the first split log data; the HBase cluster can be used for obtaining a second result after performing second processing on the second shunt log data; the result display module may be configured to display the first result and/or the second result. The embodiment of the invention also provides a Mysql database, a Mongo database and a Web application program. The Web application program is connected with the Mysql database and the Mongo database. The Mysql database is mainly used for storing resource allocation related data, and the Mongo database is mainly used for storing statistical analysis results of log data. The Web application program is also connected with the Web server, the Web server is used for receiving the interactive data which is uploaded by the Web client and used for carrying out data interaction with the Web application program, outputting the interactive data to the Web application program through an interface, processing the interactive data by the Web application program to obtain a processing result, feeding back the processing result to the Web server, feeding back the processing result to the client through the Web server, and displaying the result through a result display module in the client.
Fig. 3 is a schematic structural diagram of a terminal 1 according to an embodiment of the present invention, and as shown in fig. 3, the terminal 1 includes a memory 10, and the memory 10 stores a log processing system 100. The terminal 1 may be a mobile phone, a tablet computer, a personal digital assistant, or the like, which has an application display function. The log processing system 100 may obtain log data; shunting the log data to obtain first shunting log data and second shunting log data; inputting the first split log data into the elastic search cluster for first processing to obtain a first result; inputting the second shunt log data into the HBase cluster for second processing to obtain a second result; the first result and/or the second result are shown. By utilizing the embodiment of the invention, ELK ecology and Hadoop ecology can be combined, the elastic search cluster stores short-term log data and is mainly responsible for real-time processing of the log data, and the HBase cluster is mainly responsible for offline log data processing, so that the log processing efficiency is improved.
In this embodiment, the terminal 1 may further include a display 20 and a processor 30. The memory 10 and the display 20 may be electrically connected to the processor 30, respectively.
The memory 10 may be a different type of storage device for storing various types of data. For example, the memory and the internal memory of the terminal 1 may be provided, and a memory Card such as a flash memory, an SM Card (Smart Media Card), an SD Card (Secure Digital Card ) or the like may be provided externally to the terminal device 1. In addition, memory 10 may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card), at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 10 is used for storing various data, such as various Applications (Applications) installed in the terminal 1, data set and acquired by applying the above log processing method, and the like.
A display 20 is mounted to the terminal 1 for displaying information.
The processor 30 is configured to execute the log processing method and various software installed in the terminal 1, such as an operating system and application display software. The processor 30 includes, but is not limited to, a processor (Central Processing Unit, CPU), a micro control unit (Micro Controller Unit, MCU), etc., for interpreting the computer and processing the data in the computer software.
The log processing system 100 may include one or more modules stored in the memory 10 of the terminal 1 and configured to be executed by one or more processors (one processor 30 in this embodiment) to complete the embodiment of the present invention. For example, referring to fig. 4, the log processing system 100 may include a log collection module 101, a Kafka log distribution cluster 102, an elastic search cluster 103, an HBase cluster 104, and a result presentation module 105. Modules may be referred to in the embodiments of the present invention as program segments, which perform a particular function, more favorably than programs for describing the execution of software in a processor.
It will be appreciated that, corresponding to each embodiment of the above log processing method, the terminal 1 may include some or all of the functional modules shown in fig. 4, and the functions of each module will be described in detail below. It should be noted that the same noun related nouns and their specific explanations in the above embodiments of the log processing method may also be applied to the following functional descriptions of each module. For the sake of space saving and repetition avoidance, the description is omitted.
The log acquisition module 101 may be used to acquire log data.
The Kafka log distribution cluster 102 may be configured to split log data to obtain first split log data and second split log data.
The elastic search cluster 103 may be configured to perform a first process on the first split log data to obtain a first result.
HBase cluster 104 may be configured to perform a second processing on the second split log data to obtain a second result.
The result presentation module 105 may be configured to present the first result and/or the second result.
The embodiment of the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the log processing method in any of the above embodiments.
The modules/units of the log processing system 100/terminal 1/computer device integration, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, the present invention may implement all or part of the flow of the method of the foregoing embodiment, or may be implemented by instructing related hardware by a computer program, where the computer program may be stored on a computer readable storage medium, and the computer program may implement the steps of each of the method embodiments described above when executed by a processor. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable storage medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth.
The processor 30 may be a central processing unit (Central Processing Unit, CPU), other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, and the processor 30 is a control center of the log processing system 100/terminal 1, and connects the various parts of the entire log processing system 100/terminal 1 using various interfaces and lines.
The memory 10 is used for storing the computer program and/or the module, and the processor 30 implements various functions of the log processing system 100/terminal 1 by running or executing the computer program and/or the module stored in the memory and calling data stored in the memory 10. The memory 10 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data (such as audio data, phonebook, etc.) created according to the use of the handset, etc.
In the several embodiments provided in the present invention, it should be understood that the disclosed terminal and method may be implemented in other manners. For example, the system embodiments described above are merely illustrative, e.g., the division of the modules is merely a logical function division, and other manners of division may be implemented in practice.
It will be evident to those skilled in the art that the embodiments of the invention are not limited to the details of the foregoing illustrative embodiments, and that the embodiments of the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of embodiments being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. A plurality of units, modules or means recited in the claims can also be implemented by means of software or hardware by means of one and the same unit, module or means.
The foregoing embodiments are merely for illustrating the technical solution of the embodiment of the present invention, but not for limiting the same, although the embodiment of the present invention has been described in detail with reference to the foregoing preferred embodiments, it will be understood by those skilled in the art that modifications and equivalent substitutions may be made to the technical solution of the embodiment of the present invention without departing from the spirit and scope of the technical solution of the embodiment of the present invention.
Claims (10)
1. A log processing system, the log processing system comprising:
the log acquisition module is used for acquiring log data;
the Kafka log distribution cluster is used for distributing log data to obtain first distributed log data and second distributed log data, wherein the first distributed log data is real-time log data, and the second distributed log data is non-real-time log data;
the elastic search cluster is used for performing first processing on the first shunting log data to obtain a first result;
the HBase cluster is used for obtaining a second result after performing second processing on the second shunt log data;
and the result display module is used for displaying the first result and/or the second result.
2. A log processing method for log processing using the log processing system according to claim 1, wherein the log processing method comprises:
acquiring log data;
shunting the log data to obtain first shunting log data and second shunting log data;
inputting the first split log data into the elastic search cluster for first processing to obtain a first result;
inputting the second shunt log data into the HBase cluster for second processing to obtain a second result;
the first result and/or the second result are shown.
3. The method of log processing according to claim 2, wherein the splitting the log data to obtain the first split log data and the second split log data includes:
converting the acquired log data into a Kafka message queue through the Kafka log distribution cluster;
and carrying out shunting processing on the log data cached in the Kafka message queue, and dividing the log data into real-time log data and non-real-time log data.
4. The log processing method as set forth in claim 3, wherein before the first result is obtained after the first processing is performed by inputting the first split log data into the elastic search cluster, the method further comprises:
receiving real-time log data in different topics cached in the Kafka message queue;
and analyzing the real-time log data according to a preset analysis rule by using a Logstar log analysis module.
5. The log processing method of claim 4, wherein the inputting the first split log data into the elastic search cluster for the first processing to obtain a first result comprises:
storing the analyzed first split log data through the elastic search cluster;
performing real-time log data processing on the first split log data to obtain a real-time log data processing result, wherein the real-time log data processing comprises one or more of the following combinations: real-time retrieval processing, real-time alarm processing and online statistics processing.
6. The log processing method according to claim 3, wherein before the second result is obtained after the second processing is performed by inputting the second split log data into the HBase cluster, the method further comprises:
reading a preset analysis rule;
and analyzing the second shunt log data according to a preset analysis rule through the Spark cluster.
7. The log processing method according to claim 6, wherein the inputting the second split log data into the HBase cluster for the second processing to obtain a second result includes:
storing the analyzed second shunt log data through the HBase cluster;
performing offline log data processing on the second shunt log data to obtain an offline log data processing result, wherein the offline log data processing comprises one or more of the following steps: offline analysis processing, log backup processing and log restoration processing.
8. The log processing method according to claim 2, wherein the presenting the first result and/or the second result comprises:
acquiring log data information currently being processed by the log processing system;
when the log data information currently being processed by the log processing system is first split log data, displaying the first result;
and when the log data information currently being processed by the log processing system is second shunt log data, displaying the second result.
9. A terminal comprising a processor for implementing the log processing system according to claim 1 or the log processing method according to any one of claims 2 to 8 when executing a computer program stored in a memory.
10. A computer readable storage medium having a computer program stored thereon, wherein the computer program when executed by a processor implements the log processing system of claim 1 or the log processing method of any of claims 2 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910447683.8A CN110362544B (en) | 2019-05-27 | 2019-05-27 | Log processing system, log processing method, terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910447683.8A CN110362544B (en) | 2019-05-27 | 2019-05-27 | Log processing system, log processing method, terminal and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110362544A CN110362544A (en) | 2019-10-22 |
| CN110362544B true CN110362544B (en) | 2024-04-02 |
Family
ID=68215356
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910447683.8A Active CN110362544B (en) | 2019-05-27 | 2019-05-27 | Log processing system, log processing method, terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110362544B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110855770A (en) * | 2019-11-07 | 2020-02-28 | 京东数字科技控股有限公司 | Message processing method and device, electronic equipment and computer readable storage medium |
| CN110990218B (en) * | 2019-11-22 | 2023-12-26 | 深圳前海环融联易信息科技服务有限公司 | Visualization and alarm method and device based on massive logs and computer equipment |
| CN111008093A (en) * | 2019-12-22 | 2020-04-14 | 北京浪潮数据技术有限公司 | Fault log query method, device, equipment and medium |
| CN111241078B (en) * | 2020-01-07 | 2024-06-21 | 网易（杭州）网络有限公司 | Data analysis system, data analysis method and device |
| CN111262915B (en) * | 2020-01-10 | 2020-09-22 | 北京东方金信科技有限公司 | Kafka cluster-crossing data conversion system and method |
| CN111427858A (en) * | 2020-03-18 | 2020-07-17 | 中国邮政储蓄银行股份有限公司 | Log processing system and processing method thereof |
| CN111125121B (en) * | 2020-03-30 | 2020-07-03 | 四川新网银行股份有限公司 | Real-time data display method based on HBase table |
| CN111884883A (en) * | 2020-07-29 | 2020-11-03 | 北京宏达隆和科技有限公司 | Quick auditing processing method for service interface |
| CN112100148B (en) * | 2020-07-31 | 2022-10-28 | 紫光云(南京)数字技术有限公司 | Increment processing method for packed log |
| CN112860456B (en) * | 2021-02-08 | 2023-07-21 | 青岛海尔科技有限公司 | Log processing method and device |
| CN115190139A (en) * | 2022-03-28 | 2022-10-14 | 北京慧能分享科技有限公司 | Multi-protocol-based load balancing energy big data acquisition system and method |
| CN117215964B (en) * | 2023-11-09 | 2024-02-09 | 中央军委政治工作部军事人力资源保障中心 | Program anomaly observation method and device for service system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103399887A (en) * | 2013-07-19 | 2013-11-20 | 蓝盾信息安全技术股份有限公司 | Query and statistical analysis system for mass logs |
| CN103838867A (en) * | 2014-03-20 | 2014-06-04 | 网宿科技股份有限公司 | Log processing method and device |
| CN105933736A (en) * | 2016-04-18 | 2016-09-07 | 天脉聚源(北京)传媒科技有限公司 | Log processing method and device |
| CN107918621A (en) * | 2016-10-10 | 2018-04-17 | 阿里巴巴集团控股有限公司 | Daily record data processing method, device and operation system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11888884B2 (en) * | 2016-12-29 | 2024-01-30 | Bce Inc. | Cyber threat intelligence system infrastructure |
-
2019
- 2019-05-27 CN CN201910447683.8A patent/CN110362544B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103399887A (en) * | 2013-07-19 | 2013-11-20 | 蓝盾信息安全技术股份有限公司 | Query and statistical analysis system for mass logs |
| CN103838867A (en) * | 2014-03-20 | 2014-06-04 | 网宿科技股份有限公司 | Log processing method and device |
| CN105933736A (en) * | 2016-04-18 | 2016-09-07 | 天脉聚源(北京)传媒科技有限公司 | Log processing method and device |
| CN107918621A (en) * | 2016-10-10 | 2018-04-17 | 阿里巴巴集团控股有限公司 | Daily record data processing method, device and operation system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110362544A (en) | 2019-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110362544B (en) | Log processing system, log processing method, terminal and storage medium | |
| CN110347716B (en) | Log data processing method, device, terminal equipment and storage medium | |
| CN111241078B (en) | Data analysis system, data analysis method and device | |
| US10205643B2 (en) | Systems and methods for monitoring and analyzing performance in a computer system with severity-state sorting | |
| US10469344B2 (en) | Systems and methods for monitoring and analyzing performance in a computer system with state distribution ring | |
| US9959015B2 (en) | Systems and methods for monitoring and analyzing performance in a computer system with node pinning for concurrent comparison of nodes | |
| CN113360554B (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| CN111522922A (en) | Log information query method and device, storage medium and computer equipment | |
| CN107861981B (en) | Data processing method and device | |
| CN111352800A (en) | Big data cluster monitoring method and related equipment | |
| WO2013106595A2 (en) | Processing store visiting data | |
| CN111400361A (en) | Data real-time storage method and device, computer equipment and storage medium | |
| CN111949850A (en) | Multi-source data acquisition method, device, equipment and storage medium | |
| CN113051460A (en) | Elasticissearch-based data retrieval method and system, electronic device and storage medium | |
| CN113220710B (en) | Data query method, device, electronic equipment and storage medium | |
| CN114139040A (en) | Data storage and query method, device, equipment and readable storage medium | |
| CN112732663A (en) | Log information processing method and device | |
| CN112506887A (en) | Vehicle terminal CAN bus data processing method and device | |
| CN111159135A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111723063A (en) | Method and device for processing offline log data | |
| CN115422448A (en) | Message pushing method and device, electronic equipment and storage medium | |
| CN113326305A (en) | Method and device for processing data | |
| Ismail et al. | Reference architecture for search infrastructure | |
| CN113076254A (en) | Test case set generation method and device | |
| CN113778777A (en) | Log playback method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |