CN110362544A - Log processing system, log processing method, terminal and storage medium - Google Patents
Log processing system, log processing method, terminal and storage medium Download PDFInfo
- Publication number
- CN110362544A CN110362544A CN201910447683.8A CN201910447683A CN110362544A CN 110362544 A CN110362544 A CN 110362544A CN 201910447683 A CN201910447683 A CN 201910447683A CN 110362544 A CN110362544 A CN 110362544A
- Authority
- CN
- China
- Prior art keywords
- daily record
- record data
- log
- processing
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 35
- 238000003860 storage Methods 0.000 title claims abstract description 28
- 238000000034 method Methods 0.000 claims description 25
- 238000004590 computer program Methods 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 10
- 238000004458 analytical method Methods 0.000 claims description 9
- 238000002224 dissection Methods 0.000 claims description 8
- 230000008901 benefit Effects 0.000 claims description 3
- 230000006870 function Effects 0.000 description 10
- 230000003993 interaction Effects 0.000 description 8
- 241001178520 Stomatepia mongo Species 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 241001269238 Data Species 0.000 description 3
- 230000006399 behavior Effects 0.000 description 3
- 235000013399 edible fruits Nutrition 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 238000003306 harvesting Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000005194 fractionation Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The embodiment of the present invention provides a kind of log processing system, including log acquisition module, for obtaining daily record data；Cluster is distributed in Kafka log, for shunting to daily record data, obtains the first shunting daily record data and second and shunts daily record data；Elasticsearch cluster, for obtaining the first result after carrying out the first processing to the first shunting daily record data；HBase cluster, for obtaining the second result after carrying out second processing to the second shunting daily record data；As a result display module, for showing first result and/or second result.The embodiment of the present invention also provides a kind of log processing method, terminal and computer readable storage medium.It is carried out log by the short-term daily record data of Elasticsearch cluster-based storage using the embodiment of the present invention and is handled in real time, offline daily record data is mainly responsible for by HBase cluster and is handled, to improve log processing efficiency.
Description
Technical field
The present invention relates to log generating process optimisation technique fields, and in particular to a kind of log processing system, log processing
Method, terminal and computer readable storage medium.
Background technique
With the development of computer and networks, the data processing amount of daily record data is increasing, the data volume of daily record data
Grade is usually million grades or more, more than even hundred tera-scale, thousand tera-scale.It is first for so huge daily record data system
Higher requirement first is referred to the processing of daily record data.In the prior art, log system generally takes two schemes, one is
Using the framework of ELK, ELK is that one kind (is used to Elasticsearch (real-time full-text search and analysis engine), Logstash
Collect, analysis and the tool of filtering log) and a kind of Kibana (graphical interfaces based on Web, for searching for, analyzing and can
View is stored in the daily record data in Elasticsearch index) for three as the basic framework of core external member, this mode is real-time
Property it is good, inquiry is convenient, but due to Elasticsearch inquiry be Http agreement, so being not suitable for large batch of external offer
Log；There are also one is the framework based on Hadoop, this mode can get up log aggregation, then externally provide log text
Part, but real-time is poor, and inquiry is also not convenient enough.
Therefore, a kind of improved method for daily record data processing is needed now.
Summary of the invention
In view of the foregoing, it is necessary to which providing a kind of log processing system, log processing method, terminal and computer can
Storage medium is read, it can be by ELK ecology in conjunction with Hadoop ecology, by the short-term log of Elasticsearch cluster-based storage
Data are mainly responsible for the real-time processing of daily record data, are responsible for offline daily record data by HBase cluster and handle, improve log
Treatment effeciency.
First aspect of the embodiment of the present invention provides a kind of log processing system, and the log processing system includes:
Log acquisition module, for obtaining daily record data；
Kafka log distributes cluster and obtains the first shunting daily record data and second point for shunting to daily record data
Flow daily record data；
Elasticsearch cluster, for obtaining the first knot after carrying out the first processing to the first shunting daily record data
Fruit；
HBase cluster, for obtaining the second result after carrying out second processing to the second shunting daily record data；
As a result display module, for showing first result and/or second result.
Second aspect of the embodiment of the present invention provides a kind of log that log processing is carried out using above-mentioned log processing system
Processing method, the log processing method include:
Obtain daily record data；
The daily record data is shunted, the first shunting daily record data and second is obtained and shunts daily record data；
The first shunting daily record data is input in the Elasticsearch cluster after carrying out the first processing and is obtained
First result；
The second shunting daily record data is input to after carrying out second processing in the HBase cluster and obtains the second knot
Fruit；
Show first result and/or second result.
Further, in above-mentioned log processing method provided in an embodiment of the present invention, it is described to the daily record data into
Row shunts, and obtains the first shunting daily record data and the second shunting daily record data includes:
The daily record data that distributing cluster by the Kafka log will acquire is converted to Kafka message queue；
Shunting processing is carried out to the daily record data cached in the Kafka message queue, be divided into real-time logs data with it is non-
Real-time logs data, wherein described first shunts daily record data as real-time logs data, and the second shunting daily record data is non-
Real-time logs data.
Further, in above-mentioned log processing method provided in an embodiment of the present invention, described first is shunted described
Daily record data be input in the Elasticsearch cluster carry out first processing after obtain the first result before, the method
Further include:
Receive the real-time logs data in the different topic cached in the Kafka message queue；
Parsing behaviour is carried out to the real-time logs data according to default resolution rules by Logstash log analyzing module
Make.
Further, described to shunt day for described first in above-mentioned log processing method provided in an embodiment of the present invention
Will data are input in the Elasticsearch cluster carry out the first processing after obtain the first result and include:
Described first after dissection process, which is saved, by the Elasticsearch cluster shunts daily record data；
Daily record data is shunted to described first and carries out real-time logs data processing, obtains real-time logs data processed result,
Wherein, the real-time logs data processing includes one of following or a variety of combination: at real-time retrieval processing, Real-time Alarm
Reason is handled with Online statistics.
Further, in above-mentioned log processing method provided in an embodiment of the present invention, described second is shunted described
Daily record data be input in the HBase cluster carry out second processing after obtain the second result before, the method also includes:
Read predetermined resolution rules；
Parsing operation is carried out to the second shunting daily record data according to predetermined resolution rules by Spark cluster.
Further, described to shunt day for described second in above-mentioned log processing method provided in an embodiment of the present invention
Will data are input to carry out second processing in the HBase cluster after obtain the second result and include:
Described second after dissection process, which is saved, by the HBase cluster shunts daily record data；
Daily record data is shunted to described second and carries out offline logs data processing, obtains offline logs data processed result,
Wherein, the offline logs data processing includes one of following or a variety of: off-line analysis processing, Log backup processing and day
Will reduction treatment.
Further, described to show first result in above-mentioned log processing method provided in an embodiment of the present invention
And/or second result includes:
Obtain the daily record data information that the log processing system is presently processing；
When the daily record data information that the log processing system is presently processing is the first shunting daily record data, show
First result；
When the daily record data information that the log processing system is presently processing is the second shunting daily record data, show
Second result.
The third aspect of the embodiment of the present invention also provides a kind of terminal, and the terminal includes processor, and the processor is used for
Log processing system described above is realized when executing the computer program stored in memory or realizes above-mentioned any one institute
State log processing method.
Fourth aspect of the embodiment of the present invention also provides a kind of computer readable storage medium, the computer-readable storage medium
Be stored with computer program in matter, the computer program realized when being executed by processor log processing system described above or
Realize log processing method described in above-mentioned any one.
The embodiment of the present invention provides a kind of log processing system, log processing method, terminal and computer-readable storage
Medium, the log processing system includes: log acquisition module, for obtaining daily record data；Cluster is distributed in Kafka log, uses
It is shunted in daily record data, obtains the first shunting daily record data and second and shunt daily record data；Elasticsearch collection
Group, for obtaining the first result after carrying out the first processing to the first shunting daily record data；HBase cluster, for described
Second shunting daily record data obtains the second result after carrying out second processing；As a result display module, for showing first result
And/or second result.Using the embodiment of the present invention, can by ELK ecology in conjunction with Hadoop ecology, by
The short-term daily record data of Elasticsearch cluster-based storage is mainly responsible for the real-time processing of daily record data, by HBase cluster master
It is responsible for offline daily record data processing, ensure that the performance and real-time handled in real time, to improve log processing efficiency.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The embodiment of invention for those of ordinary skill in the art without creative efforts, can also basis
The attached drawing of offer obtains other attached drawings.
Fig. 1 is the flow chart for the log processing method that first embodiment of the invention provides.
Fig. 2 is the structural schematic diagram for the log processing system that first embodiment of the invention provides.
Fig. 3 is the structural schematic diagram of the terminal of an embodiment of the present invention.
Fig. 4 is the illustrative functional block diagram of terminal shown in Fig. 3.
Main element symbol description
The embodiment of the present invention that the following detailed description will be further explained with reference to the above drawings.
Specific embodiment
In order to be more clearly understood that the above objects, features, and advantages of the embodiment of the present invention, with reference to the accompanying drawing and
The present invention will be described in detail for specific embodiment.It should be noted that in the absence of conflict, the embodiment party of the application
Feature in formula can be combined with each other.
Embodiment in the following description, numerous specific details are set forth in order to facilitate a full understanding of the present invention, described reality
The mode of applying is only some embodiments of the invention, rather than whole embodiments.Based on the embodiment in the present invention,
Every other embodiment obtained by those of ordinary skill in the art without making creative efforts belongs to this
The range of inventive embodiments protection.
Unless otherwise defined, all technical and scientific terms used herein and the technology for belonging to the embodiment of the present invention
The normally understood meaning of the technical staff in field is identical.Term as used herein in the specification of the present invention is intended merely to
The purpose of specific embodiment is described, it is not intended that in the limitation embodiment of the present invention.
Fig. 1 is the flow chart of the log processing method of first embodiment of the invention.The log processing method can answer
For terminal 1, the terminal 1 can be such as smart phone, laptop, desk-top/tablet computer, smartwatch and a
The smart machines such as personal digital assistant (Personal Digital Assistant, PDA).As shown in Figure 1, the log processing side
Method may include steps of:
S101: daily record data is obtained.
In the present embodiment, the daily record data obtained from the operation platform of related application by log acquisition module,
The type of the daily record data may include user behavior data, application state data or device status data, not right herein
The content of daily record data, source are defined.The log acquisition module can be used Filebeat and carry out log data acquisition
(referred to as Filebeat log acquisition module), the Filebeat are log data acquisition device.The Filebeat log collection
Module supports the sender that all kinds of daily record datas are customized in log processing system 100, the Filebeat log acquisition module
It exports for obtaining daily record data, and by the daily record data to the recipient of all kinds of daily record datas.Specifically, described
Filebeat log acquisition module starts the Log Directory or text that one or more detectors (prospectors) go detection specified
Part；For each journal file that the detector is found out, the Filebeat log acquisition module starts harvesting process
(harvester)；Each described harvesting is read out the new content of a journal file, and sends the new of the journal file
For content to processing routine (spooler), the processing routine can gather these daily record datas, and the last Filebeat log is adopted
Collection module can send the daily record data of set to the place specified.It is understood that after the acquisition daily record data, institute
State method further include: convert according to preset structure to the daily record data, specifically, the preset structure of the daily record data
It may include logging time, log rank, log output class and log content etc..
S102: shunting the daily record data, obtains the first shunting daily record data and second and shunts daily record data.
In the present embodiment, the acquisition of daily record data is carried out by log acquisition module, and the daily record data is pushed away
Give cache layer of the Kafka log distribution cluster as the daily record data.Wherein, the Kafka log distribution cluster is one
Kind distributed message caches middleware, and having the characteristics that high-throughput, (even with very common hardware, Kafka can also
To support hundreds of thousands of message per second), for the caching of mass data, by way of message queue, data are distributed
And control.The daily record data received can be converted to Kafka message queue by the Kafka log distribution cluster.It is described
Kafka log distribution cluster can carry out shunting processing to the daily record data cached in the Kafka message queue, described
Elasticsearch cluster and the HBase cluster are the consumer that cluster is distributed in the Kafka log.That is, described
Kafka log distribution cluster can export a daily record data to the Elasticsearch cluster, and a daily record data is defeated
Out to the HBase cluster.
It is described that the daily record data is shunted, it obtains the first shunting daily record data and second and shunts daily record data packet
It includes: cluster being distributed by the Kafka log, the daily record data is shunted, the daily record data is divided into real-time logs
Data and non real-time daily record data, wherein the first shunting daily record data is the real-time logs data, and described second point
Flow data is the non real-time daily record data.Daily record data is shunted for described first, is outputed it to described
Elasticsearch cluster；Daily record data is shunted for described second, outputs it to the HBase cluster.It is described to pass through institute
Stating Kafka log distribution cluster and shunt to the daily record data includes using Strom streaming computing frame must be to described
The daily record data cached in Kafka message queue is analyzed and processed, and obtains real-time logs data and non real-time daily record data.
It in other embodiments, can also (ZooKeeper be one distributed, the distribution of open source code by Zookeeper
Application program coordination service) log distribution cluster classifies to the daily record data, obtain the first shunting daily record data and the
Two shunt daily record data.
S103: the first shunting daily record data is input in the Elasticsearch cluster and carries out the first processing
After obtain the first result.
In the present embodiment, described first daily record data is shunted as real-time logs data, shunting day for described first
Will data be input in the Elasticsearch cluster carry out first processing after obtain the first result before, the method is also
It include: the real-time logs data in the different topic for receiving and being cached in the Kafka message queue；Pass through Logstash log
Parsing module carries out parsing operation to the real-time logs data according to default resolution rules.It is described to pass through Logstash log solution
It includes by Logstash log analyzing module that analysis module, which carries out parsing to the real-time logs data according to default resolution rules,
The first shunting daily record data is cleaned and processed, and the first shunting log data structures are melted into different words
Section.Journal file is parsed by Logstash log analyzing module, can recognize that be processed described first shunts
Useful information in daily record data, filters out junk data.All logs are configured in the Logstash log analyzing module
The resolution file in source, the default resolution rules are the rule being arranged in the resolution file.
By after the Logstash log analyzing module dissection process it is described first shunting daily record data export to
The Elasticsearch cluster.It is described that the first shunting daily record data is input in the Elasticsearch cluster
It includes: described after dissection process by Elasticsearch cluster preservation that the first result is obtained after the first processing of progress
It is described first shunt daily record data；Daily record data is shunted to described first and carries out real-time logs data processing, obtains real-time day
Will data processed result, wherein the real-time logs data processing includes one of following or a variety of combination: real-time retrieval
Processing, Real-time Alarm processing are handled with Online statistics.The Elasticsearch cluster saves the mode that daily record data is taken
Distributed storage mode, described first, which shunts daily record data, carries out crucial literal and daily record data in such a way that inverted order indexes
Mapping.Wherein, the crucial literal includes time, field, keyword etc..One index is subjected to fragment, different fragments is deposited
On different clustered nodes, can backup log data prevent node damage cause file to be lost, daily record data can be believed
Breath is showed, and can by way of inputting the crucial literal (for example, time, field, keyword) fast search
To the information of needs.
S104: the second shunting daily record data is input in the HBase cluster after carrying out second processing and obtains the
Two results.
In the present embodiment, described second daily record data is shunted as non real-time daily record data, shunt by described second
Daily record data be input in the HBase cluster carry out second processing after obtain the second result before, the method also includes: read
Take predetermined resolution rules；Parsing behaviour is carried out to the second shunting daily record data according to predetermined resolution rules by Spark cluster
Make, the second shunting daily record data is resolved into HBase tables of data format, the HBase tables of data format after parsing is stored
To in the HBase cluster.Wherein, it is pre-set to can be system developer for the predetermined resolution rules, described predetermined
Resolution rules may include regular expression, KeyValue parsing, field value fractionation (for example, being torn open using split function
Point), String type be converted into numeric type, Json parsing, URL decoding, time-stamp Recognition and UserAgent parsing in one
Kind is a variety of.
The second shunting daily record data after dissection process is exported to the HBase cluster.It is described will be described
It includes: by described that second shunting daily record data, which is input to and obtains the second result after progress second processing in the HBase cluster,
HBase cluster saves described second after dissection process and shunts daily record data；To it is described second shunt daily record data into
Row offline logs data processing, obtains offline logs data processed result, wherein the offline logs data processing includes following
One of or it is a variety of: off-line analysis processing, Log backup processing with log reduction treatment.
S105: first result and/or second result are shown.
In the present embodiment, show first result and/or described second as a result, described by result display module
As a result display module is saved into Web client.The displaying first result and/or second result include: to obtain
The daily record data information that the log processing system 100 is presently processing；When the log processing system 100 is currently being located
When the daily record data information of reason is the first shunting daily record data, first result is shown；When the log processing system 100 is worked as
When the preceding daily record data information handled is the second shunting daily record data, second result is shown.
The embodiment of the present invention also provides Mysql database, Mongo database and web application.The Web application journey
Sequence is connect with the Mysql database and Mongo database.Wherein, the Mysql database is a kind of pass of open source code
It is type data base management system, mainly stores resource distribution related data in the Mysql database.The Mongo database
It is the database based on distributed document storage, it is intended to provide expansible high-performance data storage for WEB application and solve
Scheme mainly stores the statistic analysis result of daily record data in the Mongo database.
The web application is also connected with each other with Web server, and the Web server is for receiving Web client
What is passed is used to carry out the interaction data of data interaction with web application, and the interaction data is exported by interface to Web
Application program after web application handles interaction data, obtains processing result, and processing result is fed back to Web clothes
It is engaged in device, processing result is fed back to by client by Web server, by the result display module in the client by result
It is shown.
The embodiment of the present invention provides a kind of log processing method, obtains daily record data；The daily record data is shunted,
It obtains the first shunting daily record data and second and shunts daily record data；The first shunting daily record data is input to described
The first result is obtained after carrying out the first processing in Elasticsearch cluster；The second shunting daily record data is input to institute
It states and obtains the second result after carrying out second processing in HBase cluster；Show first result and/or second result.Benefit
It, can be short-term by Elasticsearch cluster-based storage by ELK ecology in conjunction with Hadoop ecology with the embodiment of the present invention
Daily record data is mainly responsible for the real-time processing of daily record data, is mainly responsible for offline daily record data by HBase cluster and handles,
When running some relatively time-consuming off-line analysis tasks on HBase cluster, log can be carried out in Elasticsearch cluster simultaneously
Inquiry and alarm etc. in real time processing, to improve log processing efficiency.
Fig. 2 is the structural schematic diagram for the log processing system that first embodiment of the invention provides.As shown in Fig. 2, described
Log processing system 100 include log acquisition module 101, Kafka log distribution cluster 102, Elasticsearch cluster 103,
(wherein, the result display module 105 is not shown out, and the result is shown for HBase cluster 104 and result display module 105
Module is saved into Web client, and the Web client is being also not shown in the figure).Wherein, the log acquisition module 101 can
For obtaining daily record data (Log)；The Kafka log distribution cluster 102 can be used for shunting daily record data, obtain
Daily record data and second, which is shunted, to first shunts daily record data；The first shunting daily record data is the real-time logs data,
Second streamed data is the non real-time daily record data.It exports by the first shunting daily record data to described
Before Elasticsearch cluster, it is also necessary to by Logstash log analyzing module according to default resolution rules to described
One shunting daily record data parsed (namely to it is described first shunting daily record data cleaned and processed, be structured into difference
Field) operation.It exports by the second shunting daily record data to before the HBase cluster, it is also necessary to pass through Spark collection
Group carries out parsing operation to the second shunting daily record data according to predetermined resolution rules.The Elasticsearch cluster can
For obtaining the first result after carrying out the first processing to the first shunting daily record data；The HBase cluster can be used for
The second result is obtained after carrying out second processing to the second shunting daily record data；As a result display module can be used for showing described
First result and/or second result.The embodiment of the present invention also provides Mysql database, Mongo database and Web and applies
Program.The web application is connect with the Mysql database and Mongo database.Wherein, in the Mysql database
It is main to store resource distribution related data, the statistic analysis result of daily record data is mainly stored in the Mongo database.It is described
Web application also with Web server be connected with each other, the Web server be used for receive Web client upload for
Web application carries out the interaction data of data interaction, and the interaction data is exported by interface to web application,
After web application handles interaction data, processing result is obtained, and processing result is fed back into Web server, passed through
Processing result is fed back to client by Web server, is shown result by the result display module in the client.
Fig. 3 is the structural schematic diagram of the terminal 1 of an embodiment of the present invention, as shown in figure 3, terminal 1 includes memory 10,
Log processing system 100 is stored in memory 10.The terminal 1 can be mobile phone, tablet computer, personal digital assistant etc.
Terminal 1 with application display function.The available daily record data of the log processing system 100；To the daily record data into
Row shunts, and obtains the first shunting daily record data and second and shunts daily record data；The first shunting daily record data is input to institute
It states and obtains the first result after carrying out the first processing in Elasticsearch cluster；The second shunting daily record data is input to
The second result is obtained after carrying out second processing in the HBase cluster；Show first result and/or second result.
It, can be short-term by Elasticsearch cluster-based storage by ELK ecology in conjunction with Hadoop ecology using the embodiment of the present invention
Daily record data, be mainly responsible for the real-time processing of daily record data, offline daily record data be mainly responsible for by HBase cluster and is handled,
To improve log processing efficiency.
In present embodiment, terminal 1 can also include display screen 20 and processor 30.Memory 10, display screen 20 can be with
It is electrically connected respectively with processor 30.
The memory 10 can be different type storage equipment, for storing Various types of data.For example, it may be terminal
1 memory, memory, can also be the storage card that can be external in the terminal installation 1, such as flash memory, SM card (Smart Media
Card, smart media card), SD card (Secure Digital Card, safe digital card) etc..In addition, memory 10 may include
High-speed random access memory can also include nonvolatile memory, such as hard disk, memory, plug-in type hard disk, intelligent storage
Block (Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash card (Flash Card), extremely
A few disk memory, flush memory device or other volatile solid-state parts.Memory 10 is for storing all kinds of numbers
According to for example, the types of applications program (Applications) installed in the terminal 1, setting using above-mentioned log processing method
The information such as the data set, obtained.
Display screen 20 is installed on terminal 1, for showing information.
Processor 30 is used to execute all kinds of softwares installed in the log processing method and the terminal 1, such as grasps
Make system and application display software etc..Processor 30 including but not limited to processor (Central Processing Unit,
CPU), micro-control unit (Micro Controller Unit, MCU) etc. is for interpretive machine and processing computer software
In data device.
The log processing system 100 may include one or more module, and one or more of modules are deposited
Storage is in the memory 10 of terminal 1 and is configured to be held by one or more processors (present embodiment is a processor 30)
Row, to complete the embodiment of the present invention.For example, as shown in fig.4, the log processing system 100 may include log collection mould
Block 101, Kafka log distribution cluster 102, Elasticsearch cluster 103, HBase cluster 104 and result display module
105.The so-called module of the embodiment of the present invention can be the program segment for completing a specific function, than program more suitable for describing software
Implementation procedure in the processor.
It is understood that each embodiment in corresponding above-mentioned log processing method, terminal 1 may include institute in Fig. 4
Part or all in each functional module shown, the function of each module will introduced in detail below.More than it should be noted that
In each embodiment of log processing method identical noun related terms and its it is specific illustrate be readily applicable to
Under to the function introduction of each module.For the sake of saving space and avoiding repetition, details are not described herein again.
Log acquisition module 101 can be used for obtaining daily record data.
Kafka log distribution cluster 102 can be used for shunting daily record data, obtain the first shunting daily record data and
Second shunts daily record data.
Elasticsearch cluster 103 can be used for after carrying out the first processing to the first shunting daily record data obtaining
First result.
HBase cluster 104 can be used for after carrying out second processing to the second shunting daily record data obtaining the second result.
As a result display module 105 can be used for showing first result and/or second result.
The embodiment of the present invention also provides a kind of computer readable storage medium, is stored thereon with computer program, the meter
The step of calculation machine program realizes the log processing method in any of the above-described embodiment when being executed by processor.
If the integrated module/unit of 100/ terminal of log processing system, 1/ computer equipment is with SFU software functional unit
Form realize and when sold or used as an independent product, can store in a computer readable storage medium.
Based on this understanding, the present invention realizes all or part of the process in above embodiment method, can also pass through computer
Program is completed to instruct relevant hardware, and the computer program can be stored in a computer readable storage medium, should
Computer program is when being executed by processor, it can be achieved that the step of above-mentioned each embodiment of the method.Wherein, the computer program
Including computer program code, the computer program code can be source code form, object identification code form, executable file
Or certain intermediate forms etc..The computer readable storage medium may include: that can carry the computer program code
Any entity or device, recording medium, USB flash disk, mobile hard disk, magnetic disk, CD, computer storage, read-only memory (ROM,
Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunications letter
Number and software distribution medium etc..
Alleged processor 30 can be central processing unit (Central Processing Unit, CPU), can also be
Other general processors, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit
(Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be microprocessor or the processor is also possible to any conventional processor
Deng the processor 30 is the control centre of 100/ terminal 1 of log processing system, whole using various interfaces and connection
The various pieces of a 100/ terminal 1 of log processing system.
For the memory 10 for storing the computer program and/or module, the processor 30 is by operation or holds
Row stores computer program and/or module in the memory, and calls the data being stored in memory 10, realizes
The various functions of 100/ terminal 1 of log processing system.The memory 10 can mainly include storing program area and storage number
According to area, wherein storing program area can application program needed for storage program area, at least one function (for example sound plays function
Energy, image player function etc.) etc.；Storage data area can store according to mobile phone use created data (such as audio data,
Phone directory etc.) etc..
In several specific embodiments provided by the present invention, it should be understood that disclosed terminal and method, it can be with
It realizes by another way.For example, system embodiment described above is only schematical, for example, the module
Division, only a kind of logical function partition, there may be another division manner in actual implementation.
It is obvious to a person skilled in the art that the embodiment of the present invention is not limited to the details of above-mentioned exemplary embodiment,
And without departing substantially from the spirit or essential attributes of the embodiment of the present invention, this hair can be realized in other specific forms
Bright embodiment.Therefore, in all respects, the present embodiments are to be considered as illustrative and not restrictive, this
The range of inventive embodiments is indicated by the appended claims rather than the foregoing description, it is intended that being equal for claim will be fallen in
All changes in the meaning and scope of important document are included in the embodiment of the present invention.It should not be by any attached drawing mark in claim
Note is construed as limiting the claims involved.Multiple units, module or the device stated in claim can also be by same
Unit, module or device are implemented through software or hardware.
Embodiment of above is only to illustrate the technical solution of the embodiment of the present invention rather than limits, although referring to above preferable
The embodiment of the present invention is described in detail in embodiment, those skilled in the art should understand that, it can be to this hair
The technical solution of bright embodiment is modified or equivalent replacement should not all be detached from the embodiment of the present invention technical solution spirit and
Range.
Claims (10)
1. a kind of log processing system, which is characterized in that the log processing system includes:
Log acquisition module, for obtaining daily record data；
Cluster is distributed in Kafka log, for shunting to daily record data, obtains the first shunting daily record data and second and shunts day
Will data；
Elasticsearch cluster, for obtaining the first result after carrying out the first processing to the first shunting daily record data；
HBase cluster, for obtaining the second result after carrying out second processing to the second shunting daily record data；
As a result display module, for showing first result and/or second result.
2. a kind of log processing method for carrying out log processing using log processing system as described in claim 1, feature
It is, the log processing method includes:
Obtain daily record data；
The daily record data is shunted, the first shunting daily record data and second is obtained and shunts daily record data；
The first shunting daily record data is input in the Elasticsearch cluster after carrying out the first processing and obtains first
As a result；
The second shunting daily record data is input to after carrying out second processing in the HBase cluster and obtains the second result；
Show first result and/or second result.
3. log processing method according to claim 2, which is characterized in that it is described that the daily record data is shunted,
It obtains the first shunting daily record data and the second shunting daily record data includes:
The daily record data that distributing cluster by the Kafka log will acquire is converted to Kafka message queue；
Shunting processing is carried out to the daily record data cached in the Kafka message queue, be divided into real-time logs data with it is non real-time
Daily record data, wherein described first shunts daily record data as real-time logs data, and the second shunting daily record data is non real-time
Daily record data.
4. log processing method according to claim 3, which is characterized in that shunt daily record data for described first described
It is input in the Elasticsearch cluster before obtaining the first result after carrying out the first processing, the method also includes:
Receive the real-time logs data in the different topic cached in the Kafka message queue；
Parsing operation is carried out to the real-time logs data according to default resolution rules by Logstash log analyzing module.
5. log processing method according to claim 4, which is characterized in that described that the first shunting daily record data is defeated
It obtains the first result after entering into the Elasticsearch cluster to carry out the first processing and includes:
Described first after dissection process, which is saved, by the Elasticsearch cluster shunts daily record data；
Daily record data is shunted to described first and carries out real-time logs data processing, obtains real-time logs data processed result, wherein
The real-time logs data processing includes one of following or a variety of combination: real-time retrieval processing, Real-time Alarm processing with
Online statistics processing.
6. log processing method according to claim 3, which is characterized in that shunt daily record data for described second described
It is input in the HBase cluster before obtaining the second result after carrying out second processing, the method also includes:
Read predetermined resolution rules；
Parsing operation is carried out to the second shunting daily record data according to predetermined resolution rules by Spark cluster.
7. log processing method according to claim 6, which is characterized in that described that the second shunting daily record data is defeated
Enter into the HBase cluster and obtains the second result after carrying out second processing and include:
Described second after dissection process, which is saved, by the HBase cluster shunts daily record data；
Daily record data is shunted to described second and carries out offline logs data processing, obtains offline logs data processed result, wherein
The offline logs data processing includes one of following or a variety of: off-line analysis processing, Log backup processing and log are also
Original place reason.
8. log processing method according to claim 2, which is characterized in that described to show first result and/or institute
Stating the second result includes:
Obtain the daily record data information that the log processing system is presently processing；
When the daily record data information that the log processing system is presently processing is the first shunting daily record data, described in displaying
First result；
When the daily record data information that the log processing system is presently processing is the second shunting daily record data, described in displaying
Second result.
9. a kind of terminal, which is characterized in that the terminal includes processor, and the processor is used to execute to store in memory
Log processing system as described in claim 1 is realized when computer program or is realized such as any one of claim 2 to 8 institute
State log processing method.
10. a kind of computer readable storage medium, computer program, feature are stored on the computer readable storage medium
It is, the computer program is realized log processing system as described in claim 1 or realized as weighed when being executed by processor
Benefit require any one of 2 to 8 described in log processing method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910447683.8A CN110362544B (en) | 2019-05-27 | 2019-05-27 | Log processing system, log processing method, terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910447683.8A CN110362544B (en) | 2019-05-27 | 2019-05-27 | Log processing system, log processing method, terminal and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110362544A true CN110362544A (en) | 2019-10-22 |
| CN110362544B CN110362544B (en) | 2024-04-02 |
Family
ID=68215356
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910447683.8A Active CN110362544B (en) | 2019-05-27 | 2019-05-27 | Log processing system, log processing method, terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110362544B (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110855770A (en) * | 2019-11-07 | 2020-02-28 | 京东数字科技控股有限公司 | Message processing method and device, electronic equipment and computer readable storage medium |
| CN110990218A (en) * | 2019-11-22 | 2020-04-10 | 深圳前海环融联易信息科技服务有限公司 | Visualization and alarm method and device based on mass logs and computer equipment |
| CN111008093A (en) * | 2019-12-22 | 2020-04-14 | 北京浪潮数据技术有限公司 | Fault log query method, device, equipment and medium |
| CN111125121A (en) * | 2020-03-30 | 2020-05-08 | 四川新网银行股份有限公司 | Real-time data display method based on HBase table |
| CN111241078A (en) * | 2020-01-07 | 2020-06-05 | 网易（杭州）网络有限公司 | Data analysis system, data analysis method and device |
| CN111262915A (en) * | 2020-01-10 | 2020-06-09 | 北京东方金信科技有限公司 | Kafka cluster-crossing data conversion system and method |
| CN111427858A (en) * | 2020-03-18 | 2020-07-17 | 中国邮政储蓄银行股份有限公司 | Log processing system and processing method thereof |
| CN111884883A (en) * | 2020-07-29 | 2020-11-03 | 北京宏达隆和科技有限公司 | Quick auditing processing method for service interface |
| CN112100148A (en) * | 2020-07-31 | 2020-12-18 | 紫光云(南京)数字技术有限公司 | Increment processing method for packed log |
| CN112860456A (en) * | 2021-02-08 | 2021-05-28 | 青岛海尔科技有限公司 | Log processing method and device |
| CN115190139A (en) * | 2022-03-28 | 2022-10-14 | 北京慧能分享科技有限公司 | Multi-protocol-based load balancing energy big data acquisition system and method |
| CN117215964A (en) * | 2023-11-09 | 2023-12-12 | 中央军委政治工作部军事人力资源保障中心 | Program anomaly observation method and device for service system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103399887A (en) * | 2013-07-19 | 2013-11-20 | 蓝盾信息安全技术股份有限公司 | Query and statistical analysis system for mass logs |
| CN103838867A (en) * | 2014-03-20 | 2014-06-04 | 网宿科技股份有限公司 | Log processing method and device |
| CN105933736A (en) * | 2016-04-18 | 2016-09-07 | 天脉聚源(北京)传媒科技有限公司 | Log processing method and device |
| CN107918621A (en) * | 2016-10-10 | 2018-04-17 | 阿里巴巴集团控股有限公司 | Daily record data processing method, device and operation system |
| US20180191767A1 (en) * | 2016-12-29 | 2018-07-05 | Bce Inc. | Cyber Threat Intelligence System Infrastructure |
-
2019
- 2019-05-27 CN CN201910447683.8A patent/CN110362544B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103399887A (en) * | 2013-07-19 | 2013-11-20 | 蓝盾信息安全技术股份有限公司 | Query and statistical analysis system for mass logs |
| CN103838867A (en) * | 2014-03-20 | 2014-06-04 | 网宿科技股份有限公司 | Log processing method and device |
| CN105933736A (en) * | 2016-04-18 | 2016-09-07 | 天脉聚源(北京)传媒科技有限公司 | Log processing method and device |
| CN107918621A (en) * | 2016-10-10 | 2018-04-17 | 阿里巴巴集团控股有限公司 | Daily record data processing method, device and operation system |
| US20180191767A1 (en) * | 2016-12-29 | 2018-07-05 | Bce Inc. | Cyber Threat Intelligence System Infrastructure |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110855770A (en) * | 2019-11-07 | 2020-02-28 | 京东数字科技控股有限公司 | Message processing method and device, electronic equipment and computer readable storage medium |
| CN110990218A (en) * | 2019-11-22 | 2020-04-10 | 深圳前海环融联易信息科技服务有限公司 | Visualization and alarm method and device based on mass logs and computer equipment |
| CN110990218B (en) * | 2019-11-22 | 2023-12-26 | 深圳前海环融联易信息科技服务有限公司 | Visualization and alarm method and device based on massive logs and computer equipment |
| CN111008093A (en) * | 2019-12-22 | 2020-04-14 | 北京浪潮数据技术有限公司 | Fault log query method, device, equipment and medium |
| CN111241078A (en) * | 2020-01-07 | 2020-06-05 | 网易（杭州）网络有限公司 | Data analysis system, data analysis method and device |
| CN111262915B (en) * | 2020-01-10 | 2020-09-22 | 北京东方金信科技有限公司 | Kafka cluster-crossing data conversion system and method |
| CN111262915A (en) * | 2020-01-10 | 2020-06-09 | 北京东方金信科技有限公司 | Kafka cluster-crossing data conversion system and method |
| CN111427858A (en) * | 2020-03-18 | 2020-07-17 | 中国邮政储蓄银行股份有限公司 | Log processing system and processing method thereof |
| CN111125121B (en) * | 2020-03-30 | 2020-07-03 | 四川新网银行股份有限公司 | Real-time data display method based on HBase table |
| CN111125121A (en) * | 2020-03-30 | 2020-05-08 | 四川新网银行股份有限公司 | Real-time data display method based on HBase table |
| CN111884883A (en) * | 2020-07-29 | 2020-11-03 | 北京宏达隆和科技有限公司 | Quick auditing processing method for service interface |
| CN112100148A (en) * | 2020-07-31 | 2020-12-18 | 紫光云(南京)数字技术有限公司 | Increment processing method for packed log |
| CN112860456A (en) * | 2021-02-08 | 2021-05-28 | 青岛海尔科技有限公司 | Log processing method and device |
| CN115190139A (en) * | 2022-03-28 | 2022-10-14 | 北京慧能分享科技有限公司 | Multi-protocol-based load balancing energy big data acquisition system and method |
| CN117215964A (en) * | 2023-11-09 | 2023-12-12 | 中央军委政治工作部军事人力资源保障中心 | Program anomaly observation method and device for service system |
| CN117215964B (en) * | 2023-11-09 | 2024-02-09 | 中央军委政治工作部军事人力资源保障中心 | Program anomaly observation method and device for service system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110362544B (en) | 2024-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110362544A (en) | Log processing system, log processing method, terminal and storage medium | |
| US10545964B2 (en) | Multi-phased data execution in a data processing system | |
| CN111241078A (en) | Data analysis system, data analysis method and device | |
| CN109634818A (en) | Log analysis method, system, terminal and computer readable storage medium | |
| CN106982150B (en) | Hadoop-based mobile internet user behavior analysis method | |
| US20120296967A1 (en) | Bridging Social Silos for Knowledge Discovery and Sharing | |
| CN113360554B (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| US10078843B2 (en) | Systems and methods for analyzing consumer sentiment with social perspective insight | |
| CN110347716A (en) | Daily record data processing method, device, terminal and storage medium | |
| WO2013106595A2 (en) | Processing store visiting data | |
| US20150142828A1 (en) | Ontology based categorization of users | |
| US11579860B2 (en) | Model driven state machine transitions to configure an installation of a software program | |
| CN104536830A (en) | KNN text classification method based on MapReduce | |
| CN110851234A (en) | Log processing method and device based on docker container | |
| CN110990057A (en) | Extraction method, device, equipment and medium of small program sub-chain information | |
| US11194869B2 (en) | Method and apparatus for enriching metadata via a network | |
| US11620541B1 (en) | Custom use case framework in computer analytics system | |
| Kavitha et al. | Discovering public opinions by performing sentimental analysis on real time Twitter data | |
| CN111159135A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN106557483B (en) | Data processing method, data query method, data processing equipment and data query equipment | |
| CN109167672B (en) | Return source error positioning method, device, storage medium and system | |
| CN112506887B (en) | Vehicle terminal CAN bus data processing method and device | |
| CN105245394A (en) | Method and equipment for analyzing network access log based on layered approach | |
| CN114756301B (en) | Log processing method, device and system | |
| US11734297B1 (en) | Monitoring platform job integration in computer analytics system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |