CN110347716A - Daily record data processing method, device, terminal and storage medium - Google Patents
Daily record data processing method, device, terminal and storage medium Download PDFInfo
- Publication number
- CN110347716A CN110347716A CN201910447654.1A CN201910447654A CN110347716A CN 110347716 A CN110347716 A CN 110347716A CN 201910447654 A CN201910447654 A CN 201910447654A CN 110347716 A CN110347716 A CN 110347716A
- Authority
- CN
- China
- Prior art keywords
- daily record
- record data
- real
- cluster
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 34
- 238000000034 method Methods 0.000 claims abstract description 85
- 230000008569 process Effects 0.000 claims abstract description 72
- 238000012545 processing Methods 0.000 claims abstract description 41
- 238000004458 analytical method Methods 0.000 claims abstract description 32
- 238000004590 computer program Methods 0.000 claims description 14
- 230000009467 reduction Effects 0.000 claims description 9
- 230000006399 behavior Effects 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 4
- 238000012550 audit Methods 0.000 abstract description 5
- 230000006870 function Effects 0.000 description 11
- 230000003993 interaction Effects 0.000 description 5
- 241001178520 Stomatepia mongo Species 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 241001269238 Data Species 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000003306 harvesting Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 238000005194 fractionation Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000036544 posture Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1464—Management of the backup or restore process for networked environments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/338—Presentation of query results
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the present invention provides a kind of daily record data processing method, comprising: obtains daily record data；Receive the process instruction for being directed to the daily record data；Judge that the process instruction instructs for the real-time process instruction of daily record data or daily record data processed offline；When process instruction process instruction real-time for the daily record data, the daily record data is handled in real time by Elasticsearch cluster；When the process instruction is that the daily record data processed offline instructs, processed offline is carried out to the daily record data by HBase cluster.The embodiment of the present invention also provides a kind of daily record data processing unit, terminal and computer readable storage medium.Using the embodiment of the present invention, efficient analysis and storage can be carried out for massive logs data, improves the efficiency for carrying out security audit using daily record data.
Description
Technical field
The present invention relates to journaling technique fields, and in particular to a kind of daily record data processing method, daily record data processing
Device, terminal and computer readable storage medium.
Background technique
It is all being steeply risen for the threat number amount and type of key message resource in network environment at present, it is how right in time
Active reaction is made in attack, is the research hotspot of network safety filed in recent years.By analysis daily record data to net
Network security postures, which are assessed, has obtained more and more extensive approval.With the development of computer and networks, the number of daily record data
Increasing according to treating capacity, the data magnitude of daily record data is usually million grades or more, even hundred tera-scale, thousand tera-scale with
On.For so huge daily record data system, higher requirement is referred to the processing of daily record data first.However, current
Daily record data processing system usually by log collection agency and analysis and management system form, can to the lesser log of data volume into
Row safety analysis, but in face of massive logs file large-scale, in complex network, it can not be preferable in a manner of form of tools work
Acquisition and analysis task are competent in ground, and are isolated dispersions between data, can not be associated, can not extract therein total
Property, lack the comprehensive analysis to whole daily record data, network can not be made to become an entirety to cope with security incident.
Summary of the invention
In view of the foregoing, it is necessary to provide a kind of daily record data processing method, daily record data processing unit, terminal and
Computer readable storage medium can carry out efficient analysis and storage for massive logs data, improve and utilize log number
According to the efficiency for carrying out security audit.
First aspect of the embodiment of the present invention provides a kind of daily record data processing method, the daily record data processing method packet
It includes:
Obtain daily record data；
Receive the process instruction for being directed to the daily record data；
Judge that the process instruction instructs for the real-time process instruction of daily record data or daily record data processed offline；
When process instruction process instruction real-time for the daily record data, by Elasticsearch cluster to institute
Daily record data is stated to be handled in real time；
When the process instruction is that the daily record data processed offline instructs, by HBase cluster to the log number
According to progress processed offline.
Further, described to pass through in above-mentioned daily record data processing method provided in an embodiment of the present invention
Elasticsearch cluster carries out processing in real time to the daily record data
Real-time retrieval is carried out to the daily record data according to key search mode, and retrieval knot is shown with predetermined manner
Fruit；
Real-time Alarm is carried out to the daily record data according to default alarm regulation, the default alarm regulation includes in following
One or more combinations: event alarm, statistics alarm, continuous statistics alarm and baseline compare alarm；
Rule match is carried out to the daily record data according to default statistical rules, and to meeting the default statistical rules
Daily record data carries out real-time statistics.
Further, in above-mentioned daily record data processing method provided in an embodiment of the present invention, the continuous statistics alarm
Include:
The daily record data is counted to obtain statistic analysis result according to statistical rules；
The default output-index in the statistic analysis result is verified according to pre-set level threshold value, judges the system
Whether the default output-index in meter analysis result is more than the pre-set level threshold value；
If judging, the default output-index in the statistic analysis result is more than the pre-set level threshold value, exports default accuse
Alert be prompted to is preset using responsible person.
Further, described to pass through HBase cluster in above-mentioned daily record data processing method provided in an embodiment of the present invention
Carrying out processed offline to the daily record data includes one of following or a variety of combination:
Off-line analysis is carried out to the daily record data by the HBase cluster, the off-line analysis includes offline logs
Data clusters analysis and user behavior analysis；
Log backup is carried out to the daily record data by the HBase cluster；
Log reduction is carried out to the daily record data by the HBase cluster.
Further, described to pass through the HBase in above-mentioned daily record data processing method provided in an embodiment of the present invention
Cluster carries out Log backup to the daily record data
The index information in the Elasticsearch cluster is read by Transmission Control Protocol；
The daily record data in the Elasticsearch cluster is obtained according to the index information；
The HBase cluster is written into daily record data in the Elasticsearch cluster and carries out Log backup.
Further, described to pass through the HBase in above-mentioned daily record data processing method provided in an embodiment of the present invention
Cluster carries out log reduction to the daily record data
Read the daily record data in the HBase cluster；
It will be in the HBase cluster of reading by way of the Bluk API in the Elasticsearch cluster
Daily record data writes back the Elasticsearch cluster and carries out log reduction.
Further, in above-mentioned daily record data processing method provided in an embodiment of the present invention, in the acquisition log number
According to later, the method also includes:
The daily record data is shunted by Kafka cluster, obtains real-time logs data and non real-time daily record data；
The real-time logs data are input in the Elasticsearch cluster；
The non real-time daily record data is input in the HBase cluster.
Second aspect of the embodiment of the present invention also provides a kind of daily record data processing unit, the daily record data processing unit packet
It includes:
Log acquisition module, for obtaining daily record data；
Command reception module, for receiving the process instruction for being directed to the daily record data；
Judgment module is instructed, for judging that the process instruction is offline for the real-time process instruction of daily record data or daily record data
Process instruction；
Real-time processing module, for passing through when process instruction process instruction real-time for the daily record data
Elasticsearch cluster handles the daily record data in real time；
Processed offline module, for passing through HBase when the process instruction is that the daily record data processed offline instructs
Cluster carries out processed offline to the daily record data.
The third aspect of the embodiment of the present invention also provides a kind of terminal, and the terminal includes processor, and the processor is used for
Daily record data processing method described in any of the above embodiments is realized when executing the computer program stored in memory.
Fourth aspect of the embodiment of the present invention also provides a kind of computer readable storage medium, the computer-readable storage medium
Computer program is stored in matter, the computer program realizes daily record data described in any of the above embodiments when being executed by processor
Processing method.
The embodiment of the present invention provides a kind of daily record data processing method, daily record data processing unit, terminal and computer
Readable storage medium storing program for executing obtains daily record data；Receive the process instruction for being directed to the daily record data；Judge the process instruction for day
The instruction of will generating date or the instruction of daily record data processed offline；When the process instruction is that the daily record data is handled in real time
When instruction, the daily record data is handled in real time by Elasticsearch cluster；When the process instruction is the day
When will off-line data process instruction, processed offline is carried out to the daily record data by HBase cluster.Implemented using the present invention
Example can carry out efficient analysis and storage for massive logs data, improve and carry out security audit using daily record data
Efficiency.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The embodiment of invention for those of ordinary skill in the art without creative efforts, can also basis
The attached drawing of offer obtains other attached drawings.
Fig. 1 is the flow chart for the daily record data processing method that first embodiment of the invention provides.
Fig. 2 is the structural schematic diagram of the terminal of an embodiment of the present invention.
Fig. 3 is the illustrative functional block diagram of terminal shown in Fig. 2.
Main element symbol description
| Terminal | 1 |
| Memory | 10 |
| Display screen | 20 |
| Processor | 30 |
| Daily record data processing unit | 100 |
| Log acquisition module | 101 |
| Command reception module | 102 |
| Instruct judgment module | 103 |
| Real-time processing module | 104 |
| Processed offline module | 105 |
The embodiment of the present invention that the following detailed description will be further explained with reference to the above drawings.
Specific embodiment
In order to be more clearly understood that the above objects, features, and advantages of the embodiment of the present invention, with reference to the accompanying drawing and
The present invention will be described in detail for specific embodiment.It should be noted that in the absence of conflict, the embodiment party of the application
Feature in formula can be combined with each other.
Embodiment in the following description, numerous specific details are set forth in order to facilitate a full understanding of the present invention, described reality
The mode of applying is only some embodiments of the invention, rather than whole embodiments.Based on the embodiment in the present invention,
Every other embodiment obtained by those of ordinary skill in the art without making creative efforts belongs to this
The range of inventive embodiments protection.
Unless otherwise defined, all technical and scientific terms used herein and the technology for belonging to the embodiment of the present invention
The normally understood meaning of the technical staff in field is identical.Term as used herein in the specification of the present invention is intended merely to
The purpose of specific embodiment is described, it is not intended that in the limitation embodiment of the present invention.
Fig. 1 is the flow chart of the daily record data processing method of first embodiment of the invention.The daily record data processing side
Method can be applied to terminal 1, and the terminal 1 can be such as smart phone, laptop, desk-top/tablet computer, intelligent hand
The smart machines such as table and personal digital assistant (Personal Digital Assistant, PDA).As shown in Figure 1, the day
Will data processing method may include steps of:
S101: daily record data is obtained.
In the present embodiment, the daily record data obtained from default source database by log acquisition module, it is described
The type of daily record data may include user behavior data, application state data or device status data, the default source number
It can be that system operators are pre-set according to library, the content of daily record data, source be not defined herein.The day
Will, which obtains module, can be used Filebeat progress log data acquisition (hereinafter referred to as Filebeat log acquisition module), described
Filebeat is log data acquisition device.The Filebeat log acquisition module is supported to customize the transmission of all kinds of daily record datas
Side, the Filebeat log acquisition module export the daily record data to all kinds of log numbers for obtaining daily record data
According to recipient.It is gone specifically, the Filebeat log acquisition module starts one or more detectors (prospectors)
Detection specified Log Directory or file；For each journal file that the detector is found out, the Filebeat log
Acquisition module starts harvesting process (harvester)；Each harvesting is read out the new content of a journal file, and
The new content of the journal file is sent to processing routine (spooler), the processing routine can gather these daily record datas, most
The Filebeat log acquisition module can send the daily record data of set to specified place afterwards.It is understood that institute
It states after obtaining daily record data, the daily record data can also be converted according to preset structure, specifically, the log number
According to preset structure may include logging time, log rank, log output class and log content etc..
In the present embodiment, after the acquisition daily record data, the method also includes: pass through Kafka cluster (institute
Stating Kafka cluster is a kind of distributed message caching middleware, has the characteristics that high-throughput (even with very common
Hardware, Kafka can also support hundreds of thousands of message per second), for the caching of mass data, by way of message queue,
Data are distributed and are controlled.) daily record data is shunted, obtain real-time logs data and non real-time log number
According to；The real-time logs data are input in the Elasticsearch cluster；The non real-time daily record data is input to
In the HBase cluster.Wherein, described that the daily record data shunt including using Strom streaming by Kafka cluster
Computational frame must be analyzed and processed the daily record data cached in the Kafka message queue, obtain real-time logs data with
And non real-time daily record data.It in other embodiments, can also (ZooKeeper be a distribution by Zookeeper
, the distributed application program coordination service of open source code) cluster classifies to the daily record data, obtain real-time logs number
Accordingly and non real-time daily record data.It is understood that the real-time logs data are input to the Elasticsearch
Before in cluster, the method also includes: receive the real-time logs data in the different topic cached in Kafka message queue；
Parsing operation is carried out to the real-time logs data according to default resolution rules by Logstash log analyzing module.It is described logical
Crossing Logstash log analyzing module and carrying out parsing to the real-time logs data according to default resolution rules includes passing through
Logstash log analyzing module is cleaned and is processed to the real-time logs data, and by the real-time logs data structure
It is melted into different fields.Journal file is parsed by Logstash log analyzing module, can recognize that be processed
Described first shunts the useful information in daily record data, filters out junk data.Match in the Logstash log analyzing module
It is equipped with the resolution file in all log sources, the default resolution rules are the rule being arranged in the resolution file.
Before the non real-time daily record data is input in the HBase cluster, the method also includes: it reads pre-
Determine resolution rules；Parsing operation is carried out to the real-time logs data according to predetermined resolution rules by Spark cluster, it will be described
Real-time logs data resolve to HBase tables of data format, and the HBase tables of data format after parsing is stored to the HBase collection
In group.Wherein, the predetermined resolution rules can be that system developer is pre-set, and the predetermined resolution rules can wrap
Include regular expression, KeyValue parsing, field value fractionation (for example, being split using split function), String type turn
Change one of numeric type, Json parsing, URL decoding, time-stamp Recognition and UserAgent parsing or a variety of into.
S102: the process instruction for being directed to the daily record data is received.
In the present embodiment, the process instruction for being directed to the daily record data, the process instruction of the daily record data are received
It is instructed including the real-time process instruction of daily record data and daily record data processed offline, wherein the real-time process instruction of daily record data
Including real-time retrieval instruction, Real-time Alarm instruction and Online statistics instruction, daily record data processed offline instruction include from
Line analysis instruction, Log backup instruction and log reduction instruction.The embodiment of the present invention provides an interactive interface, in the interaction
On interface, corresponding touch area is provided with for the process instruction of each daily record data.By receiving in corresponding touching
The predetermined registration operation (for example, mouse click or finger touching etc.) of control region output obtains referring to for the processing of the daily record data
It enables.
S103: judge that the process instruction instructs for the real-time process instruction of daily record data or daily record data processed offline.
In the present embodiment, after receiving the process instruction for the daily record data, judge the process instruction
It is instructed for the real-time process instruction of daily record data or daily record data processed offline, when the process instruction is that the daily record data is real-time
When process instruction, step S104 is executed；When the process instruction is that the daily record data processed offline instructs, step is executed
S105。
S104: the daily record data is handled in real time by Elasticsearch cluster.
In the present embodiment, when process instruction process instruction real-time for the daily record data, pass through
Elasticsearch cluster handles the daily record data in real time.It is described by Elasticsearch cluster to the day
It includes: according to key search mode to daily record data progress real-time retrieval that will data, which carry out processing in real time, and with default
Mode shows search result；The predetermined manner includes bright to testing result progress overstriking, mark.For the log comprising keyword
Data are also supported to check the context that the daily record data comprising log keyword prints.
Alternatively, carrying out Real-time Alarm to the daily record data according to default alarm regulation, the default alarm regulation includes
One of below or a variety of combinations: event alarm, statistics alarm, continuous statistics alarm, baseline compare alarm and system
Meter alarm；Wherein, for the event alarm rule, the alarm triggered condition based on daily record data search result is created, for example,
The preset threshold number for triggering alarm in preset time range is set, if the quantity of practical triggering alarm is greater than the preset threshold
Number, then carry out alarm prompt.For the statistics alarm regulation, the alarm setting for field contents is provided, is being triggered
Field contents can be filled in condition, statistical can select in the combobox of interactive interface, including cardinality
(separate counts), sum (summation), avg (average value), max (maximum value) and min (minimum value).The continuous statistics is accused
Police regulations then, provide continuous trigger alarm setting, alarm conditions are arranged, when alarm conditions continuous trigger within a preset period of time
When number reaches preset threshold, then alarm is triggered.Alarm regulation is compared for the baseline, a system can be set the threshold to
The baseline value (baseline value can change at any time) of meter, the time range for selecting baseline to generate.Meanwhile baseline comparison alarm mentions
Supplied more flexible trigger range setting means, for example, can select to be greater than in combobox, be less than, in section and
Outside section.For counting alarm regulation, the statistics alarm includes: count to the daily record data according to statistical rules
To statistic analysis result；The default output-index in the statistic analysis result is verified according to pre-set level threshold value, is sentenced
Whether the default output-index in the statistic analysis result of breaking is more than the pre-set level threshold value；If judging the statistical analysis
As a result the default output-index in is more than the pre-set level threshold value, exports default alarm prompt to default using responsible person.Institute
Stating statistical rules includes default statistical item and default output-index, wherein the default statistical item includes preassigning
The field information to be counted (for example, the field informations such as clientip, requestURL).The default output-index includes described
(for example, default output-index is quantity (count), the quantity may include described preparatory to the output valve of default statistical item
The specified statistical magnitude for wanting static fields information).The pre-set level threshold value is the pre-set value of terminal user.It is described default
It is pre-set using responsible person using artificial terminal user is responsible for.
Alternatively, carrying out rule match to the daily record data according to default statistical rules, and to meeting the default statistics
The daily record data of rule carries out real-time statistics.It is described that rule match is carried out to the daily record data according to default statistical rules, and
Carrying out real-time statistics to the daily record data for meeting the default statistical rules includes: according to the default statistical rules received to institute
It states daily record data and carries out rule match, and the information to be counted for meeting the default statistical rules is counted, output statistics
As a result.The statistical result can be shown by forms such as broken line, table, bar shaped, pie.The default statistical rules can prop up
It holds and the operation such as is increased, modified, deleted, search and stored on interactive interface.
S105: processed offline is carried out to the daily record data by HBase cluster.
In the present embodiment, when the process instruction is that the daily record data processed offline instructs, pass through HBase collection
Group carries out processed offline to the daily record data.It is described to include to daily record data progress processed offline by HBase cluster
One of below or a variety of combinations: off-line analysis is carried out to the daily record data by the HBase cluster, it is described offline
Analysis includes the analysis of offline logs data clusters and user behavior analysis；By the HBase cluster to the daily record data into
Row Log backup；Log reduction is carried out to the daily record data by the HBase cluster.
Wherein, it is described by the HBase cluster to the daily record data carry out Log backup include: to pass through Transmission Control Protocol
Read the index information in the Elasticsearch cluster；The Elasticsearch collection is obtained according to the index information
Daily record data in group；It is standby that the HBase cluster progress log is written into daily record data in the Elasticsearch cluster
Part.It is described that the daily record data is carried out log to restore including: to read in the HBase cluster by the HBase cluster
Daily record data；Pass through Bluk API (the Bluk interface, in an interface calls in the Elasticsearch cluster
Include multiple index operations) mode the daily record data in the HBase cluster of reading is write back into the Elasticsearch
Cluster carries out log reduction.
In the present embodiment, the daily record data is handled in real time by Elasticsearch cluster described
Later, real-time processing result is exported；It is described by HBase cluster to the daily record data carry out processed offline after, output
Processed offline result.The real-time processing result and the processed offline result can be shown by the result in Web client
Module is shown.The embodiment of the present invention also provides Mysql database, Mongo database and web application.The Web is answered
It is connect with program with the Mysql database and Mongo database.Wherein, the Mysql database is a kind of open source code
Relational DBMS, mainly store resource distribution related data in the Mysql database.The Mongo number
It is the database based on distributed document storage according to library, it is intended to which expansible high-performance data storage is provided for WEB application
Solution mainly stores the statistic analysis result of daily record data in the Mongo database.
The web application is also connected with each other with Web server, and the Web server is for receiving Web client
What is passed is used to carry out the interaction data of data interaction with web application, and the interaction data is exported by interface to Web
Application program after web application handles interaction data, obtains processing result, and processing result is fed back to Web clothes
Business device, feeds back to Web client for processing result by Web server, passes through the result display module in the Web client
Result is shown.
The embodiment of the present invention provides a kind of daily record data processing method, obtains daily record data；It receives and is directed to the log number
According to process instruction；Judge that the process instruction instructs for the real-time process instruction of daily record data or daily record data processed offline；When
When the process instruction is the daily record data real-time process instruction, by Elasticsearch cluster to the daily record data
It is handled in real time；When the process instruction is that the daily record data processed offline instructs, by HBase cluster to the day
Will data carry out processed offline.Using the embodiment of the present invention, efficient analysis and storage can be carried out for massive logs data,
Improve the efficiency that security audit is carried out using daily record data.
Fig. 2 is the structural schematic diagram of the terminal 1 of an embodiment of the present invention, as shown in Fig. 2, terminal 1 includes memory 10,
Daily record data processing unit 100 is stored in memory 10.The terminal 1 can be mobile phone, tablet computer, individual digital and help
Reason etc. has the terminal 1 using display function.The available daily record data of the daily record data processing unit 100；Reception is directed to
The process instruction of the daily record data；Judge that the process instruction is located offline for the real-time process instruction of daily record data or daily record data
Reason instruction；When process instruction process instruction real-time for the daily record data, by Elasticsearch cluster to described
Daily record data is handled in real time；When the process instruction is that the daily record data processed offline instructs, pass through HBase cluster
Processed offline is carried out to the daily record data.Using the embodiment of the present invention, can efficiently be divided for massive logs data
Analysis and storage, improve the efficiency that security audit is carried out using daily record data.
In present embodiment, terminal 1 can also include display screen 20 and processor 30.Memory 10, display screen 20 can be with
It is electrically connected respectively with processor 30.
The memory 10 can be different type storage equipment, for storing Various types of data.For example, it may be terminal
1 memory, memory, can also be the storage card that can be external in the terminal 1, as flash memory, SM card (Smart Media Card,
Smart media card), SD card (Secure Digital Card, safe digital card) etc..In addition, memory 10 may include high speed
Random access memory can also include nonvolatile memory, such as hard disk, memory, plug-in type hard disk, intelligent memory card
(Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash card (Flash Card), at least
One disk memory, flush memory device or other volatile solid-state parts.Memory 10 is used to store Various types of data,
For example, the types of applications program (Applications) installed in the terminal 1, setting using above-mentioned daily record data processing method
The information such as the data set, obtained.
Display screen 20 is installed on terminal 1, for showing information.
Processor 30 is used to execute all kinds of softwares installed in the daily record data processing method and the terminal 1, example
Such as operating system and application display software.Processor 30 is including but not limited to processor (Central Processing
Unit, CPU), micro-control unit (Micro Controller Unit, MCU) etc. is for interpretive machine and processing computer
The device of data in software.
The daily record data processing unit 100 may include one or more modules, one or more of modules
Be stored in the memory 10 of terminal 1 and be configured to by one or more processors (present embodiment be a processor
30) it executes, to complete the embodiment of the present invention.For example, as shown in fig.3, the daily record data processing unit 100 may include day
Will obtains module 101, command reception module 102, instruction judgment module 103, real-time processing module 104 and processed offline module
105.The so-called module of the embodiment of the present invention can be the program segment for completing a specific function, than program more suitable for describing software
Implementation procedure in the processor.
It is understood that each embodiment in corresponding above-mentioned daily record data processing method, terminal 1 may include Fig. 3
Shown in part or all in each functional module, the function of each module will introduced in detail below.It should be noted that
Identical noun related terms and its specific illustrate can also be in each embodiment of the above daily record data processing method
Suitable for the function introduction below to each module.For the sake of saving space and avoiding repetition, details are not described herein again.
Log acquisition module 101 can be used for obtaining daily record data.
Command reception module 102 can be used for receiving the process instruction for being directed to the daily record data.
Instruction judgment module 103 can be used for judging the process instruction for the real-time process instruction of daily record data or log number
It is instructed according to processed offline.
Real-time processing module 104 can be used for leading to when process instruction process instruction real-time for the daily record data
Elasticsearch cluster is crossed to handle the daily record data in real time.
Processed offline module 105 can be used for leading to when the process instruction is daily record data processed offline instruction
It crosses HBase cluster and processed offline is carried out to the daily record data.
The embodiment of the present invention also provides a kind of computer readable storage medium, is stored thereon with computer program, the meter
Calculation machine program realizes the step of daily record data processing method in any of the above-described embodiment when being executed by processor.
If the integrated module/unit of 100/ terminal of daily record data processing unit, 1/ computer equipment is with software function
The form of unit is realized and when sold or used as an independent product, can store in a computer-readable storage medium
In.Based on this understanding, the present invention realizes all or part of the process in above embodiment method, can also pass through calculating
Machine program is completed to instruct relevant hardware, and the computer program can be stored in a computer readable storage medium,
The computer program is when being executed by processor, it can be achieved that the step of above-mentioned each embodiment of the method.Wherein, the computer journey
Sequence includes computer program code, and the computer program code can be source code form, object identification code form, executable text
Part or certain intermediate forms etc..The computer readable storage medium may include: that can carry the computer program code
Any entity or device, recording medium, USB flash disk, mobile hard disk, magnetic disk, CD, computer storage, read-only memory (ROM,
Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunications letter
Number and software distribution medium etc..
Alleged processor 30 can be central processing unit (Central Processing Unit, CPU), can also be
Other general processors, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit
(Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be microprocessor or the processor is also possible to any conventional processor
Deng the processor 30 is the control centre of 100/ terminal 1 of daily record data processing unit, and various interfaces and route is utilized to connect
Connect the various pieces of entire 100/ terminal 1 of daily record data processing unit.
For the memory 10 for storing the computer program and/or module, the processor 30 is by operation or holds
Row stores computer program and/or module in the memory, and calls the data being stored in memory 10, realizes
The various functions of 100/ terminal 1 of daily record data processing unit.The memory 10 can mainly include storing program area and deposit
Store up data field, wherein storing program area can application program needed for storage program area, at least one function (for example sound is broadcast
Playing function, image player function etc.) etc.；Storage data area, which can be stored, uses created data (such as audio according to mobile phone
Data, phone directory etc.) etc..
In several specific embodiments provided by the present invention, it should be understood that disclosed terminal and method, it can be with
It realizes by another way.For example, system embodiment described above is only schematical, for example, the module
Division, only a kind of logical function partition, there may be another division manner in actual implementation.
It is obvious to a person skilled in the art that the embodiment of the present invention is not limited to the details of above-mentioned exemplary embodiment,
And without departing substantially from the spirit or essential attributes of the embodiment of the present invention, this hair can be realized in other specific forms
Bright embodiment.Therefore, in all respects, the present embodiments are to be considered as illustrative and not restrictive, this
The range of inventive embodiments is indicated by the appended claims rather than the foregoing description, it is intended that being equal for claim will be fallen in
All changes in the meaning and scope of important document are included in the embodiment of the present invention.It should not be by any attached drawing mark in claim
Note is construed as limiting the claims involved.Multiple units, module or the device stated in claim can also be by same
Unit, module or device are implemented through software or hardware.
Embodiment of above is only to illustrate the technical solution of the embodiment of the present invention rather than limits, although referring to above preferable
The embodiment of the present invention is described in detail in embodiment, those skilled in the art should understand that, it can be to this hair
The technical solution of bright embodiment is modified or equivalent replacement should not all be detached from the embodiment of the present invention technical solution spirit and
Range.
Claims (10)
1. a kind of daily record data processing method, which is characterized in that the daily record data processing method includes:
Obtain daily record data；
Receive the process instruction for being directed to the daily record data；
Judge that the process instruction instructs for the real-time process instruction of daily record data or daily record data processed offline；
When process instruction process instruction real-time for the daily record data, by Elasticsearch cluster to the day
Will data are handled in real time；
When the process instruction be the daily record data processed offline instruct when, by HBase cluster to the daily record data into
Row processed offline.
2. daily record data processing method according to claim 1, which is characterized in that described to pass through Elasticsearch collection
Group carries out processing in real time to the daily record data
Real-time retrieval is carried out to the daily record data according to key search mode, and search result is shown with predetermined manner；
Real-time Alarm is carried out to the daily record data according to default alarm regulation, the default alarm regulation includes one in following
Kind or a variety of combinations: event alarm, statistics alarm, continuous statistics alarm, baseline comparison alarm are alerted with statistics；
Rule match, and the log to the default statistical rules is met are carried out to the daily record data according to default statistical rules
Data carry out real-time statistics.
3. daily record data processing method according to claim 2, which is characterized in that the statistics, which alerts, includes:
The daily record data is counted to obtain statistic analysis result according to statistical rules；
The default output-index in the statistic analysis result is verified according to pre-set level threshold value, judges the statistical
Whether the default output-index analysed in result is more than the pre-set level threshold value；
If judging, the default output-index in the statistic analysis result is more than the pre-set level threshold value, exports default alarm and mentions
Show to default using responsible person.
4. daily record data processing method according to claim 1, which is characterized in that it is described by HBase cluster to described
It includes one of following or a variety of combination that daily record data, which carries out processed offline:
Off-line analysis is carried out to the daily record data by the HBase cluster, the off-line analysis includes offline logs data
Clustering and user behavior analysis；
Log backup is carried out to the daily record data by the HBase cluster；
Log reduction is carried out to the daily record data by the HBase cluster.
5. daily record data processing method according to claim 4, which is characterized in that described to pass through the HBase cluster pair
The daily record data carries out Log backup
The index information in the Elasticsearch cluster is read by Transmission Control Protocol；
The daily record data in the Elasticsearch cluster is obtained according to the index information；
The HBase cluster is written into daily record data in the Elasticsearch cluster and carries out Log backup.
6. daily record data processing method according to claim 5, which is characterized in that described to pass through the HBase cluster pair
The daily record data carries out log reduction
Read the daily record data in the HBase cluster；
By the log in the HBase cluster of reading by way of the Bluk API in the Elasticsearch cluster
Data write back the Elasticsearch cluster and carry out log reduction.
7. daily record data processing method according to claim 1, which is characterized in that after the acquisition daily record data,
The method also includes:
The daily record data is shunted by Kafka cluster, obtains real-time logs data and non real-time daily record data；
The real-time logs data are input in the Elasticsearch cluster；
The non real-time daily record data is input in the HBase cluster.
8. a kind of daily record data processing unit, which is characterized in that the daily record data processing unit includes:
Log acquisition module, for obtaining daily record data；
Command reception module, for receiving the process instruction for being directed to the daily record data；
Judgment module is instructed, for judging the process instruction for the real-time process instruction of daily record data or daily record data processed offline
Instruction；
Real-time processing module, for passing through when process instruction process instruction real-time for the daily record data
Elasticsearch cluster handles the daily record data in real time；
Processed offline module, for passing through HBase cluster when the process instruction is that the daily record data processed offline instructs
Processed offline is carried out to the daily record data.
9. a kind of terminal, which is characterized in that the terminal includes processor, and the processor is used to execute to store in memory
Such as claim 1-7 described in any item daily record data processing methods are realized when computer program.
10. a kind of computer readable storage medium, computer program, feature are stored on the computer readable storage medium
It is, such as the described in any item daily record data processing sides claim 1-7 is realized when the computer program is executed by processor
Method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910447654.1A CN110347716B (en) | 2019-05-27 | 2019-05-27 | Log data processing method, device, terminal equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910447654.1A CN110347716B (en) | 2019-05-27 | 2019-05-27 | Log data processing method, device, terminal equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110347716A true CN110347716A (en) | 2019-10-18 |
| CN110347716B CN110347716B (en) | 2024-04-02 |
Family
ID=68173983
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910447654.1A Active CN110347716B (en) | 2019-05-27 | 2019-05-27 | Log data processing method, device, terminal equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110347716B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111125042A (en) * | 2019-11-13 | 2020-05-08 | 中国建设银行股份有限公司 | Method and device for determining risk operation event |
| CN111404909A (en) * | 2020-03-10 | 2020-07-10 | 上海豌豆信息技术有限公司 | Security detection system and method based on log analysis |
| CN111881011A (en) * | 2020-07-31 | 2020-11-03 | 网易（杭州）网络有限公司 | Log management method, platform, server and storage medium |
| CN112131283A (en) * | 2020-09-30 | 2020-12-25 | 重庆市海普软件产业有限公司 | Intelligent acquisition system capable of being flexibly expanded |
| CN113221033A (en) * | 2021-04-24 | 2021-08-06 | 上海钢银科技发展有限公司 | Buried point acquisition and statistical analysis method, system, equipment and storage medium |
| CN113238912A (en) * | 2021-05-08 | 2021-08-10 | 国家计算机网络与信息安全管理中心 | Aggregation processing method for network security log data |
| CN113283884A (en) * | 2020-12-31 | 2021-08-20 | 深圳怡化电脑股份有限公司 | Log processing method and device |
| CN113312353A (en) * | 2021-06-10 | 2021-08-27 | 中国民航信息网络股份有限公司 | Storage method and system for tracking journal |
| CN113411206A (en) * | 2021-05-26 | 2021-09-17 | 北京沃东天骏信息技术有限公司 | Log auditing method, device, equipment and computer storage medium |
| CN113783849A (en) * | 2021-08-25 | 2021-12-10 | 福建天泉教育科技有限公司 | Sensitive information detection method and terminal |
| CN113902469A (en) * | 2021-09-17 | 2022-01-07 | 作业帮教育科技（北京）有限公司 | Advertisement diagnosis platform, device and electronic equipment |
| CN116991661A (en) * | 2023-07-20 | 2023-11-03 | 北京直客通科技有限公司 | Problem alarm system and method for software system |
| CN113312353B (en) * | 2021-06-10 | 2024-06-04 | 中国民航信息网络股份有限公司 | Storage method and system for tracking belt log |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106790718A (en) * | 2017-03-16 | 2017-05-31 | 北京搜狐新媒体信息技术有限公司 | Service call link analysis method and system |
| US20170169078A1 (en) * | 2015-12-14 | 2017-06-15 | Siemens Aktiengesellschaft | Log Mining with Big Data |
| CN107294801A (en) * | 2016-12-30 | 2017-10-24 | 江苏号百信息服务有限公司 | Stream Processing method and system based on magnanimity real-time Internet DPI data |
| CN107577588A (en) * | 2017-09-26 | 2018-01-12 | 北京中安智达科技有限公司 | A kind of massive logs data intelligence operational system |
| US20180101423A1 (en) * | 2016-10-11 | 2018-04-12 | Oracle International Corporation | Cluster-based processing of unstructured log messages |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
-
2019
- 2019-05-27 CN CN201910447654.1A patent/CN110347716B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170169078A1 (en) * | 2015-12-14 | 2017-06-15 | Siemens Aktiengesellschaft | Log Mining with Big Data |
| US20180101423A1 (en) * | 2016-10-11 | 2018-04-12 | Oracle International Corporation | Cluster-based processing of unstructured log messages |
| CN107294801A (en) * | 2016-12-30 | 2017-10-24 | 江苏号百信息服务有限公司 | Stream Processing method and system based on magnanimity real-time Internet DPI data |
| CN106790718A (en) * | 2017-03-16 | 2017-05-31 | 北京搜狐新媒体信息技术有限公司 | Service call link analysis method and system |
| CN107577588A (en) * | 2017-09-26 | 2018-01-12 | 北京中安智达科技有限公司 | A kind of massive logs data intelligence operational system |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111125042A (en) * | 2019-11-13 | 2020-05-08 | 中国建设银行股份有限公司 | Method and device for determining risk operation event |
| CN111404909B (en) * | 2020-03-10 | 2022-05-31 | 上海豌豆信息技术有限公司 | Safety detection system and method based on log analysis |
| CN111404909A (en) * | 2020-03-10 | 2020-07-10 | 上海豌豆信息技术有限公司 | Security detection system and method based on log analysis |
| CN111881011A (en) * | 2020-07-31 | 2020-11-03 | 网易（杭州）网络有限公司 | Log management method, platform, server and storage medium |
| CN112131283A (en) * | 2020-09-30 | 2020-12-25 | 重庆市海普软件产业有限公司 | Intelligent acquisition system capable of being flexibly expanded |
| CN113283884A (en) * | 2020-12-31 | 2021-08-20 | 深圳怡化电脑股份有限公司 | Log processing method and device |
| CN113221033A (en) * | 2021-04-24 | 2021-08-06 | 上海钢银科技发展有限公司 | Buried point acquisition and statistical analysis method, system, equipment and storage medium |
| CN113238912A (en) * | 2021-05-08 | 2021-08-10 | 国家计算机网络与信息安全管理中心 | Aggregation processing method for network security log data |
| CN113238912B (en) * | 2021-05-08 | 2022-12-06 | 国家计算机网络与信息安全管理中心 | Aggregation processing method for network security log data |
| CN113411206A (en) * | 2021-05-26 | 2021-09-17 | 北京沃东天骏信息技术有限公司 | Log auditing method, device, equipment and computer storage medium |
| CN113411206B (en) * | 2021-05-26 | 2022-09-06 | 北京沃东天骏信息技术有限公司 | Log auditing method, device, equipment and computer storage medium |
| CN113312353A (en) * | 2021-06-10 | 2021-08-27 | 中国民航信息网络股份有限公司 | Storage method and system for tracking journal |
| CN113312353B (en) * | 2021-06-10 | 2024-06-04 | 中国民航信息网络股份有限公司 | Storage method and system for tracking belt log |
| CN113783849A (en) * | 2021-08-25 | 2021-12-10 | 福建天泉教育科技有限公司 | Sensitive information detection method and terminal |
| CN113902469A (en) * | 2021-09-17 | 2022-01-07 | 作业帮教育科技（北京）有限公司 | Advertisement diagnosis platform, device and electronic equipment |
| CN116991661A (en) * | 2023-07-20 | 2023-11-03 | 北京直客通科技有限公司 | Problem alarm system and method for software system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110347716B (en) | 2024-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110347716A (en) | Daily record data processing method, device, terminal and storage medium | |
| JP7373611B2 (en) | Log auditing methods, equipment, electronic equipment, media and computer programs | |
| CN106815125A (en) | A kind of log audit method and platform | |
| CN110362544A (en) | Log processing system, log processing method, terminal and storage medium | |
| CN109634818A (en) | Log analysis method, system, terminal and computer readable storage medium | |
| CN109034993A (en) | Account checking method, equipment, system and computer readable storage medium | |
| CN110428127B (en) | Automatic analysis method, user equipment, storage medium and device | |
| CN111431926B (en) | Data association analysis method, system, equipment and readable storage medium | |
| CN107111625A (en) | Realize the method and system of the efficient classification and exploration of data | |
| CN102323873B (en) | In order to trigger the method and system that icon is replied in instant messaging | |
| CN108073625A (en) | For the system and method for metadata information management | |
| CN109240895A (en) | A kind of processing method and processing device for analyzing log failure | |
| CN113254255B (en) | Cloud platform log analysis method, system, device and medium | |
| CN113157947A (en) | Knowledge graph construction method, tool, device and server | |
| US11568344B2 (en) | Systems and methods for automated pattern detection in service tickets | |
| CN107480189A (en) | A kind of various dimensions real-time analyzer and method | |
| CN111858560A (en) | Financial data automated testing and monitoring system based on data warehouse | |
| CN110677271A (en) | Big data alarm method, device, equipment and storage medium based on ELK | |
| CN115964392A (en) | Real-time monitoring method, device and equipment based on flink and readable storage medium | |
| CN115495587A (en) | Alarm analysis method and device based on knowledge graph | |
| CN105653533A (en) | Method and device for updating classified associated word set | |
| CN115408236A (en) | Log data auditing system, method, equipment and medium | |
| WO2021129849A1 (en) | Log processing method, apparatus and device, and storage medium | |
| CN113595886A (en) | Instant messaging message processing method and device, electronic equipment and storage medium | |
| JP6070338B2 (en) | Classification device for processing system included in multi-tier system, classification program for processing system included in multi-tier system, and classification method for processing system included in multi-tier system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |